API » History » Version 53
Maarten Plieger, 03/30/2016 03:07 PM
1 | 1 | Maarten Plieger | h1. API |
---|---|---|---|
2 | 1 | Maarten Plieger | |
3 | 48 | Maarten Plieger | h2. Clipc combine process |
4 | 48 | Maarten Plieger | |
5 | 49 | Maarten Plieger | The WPS to combine two netcdf files: [[Clipc Combine process]] |
6 | 48 | Maarten Plieger | |
7 | 28 | Maarten Plieger | h2. Access token API - service "account" |
8 | 1 | Maarten Plieger | |
9 | 28 | Maarten Plieger | *1). Generate a token for the current user which can be used to access C4I services by using "generatetoken":* |
10 | 33 | Maarten Plieger | * https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken |
11 | 1 | Maarten Plieger | |
12 | 17 | Maarten Plieger | The token has the following structure: |
13 | 17 | Maarten Plieger | <pre><code class="json"> |
14 | 18 | Maarten Plieger | { |
15 | 18 | Maarten Plieger | "creationdate" : "2015-10-23T09:50:05Z", |
16 | 17 | Maarten Plieger | "notafter" : "2015-10-30T09:50:05Z", |
17 | 17 | Maarten Plieger | "notbefore" : "2015-10-23T09:50:05Z", |
18 | 22 | Maarten Plieger | "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>", |
19 | 24 | Maarten Plieger | "token" : "daad2944-1def-4bee-b032-b08048f88302", |
20 | 22 | Maarten Plieger | "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>" |
21 | 17 | Maarten Plieger | } |
22 | 17 | Maarten Plieger | </code></pre> |
23 | 17 | Maarten Plieger | |
24 | 28 | Maarten Plieger | *2). List all tokens available for the current user by using "listtokens":* |
25 | 33 | Maarten Plieger | * https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=listtokens |
26 | 1 | Maarten Plieger | |
27 | 1 | Maarten Plieger | *Service endpoints which can be used with a token:* |
28 | 1 | Maarten Plieger | * WMS can be used via <pre>https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?</pre> |
29 | 1 | Maarten Plieger | * WCS can be used via <pre>https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?</pre> |
30 | 1 | Maarten Plieger | * WPS can be used via <pre>https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?</pre> |
31 | 1 | Maarten Plieger | * File download requests and OpenDAP requests from the basket can be done via <pre>https://dev.climate4impact.eu/impactportal/DAP/<accesstoken>/<userid>/<file>"</pre> |
32 | 29 | Maarten Plieger | * Basket requests can be done via <pre>https://dev.climate4impact.eu/impactportal/basket/<accesstoken>?&service=basket&request=getoverview&</pre> |
33 | 39 | Maarten Plieger | |
34 | 41 | Maarten Plieger | *3). Crash course token generation:* |
35 | 39 | Maarten Plieger | # Login with a google account at https://dev.climate4impact.eu/impactportal/account/OAuth2.jsp |
36 | 40 | Maarten Plieger | # Generate a new token via https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken |
37 | 39 | Maarten Plieger | # Copy paste the accesstoken value in your URL |
38 | 1 | Maarten Plieger | |
39 | 50 | Ernst de Vreede | h2. CLIP-C Access token API - service "clipcaccount" |
40 | 50 | Ernst de Vreede | |
41 | 51 | Ernst de Vreede | This is not a public API, but a machine to machine service end point, which the CLIPC server can use to request tokens onbehalfof the logged-in user. |
42 | 51 | Ernst de Vreede | This token then enables theCLIPC server to |
43 | 51 | Ernst de Vreede | |
44 | 50 | Ernst de Vreede | *1). Generate a token for the current user which can be used to access C4I services by using "generatetoken":* |
45 | 50 | Ernst de Vreede | * https://dev.climate4impact.eu/impactportal/tokenapi?service=clipcaccount&request=generatetoken&userid=<userid> |
46 | 50 | Ernst de Vreede | |
47 | 50 | Ernst de Vreede | The token has the following structure: |
48 | 50 | Ernst de Vreede | <pre><code class="json"> |
49 | 50 | Ernst de Vreede | { |
50 | 50 | Ernst de Vreede | "creationdate" : "2015-10-23T09:50:05Z", |
51 | 50 | Ernst de Vreede | "notafter" : "2015-10-30T09:50:05Z", |
52 | 50 | Ernst de Vreede | "notbefore" : "2015-10-23T09:50:05Z", |
53 | 50 | Ernst de Vreede | "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>", |
54 | 50 | Ernst de Vreede | "token" : "daad2944-1def-4bee-b032-b08048f88302", |
55 | 50 | Ernst de Vreede | "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>" |
56 | 50 | Ernst de Vreede | } |
57 | 50 | Ernst de Vreede | </code></pre> |
58 | 50 | Ernst de Vreede | |
59 | 5 | Maarten Plieger | h2. Basket requests |
60 | 1 | Maarten Plieger | |
61 | 2 | Maarten Plieger | * Get overview of all files in the users basket: service=basket&request=getoverview, |
62 | 33 | Maarten Plieger | ** e.g. <pre>https://dev.climate4impact.eu/impactportal/basket?&service=basket&request=getoverview&</pre> |
63 | 33 | Maarten Plieger | ** For commandline access with an access token: <pre>https://dev.climate4impact.eu/impactportal/basket/<accesstoken>/?&service=basket&request=getoverview&</pre> |
64 | 7 | Maarten Plieger | |
65 | 14 | Maarten Plieger | The result is a json file with nested childs representing te directory structure at C4I |
66 | 10 | Maarten Plieger | <pre><code class="json"> |
67 | 12 | Maarten Plieger | { "browser" : "/impactportal/data/catalogbrowser.jsp?", |
68 | 12 | Maarten Plieger | "children" : [ { "children" : [ ], |
69 | 12 | Maarten Plieger | "date" : "", |
70 | 12 | Maarten Plieger | "expanded" : true, |
71 | 12 | Maarten Plieger | "iconCls" : "typeFolder", |
72 | 12 | Maarten Plieger | "leaf" : false, |
73 | 12 | Maarten Plieger | "text" : "Remote data" |
74 | 1 | Maarten Plieger | }, |
75 | 13 | Maarten Plieger | { "children" : [ { "dapurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc", |
76 | 12 | Maarten Plieger | "date" : "2014-06-06 14:31:23Z", |
77 | 12 | Maarten Plieger | "filesize" : "611.573K", |
78 | 1 | Maarten Plieger | "hasdap" : true, |
79 | 12 | Maarten Plieger | "hashttp" : true, |
80 | 13 | Maarten Plieger | "httpurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc", |
81 | 12 | Maarten Plieger | "iconCls" : "typeOF", |
82 | 12 | Maarten Plieger | "id" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc", |
83 | 12 | Maarten Plieger | "index" : 1, |
84 | 12 | Maarten Plieger | "leaf" : true, |
85 | 1 | Maarten Plieger | "text" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc", |
86 | 12 | Maarten Plieger | "type" : "file" |
87 | 12 | Maarten Plieger | }, |
88 | 13 | Maarten Plieger | { "dapurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc", |
89 | 1 | Maarten Plieger | "date" : "2015-10-23 09:37:33Z", |
90 | 12 | Maarten Plieger | "filesize" : "47.867M", |
91 | 12 | Maarten Plieger | "hasdap" : true, |
92 | 12 | Maarten Plieger | "hashttp" : true, |
93 | 13 | Maarten Plieger | "httpurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc", |
94 | 12 | Maarten Plieger | "iconCls" : "typeOF", |
95 | 12 | Maarten Plieger | "id" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc", |
96 | 12 | Maarten Plieger | "index" : 2, |
97 | 12 | Maarten Plieger | "leaf" : true, |
98 | 12 | Maarten Plieger | "text" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc", |
99 | 12 | Maarten Plieger | "type" : "file" |
100 | 12 | Maarten Plieger | } |
101 | 12 | Maarten Plieger | ], |
102 | 12 | Maarten Plieger | "date" : "", |
103 | 12 | Maarten Plieger | "expanded" : true, |
104 | 12 | Maarten Plieger | "iconCls" : "typeFolder", |
105 | 1 | Maarten Plieger | "leaf" : false, |
106 | 12 | Maarten Plieger | "text" : "My data" |
107 | 12 | Maarten Plieger | } |
108 | 12 | Maarten Plieger | ], |
109 | 12 | Maarten Plieger | "leaf" : false, |
110 | 13 | Maarten Plieger | "text" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<openid>", |
111 | 11 | Maarten Plieger | "viewer" : "/impactportal/data/datasetviewer.jsp?" |
112 | 11 | Maarten Plieger | } |
113 | 10 | Maarten Plieger | </code> |
114 | 9 | Maarten Plieger | </pre> |
115 | 30 | Maarten Plieger | |
116 | 30 | Maarten Plieger | h2. WPS Requests |
117 | 30 | Maarten Plieger | |
118 | 36 | Maarten Plieger | Climate4impact uses PyWPS, see http://pywps.wald.intevation.org/documentation/course/ for more information |
119 | 36 | Maarten Plieger | |
120 | 36 | Maarten Plieger | * WPS can be used via <pre>https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?</pre> |
121 | 36 | Maarten Plieger | |
122 | 44 | Maarten Plieger | |
123 | 44 | Maarten Plieger | |
124 | 44 | Maarten Plieger | *Example Python WPS client on WPS Process "binaryoperatoronnumbers"* |
125 | 36 | Maarten Plieger | |
126 | 46 | Maarten Plieger | The following Python example behaves like a minimal WPS client, using C4I services to perform analysis. The example below makes use of the binaryoperatorfornumbers process, which enables operations on two numbers. The process needs five seconds to run, listing progress in percentages. This is done to ease progress monitoring developments. The process is executed asynchronously and is executed with the GET method. PyWPS also supports POST, this can be used for posting bigger inputs. This is described here: [[WPS]] |
127 | 46 | Maarten Plieger | |
128 | 46 | Maarten Plieger | The process has three inputs: |
129 | 34 | Maarten Plieger | # inputa - a float number |
130 | 34 | Maarten Plieger | # inputb - a float number |
131 | 34 | Maarten Plieger | # operation - Can be "add", "substract", "divide" or "multiply" |
132 | 35 | Maarten Plieger | |
133 | 34 | Maarten Plieger | |
134 | 30 | Maarten Plieger | <pre><code class="python"> |
135 | 30 | Maarten Plieger | #!/bin/python |
136 | 30 | Maarten Plieger | |
137 | 30 | Maarten Plieger | """ |
138 | 30 | Maarten Plieger | Obtain an access token: |
139 | 33 | Maarten Plieger | - Login with a google account at https://dev.climate4impact.eu/impactportal/account/OAuth2.jsp |
140 | 30 | Maarten Plieger | - Generate via https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken |
141 | 32 | Maarten Plieger | - Copy paste the accesstoken value in the "token" variable below. |
142 | 31 | Maarten Plieger | """ |
143 | 30 | Maarten Plieger | |
144 | 30 | Maarten Plieger | token="<access_token>" |
145 | 30 | Maarten Plieger | |
146 | 30 | Maarten Plieger | |
147 | 33 | Maarten Plieger | wpsendpoint="https://dev.climate4impact.eu/impactportal/WPS/"+token+"/?" |
148 | 30 | Maarten Plieger | |
149 | 30 | Maarten Plieger | inputa=11; |
150 | 30 | Maarten Plieger | inputb=22; |
151 | 30 | Maarten Plieger | operator="divide" # Any of add, substract, multiply, divide |
152 | 30 | Maarten Plieger | |
153 | 30 | Maarten Plieger | print "Calculating ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"]" |
154 | 30 | Maarten Plieger | |
155 | 30 | Maarten Plieger | import xml.etree.ElementTree as et |
156 | 30 | Maarten Plieger | import urllib2 |
157 | 30 | Maarten Plieger | import time |
158 | 30 | Maarten Plieger | |
159 | 30 | Maarten Plieger | """ Parses URL to XML tree """ |
160 | 30 | Maarten Plieger | def getandparsexml(url): |
161 | 30 | Maarten Plieger | response = urllib2.urlopen(url) |
162 | 30 | Maarten Plieger | html = response.read() |
163 | 30 | Maarten Plieger | #print html |
164 | 30 | Maarten Plieger | return et.fromstring(html) |
165 | 30 | Maarten Plieger | |
166 | 30 | Maarten Plieger | # Execute the ultimatequestionprocess aynchronousely and obtain the statusLocation for progress |
167 | 30 | Maarten Plieger | datainputs = "inputa="+str(inputa)+";inputb="+str(inputb)+";operator="+str(operator) |
168 | 30 | Maarten Plieger | root = getandparsexml(wpsendpoint+"service=WPS&version=1.0.0&request=execute&storeExecuteResponse=true&status=true&identifier=binaryoperatorfornumbers&datainputs="+datainputs) |
169 | 30 | Maarten Plieger | monitorURL = root.attrib['statusLocation'] |
170 | 30 | Maarten Plieger | |
171 | 30 | Maarten Plieger | print "Process executed checking " +monitorURL |
172 | 30 | Maarten Plieger | |
173 | 30 | Maarten Plieger | # Get starttime of process, sometimes the monitorURL is not immediately available. Iterate until a valid creationtime is available would be good practice. |
174 | 30 | Maarten Plieger | numberOfTries=10 |
175 | 30 | Maarten Plieger | while numberOfTries>0: |
176 | 30 | Maarten Plieger | numberOfTries = numberOfTries-1 |
177 | 30 | Maarten Plieger | try: |
178 | 30 | Maarten Plieger | root = getandparsexml(monitorURL) |
179 | 30 | Maarten Plieger | numberOfTries = -1 |
180 | 30 | Maarten Plieger | print "Process started at "+str(root[1].attrib['creationTime']) |
181 | 30 | Maarten Plieger | except: |
182 | 30 | Maarten Plieger | time.sleep(1) |
183 | 30 | Maarten Plieger | pass |
184 | 30 | Maarten Plieger | |
185 | 30 | Maarten Plieger | # Monitor progress, polling the statuslocation frequently |
186 | 30 | Maarten Plieger | percentage=0 |
187 | 30 | Maarten Plieger | while percentage<100: |
188 | 30 | Maarten Plieger | root = getandparsexml(monitorURL) |
189 | 30 | Maarten Plieger | try: |
190 | 30 | Maarten Plieger | percentage = float(root[1][0].attrib['percentCompleted']) |
191 | 30 | Maarten Plieger | except: |
192 | 30 | Maarten Plieger | percentage=100.0 |
193 | 30 | Maarten Plieger | pass |
194 | 30 | Maarten Plieger | print "Progress: "+str(percentage)+ " complete." |
195 | 30 | Maarten Plieger | time.sleep(1) |
196 | 30 | Maarten Plieger | |
197 | 30 | Maarten Plieger | # Obtain the answer |
198 | 30 | Maarten Plieger | root = getandparsexml(monitorURL) |
199 | 30 | Maarten Plieger | try: |
200 | 30 | Maarten Plieger | print "The answer for ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"] is "+(root[2][0][2][0].text) |
201 | 30 | Maarten Plieger | except: |
202 | 30 | Maarten Plieger | print "Processing failed: "+root[1][0][0][0][0].text |
203 | 30 | Maarten Plieger | pass |
204 | 30 | Maarten Plieger | </code> |
205 | 30 | Maarten Plieger | </pre> |
206 | 42 | Maarten Plieger | |
207 | 42 | Maarten Plieger | The result is: |
208 | 42 | Maarten Plieger | <pre> |
209 | 42 | Maarten Plieger | (pyvirtenv)c4m@climate4impact:~$ python c4i_testwps_binaryoperator.py |
210 | 42 | Maarten Plieger | Calculating [11 divide 22] |
211 | 42 | Maarten Plieger | Process executed checking http://climate4impact.eu/impactportal/WPS?OUTPUT=/pywps-144559994232.xml |
212 | 42 | Maarten Plieger | Process started at 2015-10-23T11:32:23Z |
213 | 42 | Maarten Plieger | Progress: 20.0 complete. |
214 | 42 | Maarten Plieger | Progress: 40.0 complete. |
215 | 42 | Maarten Plieger | Progress: 60.0 complete. |
216 | 42 | Maarten Plieger | Progress: 80.0 complete. |
217 | 47 | Maarten Plieger | Progress: 100.0 complete. |
218 | 47 | Maarten Plieger | The answer for [11 divide 22] is 0.5 |
219 | 47 | Maarten Plieger | (pyvirtenv)c4m@climate4impact:~$ |
220 | 47 | Maarten Plieger | </pre> |
221 | 52 | Maarten Plieger | |
222 | 52 | Maarten Plieger | h3. WPS JobList requests |
223 | 52 | Maarten Plieger | |
224 | 52 | Maarten Plieger | All submitted jobs are captured and stored in a per user joblist. The joblist can be obtained via: |
225 | 52 | Maarten Plieger | |
226 | 52 | Maarten Plieger | * https://dev.climate4impact.eu/impactportal/ImpactService?service=processor&request=getProcessorStatusOverview |
227 | 52 | Maarten Plieger | |
228 | 52 | Maarten Plieger | A json with array called jobs is returned: |
229 | 52 | Maarten Plieger | |
230 | 52 | Maarten Plieger | <pre> |
231 | 52 | Maarten Plieger | { |
232 | 52 | Maarten Plieger | jobs:[{ |
233 | 52 | Maarten Plieger | "wpspostdata": ..., |
234 | 52 | Maarten Plieger | "creationdate": "2016-03-30 16:57:03Z", |
235 | 52 | Maarten Plieger | "wpsid": "clipc_combine_execute", |
236 | 52 | Maarten Plieger | "processid": "pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml", |
237 | 52 | Maarten Plieger | "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml", |
238 | 52 | Maarten Plieger | "progress": "ready", // Ready indicated completed job |
239 | 52 | Maarten Plieger | "error": false, |
240 | 52 | Maarten Plieger | "status": "ready" // Ready, running, failed |
241 | 52 | Maarten Plieger | },{ |
242 | 52 | Maarten Plieger | "wpspostdata": ..., |
243 | 52 | Maarten Plieger | "creationdate": "2016-03-30 17:01:36Z", |
244 | 52 | Maarten Plieger | "wpsid": "wps_simple_indice", |
245 | 52 | Maarten Plieger | "processid": "pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml", |
246 | 52 | Maarten Plieger | "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml", |
247 | 52 | Maarten Plieger | "progress": "0 %", //Percentage completed |
248 | 52 | Maarten Plieger | "error": false, |
249 | 52 | Maarten Plieger | "status": "running" // Ready, running, failed |
250 | 52 | Maarten Plieger | }] |
251 | 52 | Maarten Plieger | } |
252 | 52 | Maarten Plieger | </pre> |
253 | 53 | Maarten Plieger | |
254 | 53 | Maarten Plieger | A job can be removed from the list by doing: |
255 | 53 | Maarten Plieger | |
256 | 53 | Maarten Plieger | * https://dev.climate4impact.eu/impactportal/ImpactService?service=processor&request=removeFromList&id=pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml |