Project

General

Profile

API » History » Version 67

Maarten Plieger, 04/14/2016 11:49 AM

1 1 Maarten Plieger
h1. API
2 1 Maarten Plieger
3 48 Maarten Plieger
h2. Clipc combine process
4 48 Maarten Plieger
5 67 Maarten Plieger
* PyWPS process to combine two netcdf files: [[Clipc Combine process]]
6 67 Maarten Plieger
* The clipc combine project page (wiki and repo) located under https://dev.knmi.nl/projects/clipccombine/
7 48 Maarten Plieger
8 28 Maarten Plieger
h2. Access token API - service "account"
9 1 Maarten Plieger
10 28 Maarten Plieger
*1). Generate a token for the current user which can be used to access C4I services by using "generatetoken":*
11 33 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
12 1 Maarten Plieger
13 17 Maarten Plieger
The token has the following structure:
14 17 Maarten Plieger
<pre><code class="json">
15 18 Maarten Plieger
{ 
16 18 Maarten Plieger
  "creationdate" : "2015-10-23T09:50:05Z",
17 17 Maarten Plieger
  "notafter" : "2015-10-30T09:50:05Z",
18 17 Maarten Plieger
  "notbefore" : "2015-10-23T09:50:05Z",
19 22 Maarten Plieger
  "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>",
20 24 Maarten Plieger
  "token" : "daad2944-1def-4bee-b032-b08048f88302", 
21 22 Maarten Plieger
  "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>"
22 17 Maarten Plieger
}
23 17 Maarten Plieger
</code></pre>
24 17 Maarten Plieger
25 28 Maarten Plieger
*2). List all tokens available for the current user by using "listtokens":*
26 33 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=listtokens
27 1 Maarten Plieger
28 1 Maarten Plieger
*Service endpoints which can be used with a token:*
29 1 Maarten Plieger
* WMS can be used via <pre>https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?</pre>
30 1 Maarten Plieger
* WCS can be used via <pre>https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?</pre>
31 1 Maarten Plieger
* WPS can be used via <pre>https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?</pre>
32 1 Maarten Plieger
* File download requests and OpenDAP requests from the basket can be done via <pre>https://dev.climate4impact.eu/impactportal/DAP/<accesstoken>/<userid>/<file>"</pre>
33 29 Maarten Plieger
* Basket requests can be done via <pre>https://dev.climate4impact.eu/impactportal/basket/<accesstoken>?&service=basket&request=getoverview&</pre>
34 55 Maarten Plieger
* ImpactService requests can be done via <pre>https://dev.climate4impact.eu/impactportal/ImpactService/<accesstoken>/?</pre>
35 39 Maarten Plieger
36 41 Maarten Plieger
*3). Crash course token generation:*
37 39 Maarten Plieger
# Login with a google account at https://dev.climate4impact.eu/impactportal/account/OAuth2.jsp 
38 40 Maarten Plieger
# Generate a new token via https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
39 39 Maarten Plieger
# Copy paste the accesstoken value in your URL
40 1 Maarten Plieger
41 50 Ernst de Vreede
h2. CLIP-C Access token API - service "clipcaccount"
42 50 Ernst de Vreede
43 51 Ernst de Vreede
This is not a public API, but a machine to machine service end point, which the CLIPC server can use to request tokens onbehalfof the logged-in user.
44 51 Ernst de Vreede
This token then enables theCLIPC server to
45 51 Ernst de Vreede
 
46 50 Ernst de Vreede
*1). Generate a token for the current user which can be used to access C4I services by using "generatetoken":*
47 50 Ernst de Vreede
* https://dev.climate4impact.eu/impactportal/tokenapi?service=clipcaccount&request=generatetoken&userid=<userid>
48 50 Ernst de Vreede
49 50 Ernst de Vreede
The token has the following structure:
50 50 Ernst de Vreede
<pre><code class="json">
51 50 Ernst de Vreede
{ 
52 50 Ernst de Vreede
  "creationdate" : "2015-10-23T09:50:05Z",
53 50 Ernst de Vreede
  "notafter" : "2015-10-30T09:50:05Z",
54 50 Ernst de Vreede
  "notbefore" : "2015-10-23T09:50:05Z",
55 50 Ernst de Vreede
  "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>",
56 50 Ernst de Vreede
  "token" : "daad2944-1def-4bee-b032-b08048f88302", 
57 50 Ernst de Vreede
  "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>"
58 50 Ernst de Vreede
}
59 50 Ernst de Vreede
</code></pre>
60 50 Ernst de Vreede
61 5 Maarten Plieger
h2. Basket requests
62 1 Maarten Plieger
63 2 Maarten Plieger
* Get overview of all files in the users basket: service=basket&request=getoverview, 
64 33 Maarten Plieger
** e.g. <pre>https://dev.climate4impact.eu/impactportal/basket?&service=basket&request=getoverview&</pre>
65 33 Maarten Plieger
** For commandline access with an access token: <pre>https://dev.climate4impact.eu/impactportal/basket/<accesstoken>/?&service=basket&request=getoverview&</pre>
66 7 Maarten Plieger
67 14 Maarten Plieger
The result is a json file with nested childs representing te directory structure at C4I
68 10 Maarten Plieger
<pre><code class="json">
69 12 Maarten Plieger
{ "browser" : "/impactportal/data/catalogbrowser.jsp?",
70 12 Maarten Plieger
  "children" : [ { "children" : [  ],
71 12 Maarten Plieger
        "date" : "",
72 12 Maarten Plieger
        "expanded" : true,
73 12 Maarten Plieger
        "iconCls" : "typeFolder",
74 12 Maarten Plieger
        "leaf" : false,
75 12 Maarten Plieger
        "text" : "Remote data"
76 1 Maarten Plieger
      },
77 13 Maarten Plieger
      { "children" : [ { "dapurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
78 12 Maarten Plieger
              "date" : "2014-06-06 14:31:23Z",
79 12 Maarten Plieger
              "filesize" : "611.573K",
80 1 Maarten Plieger
              "hasdap" : true,
81 12 Maarten Plieger
              "hashttp" : true,
82 13 Maarten Plieger
              "httpurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
83 12 Maarten Plieger
              "iconCls" : "typeOF",
84 12 Maarten Plieger
              "id" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
85 12 Maarten Plieger
              "index" : 1,
86 12 Maarten Plieger
              "leaf" : true,
87 1 Maarten Plieger
              "text" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
88 12 Maarten Plieger
              "type" : "file"
89 12 Maarten Plieger
            },
90 13 Maarten Plieger
            { "dapurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
91 1 Maarten Plieger
              "date" : "2015-10-23 09:37:33Z",
92 12 Maarten Plieger
              "filesize" : "47.867M",
93 12 Maarten Plieger
              "hasdap" : true,
94 12 Maarten Plieger
              "hashttp" : true,
95 13 Maarten Plieger
              "httpurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
96 12 Maarten Plieger
              "iconCls" : "typeOF",
97 12 Maarten Plieger
              "id" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
98 12 Maarten Plieger
              "index" : 2,
99 12 Maarten Plieger
              "leaf" : true,
100 12 Maarten Plieger
              "text" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
101 12 Maarten Plieger
              "type" : "file"
102 12 Maarten Plieger
            }
103 12 Maarten Plieger
          ],
104 12 Maarten Plieger
        "date" : "",
105 12 Maarten Plieger
        "expanded" : true,
106 12 Maarten Plieger
        "iconCls" : "typeFolder",
107 1 Maarten Plieger
        "leaf" : false,
108 12 Maarten Plieger
        "text" : "My data"
109 12 Maarten Plieger
      }
110 12 Maarten Plieger
    ],
111 12 Maarten Plieger
  "leaf" : false,
112 13 Maarten Plieger
  "text" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<openid>",
113 11 Maarten Plieger
  "viewer" : "/impactportal/data/datasetviewer.jsp?"
114 11 Maarten Plieger
}
115 10 Maarten Plieger
</code>
116 56 Maarten Plieger
117 57 Maarten Plieger
</pre>
118 57 Maarten Plieger
119 57 Maarten Plieger
120 56 Maarten Plieger
*Remove a file from the basket*
121 56 Maarten Plieger
122 57 Maarten Plieger
<pre>
123 61 Maarten Plieger
GET request:
124 59 Maarten Plieger
https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?&service=basket&request=removeFromList&id[]=<id from basket request>
125 57 Maarten Plieger
</pre>
126 56 Maarten Plieger
e.g.:
127 57 Maarten Plieger
<pre>
128 60 Maarten Plieger
https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?&service=basket&request=removeFromList&id[]=INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc
129 58 Maarten Plieger
</pre>
130 61 Maarten Plieger
131 63 Maarten Plieger
To remove multiple files at once do:
132 63 Maarten Plieger
<pre>
133 63 Maarten Plieger
https://dev.climate4impact.eu/impactportal/ImpactService?&service=basket&request=removeFromList&id[]=WPS_clipc_combine_execute_20160222T150058Z&id[]=WPS_clipc_combine_execute_20160222T150045Z&id[]=WPS_clipc_combine_execute_20160222T145836Z&id[]=WPS_clipc_combine_execute_20160222T145649Z&id[]=WPS_clipc_combine_execute_20160222T145633Z&id[]=WPS_clipc_combine_execute_20160222T145449Z&id[]=WPS_clipc_combine_execute_20160222T145323Z&id[]=WPS_clipc_combine_execute_20160222T145308Z&id[]=WPS_clipc_combine_execute_20160222T145230Z&id[]=WPS_clipc_combine_execute_20160222T145030Z&id[]=WPS_clipc_combine_execute_20160222T144854Z&id[]=WPS_clipc_combine_execute_20160222T144852Z&id[]=WPS_clipc_combine_execute_20160222T144704Z&id[]=WPS_clipc_combine_execute_20160222T144549Z
134 63 Maarten Plieger
</pre>
135 63 Maarten Plieger
136 63 Maarten Plieger
137 61 Maarten Plieger
*Add a file to the basket*
138 61 Maarten Plieger
139 61 Maarten Plieger
<pre>
140 65 Maarten Plieger
GET request to https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?service=basket&mode=add&id=<file basename>&opendap=<opendap url>
141 1 Maarten Plieger
</pre>
142 64 Maarten Plieger
E.g add 
143 66 Maarten Plieger
144 64 Maarten Plieger
* opendap: http://thredds.met.no/thredds/dodsC/osisaf/met.no/ice/conc/2016/04/ice_conc_sh_polstere-100_multi_201604091200.nc
145 64 Maarten Plieger
* id: ice_conc_sh_polstere-100_multi_201604091200.nc
146 64 Maarten Plieger
147 64 Maarten Plieger
Example:
148 61 Maarten Plieger
<pre>
149 65 Maarten Plieger
https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?service=basket&mode=add&id=ice_conc_sh_polstere-100_multi_201604091200.nc&OPENDAP=http%3A%2F%2Fthredds.met.no%2Fthredds%2FdodsC%2Fosisaf%2Fmet.no%2Fice%2Fconc%2F2016%2F04%2Fice_conc_sh_polstere-100_multi_201604091200.nc
150 58 Maarten Plieger
</pre>
151 58 Maarten Plieger
152 58 Maarten Plieger
153 9 Maarten Plieger
154 30 Maarten Plieger
155 30 Maarten Plieger
h2. WPS Requests
156 30 Maarten Plieger
157 36 Maarten Plieger
Climate4impact uses PyWPS, see http://pywps.wald.intevation.org/documentation/course/ for more information
158 36 Maarten Plieger
159 36 Maarten Plieger
* WPS can be used via <pre>https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?</pre>
160 36 Maarten Plieger
161 44 Maarten Plieger
162 44 Maarten Plieger
163 44 Maarten Plieger
*Example Python WPS client on WPS Process "binaryoperatoronnumbers"*
164 36 Maarten Plieger
165 46 Maarten Plieger
The following Python example behaves like a minimal WPS client, using C4I services to perform analysis. The example below makes use of the binaryoperatorfornumbers process, which enables operations on two numbers. The process needs five seconds to run, listing progress in percentages. This is done to ease progress monitoring developments. The process is executed asynchronously and is executed with the GET method. PyWPS also supports POST, this can be used for posting bigger inputs. This is described here: [[WPS]] 
166 46 Maarten Plieger
167 46 Maarten Plieger
The process has three inputs:
168 34 Maarten Plieger
# inputa - a float number
169 34 Maarten Plieger
# inputb - a float number
170 34 Maarten Plieger
# operation -  Can be "add", "substract", "divide" or "multiply"
171 35 Maarten Plieger
172 34 Maarten Plieger
173 30 Maarten Plieger
<pre><code class="python">
174 30 Maarten Plieger
#!/bin/python
175 30 Maarten Plieger
176 30 Maarten Plieger
"""
177 30 Maarten Plieger
Obtain an access token:
178 33 Maarten Plieger
- Login with a google account at https://dev.climate4impact.eu/impactportal/account/OAuth2.jsp 
179 30 Maarten Plieger
- Generate via https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
180 32 Maarten Plieger
- Copy paste the accesstoken value in the "token" variable below.
181 31 Maarten Plieger
"""
182 30 Maarten Plieger
183 30 Maarten Plieger
token="<access_token>" 
184 30 Maarten Plieger
 
185 30 Maarten Plieger
186 33 Maarten Plieger
wpsendpoint="https://dev.climate4impact.eu/impactportal/WPS/"+token+"/?"
187 30 Maarten Plieger
188 30 Maarten Plieger
inputa=11;
189 30 Maarten Plieger
inputb=22;
190 30 Maarten Plieger
operator="divide" # Any of add, substract, multiply, divide
191 30 Maarten Plieger
192 30 Maarten Plieger
print "Calculating ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"]"
193 30 Maarten Plieger
194 30 Maarten Plieger
import xml.etree.ElementTree as et
195 30 Maarten Plieger
import urllib2
196 30 Maarten Plieger
import time
197 30 Maarten Plieger
198 30 Maarten Plieger
""" Parses URL to XML tree """
199 30 Maarten Plieger
def getandparsexml(url):
200 30 Maarten Plieger
  response = urllib2.urlopen(url)
201 30 Maarten Plieger
  html = response.read()
202 30 Maarten Plieger
  #print html
203 30 Maarten Plieger
  return et.fromstring(html)
204 30 Maarten Plieger
  
205 30 Maarten Plieger
# Execute the ultimatequestionprocess aynchronousely and obtain the statusLocation for progress
206 30 Maarten Plieger
datainputs = "inputa="+str(inputa)+";inputb="+str(inputb)+";operator="+str(operator)
207 30 Maarten Plieger
root = getandparsexml(wpsendpoint+"service=WPS&version=1.0.0&request=execute&storeExecuteResponse=true&status=true&identifier=binaryoperatorfornumbers&datainputs="+datainputs)    
208 30 Maarten Plieger
monitorURL = root.attrib['statusLocation']
209 30 Maarten Plieger
210 30 Maarten Plieger
print "Process executed checking " +monitorURL
211 30 Maarten Plieger
212 30 Maarten Plieger
# Get starttime of process, sometimes the monitorURL is not immediately available. Iterate until a valid creationtime is available would be good practice.
213 30 Maarten Plieger
numberOfTries=10
214 30 Maarten Plieger
while numberOfTries>0:
215 30 Maarten Plieger
  numberOfTries = numberOfTries-1 
216 30 Maarten Plieger
  try:
217 30 Maarten Plieger
    root = getandparsexml(monitorURL)
218 30 Maarten Plieger
    numberOfTries = -1
219 30 Maarten Plieger
    print "Process started at "+str(root[1].attrib['creationTime'])
220 30 Maarten Plieger
  except:
221 30 Maarten Plieger
    time.sleep(1)
222 30 Maarten Plieger
    pass
223 30 Maarten Plieger
224 30 Maarten Plieger
# Monitor progress, polling the statuslocation frequently
225 30 Maarten Plieger
percentage=0
226 30 Maarten Plieger
while percentage<100:
227 30 Maarten Plieger
  root = getandparsexml(monitorURL)
228 30 Maarten Plieger
  try:
229 30 Maarten Plieger
    percentage = float(root[1][0].attrib['percentCompleted'])
230 30 Maarten Plieger
  except:
231 30 Maarten Plieger
    percentage=100.0
232 30 Maarten Plieger
    pass
233 30 Maarten Plieger
  print "Progress: "+str(percentage)+ " complete."
234 30 Maarten Plieger
  time.sleep(1)
235 30 Maarten Plieger
236 30 Maarten Plieger
# Obtain the answer  
237 30 Maarten Plieger
root = getandparsexml(monitorURL)
238 30 Maarten Plieger
try:
239 30 Maarten Plieger
  print "The answer for ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"] is "+(root[2][0][2][0].text)
240 30 Maarten Plieger
except:
241 30 Maarten Plieger
  print "Processing failed: "+root[1][0][0][0][0].text
242 30 Maarten Plieger
  pass
243 30 Maarten Plieger
</code>
244 30 Maarten Plieger
</pre>
245 42 Maarten Plieger
246 42 Maarten Plieger
The result is:
247 42 Maarten Plieger
<pre>
248 42 Maarten Plieger
(pyvirtenv)c4m@climate4impact:~$ python c4i_testwps_binaryoperator.py 
249 42 Maarten Plieger
Calculating [11 divide 22]
250 42 Maarten Plieger
Process executed checking http://climate4impact.eu/impactportal/WPS?OUTPUT=/pywps-144559994232.xml
251 42 Maarten Plieger
Process started at 2015-10-23T11:32:23Z
252 42 Maarten Plieger
Progress: 20.0 complete.
253 42 Maarten Plieger
Progress: 40.0 complete.
254 42 Maarten Plieger
Progress: 60.0 complete.
255 42 Maarten Plieger
Progress: 80.0 complete.
256 47 Maarten Plieger
Progress: 100.0 complete.
257 47 Maarten Plieger
The answer for [11 divide 22] is 0.5
258 47 Maarten Plieger
(pyvirtenv)c4m@climate4impact:~$ 
259 47 Maarten Plieger
</pre>
260 52 Maarten Plieger
261 52 Maarten Plieger
h3. WPS JobList requests
262 52 Maarten Plieger
263 52 Maarten Plieger
All submitted jobs are captured and stored in a per user joblist. The joblist can be obtained via:
264 52 Maarten Plieger
265 52 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/ImpactService?service=processor&request=getProcessorStatusOverview
266 52 Maarten Plieger
267 52 Maarten Plieger
A json with array called jobs is returned:
268 52 Maarten Plieger
269 52 Maarten Plieger
<pre>
270 52 Maarten Plieger
{
271 52 Maarten Plieger
  jobs:[{
272 52 Maarten Plieger
    "wpspostdata": ...,
273 52 Maarten Plieger
    "creationdate": "2016-03-30 16:57:03Z",
274 52 Maarten Plieger
    "wpsid": "clipc_combine_execute",
275 52 Maarten Plieger
    "processid": "pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml",
276 52 Maarten Plieger
    "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml",
277 52 Maarten Plieger
    "progress": "ready", // Ready indicated completed job
278 52 Maarten Plieger
    "error": false,
279 52 Maarten Plieger
    "status": "ready" // Ready, running, failed
280 52 Maarten Plieger
  },{
281 52 Maarten Plieger
    "wpspostdata":  ...,
282 52 Maarten Plieger
    "creationdate": "2016-03-30 17:01:36Z",
283 52 Maarten Plieger
    "wpsid": "wps_simple_indice",
284 52 Maarten Plieger
    "processid": "pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml",
285 52 Maarten Plieger
    "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml",
286 52 Maarten Plieger
    "progress": "0 %", //Percentage completed
287 52 Maarten Plieger
    "error": false,
288 52 Maarten Plieger
    "status": "running" // Ready, running, failed
289 52 Maarten Plieger
  }]
290 52 Maarten Plieger
}
291 52 Maarten Plieger
</pre>
292 53 Maarten Plieger
293 53 Maarten Plieger
A job can be removed from the list by doing:
294 53 Maarten Plieger
295 53 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/ImpactService?service=processor&request=removeFromList&id=pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml