Project

General

Profile

API » History » Version 70

Maarten Plieger, 09/08/2016 10:43 AM

1 1 Maarten Plieger
h1. API
2 1 Maarten Plieger
3 48 Maarten Plieger
h2. Clipc combine process
4 48 Maarten Plieger
5 67 Maarten Plieger
* PyWPS process to combine two netcdf files: [[Clipc Combine process]]
6 67 Maarten Plieger
* The clipc combine project page (wiki and repo) located under https://dev.knmi.nl/projects/clipccombine/
7 48 Maarten Plieger
8 68 Maarten Plieger
h2. Search GUI
9 68 Maarten Plieger
10 68 Maarten Plieger
* Check https://dev.knmi.nl/projects/impactportal/wiki/C4I_GUI_Components
11 68 Maarten Plieger
12 28 Maarten Plieger
h2. Access token API - service "account"
13 1 Maarten Plieger
14 28 Maarten Plieger
*1). Generate a token for the current user which can be used to access C4I services by using "generatetoken":*
15 33 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
16 1 Maarten Plieger
17 17 Maarten Plieger
The token has the following structure:
18 17 Maarten Plieger
<pre><code class="json">
19 18 Maarten Plieger
{ 
20 18 Maarten Plieger
  "creationdate" : "2015-10-23T09:50:05Z",
21 17 Maarten Plieger
  "notafter" : "2015-10-30T09:50:05Z",
22 17 Maarten Plieger
  "notbefore" : "2015-10-23T09:50:05Z",
23 22 Maarten Plieger
  "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>",
24 24 Maarten Plieger
  "token" : "daad2944-1def-4bee-b032-b08048f88302", 
25 22 Maarten Plieger
  "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>"
26 17 Maarten Plieger
}
27 17 Maarten Plieger
</code></pre>
28 17 Maarten Plieger
29 28 Maarten Plieger
*2). List all tokens available for the current user by using "listtokens":*
30 33 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=listtokens
31 1 Maarten Plieger
32 1 Maarten Plieger
*Service endpoints which can be used with a token:*
33 1 Maarten Plieger
* WMS can be used via <pre>https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?</pre>
34 1 Maarten Plieger
* WCS can be used via <pre>https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?</pre>
35 1 Maarten Plieger
* WPS can be used via <pre>https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?</pre>
36 1 Maarten Plieger
* File download requests and OpenDAP requests from the basket can be done via <pre>https://dev.climate4impact.eu/impactportal/DAP/<accesstoken>/<userid>/<file>"</pre>
37 29 Maarten Plieger
* Basket requests can be done via <pre>https://dev.climate4impact.eu/impactportal/basket/<accesstoken>?&service=basket&request=getoverview&</pre>
38 55 Maarten Plieger
* ImpactService requests can be done via <pre>https://dev.climate4impact.eu/impactportal/ImpactService/<accesstoken>/?</pre>
39 39 Maarten Plieger
40 41 Maarten Plieger
*3). Crash course token generation:*
41 39 Maarten Plieger
# Login with a google account at https://dev.climate4impact.eu/impactportal/account/OAuth2.jsp 
42 40 Maarten Plieger
# Generate a new token via https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
43 39 Maarten Plieger
# Copy paste the accesstoken value in your URL
44 1 Maarten Plieger
45 50 Ernst de Vreede
h2. CLIP-C Access token API - service "clipcaccount"
46 50 Ernst de Vreede
47 70 Maarten Plieger
The accesstoken API is a machine to machine service endpoint, the CLIPC server uses this to request tokens onbehalfof the logged-in user at the CLIPC portal.
48 51 Ernst de Vreede
This token then enables theCLIPC server to
49 51 Ernst de Vreede
 
50 50 Ernst de Vreede
*1). Generate a token for the current user which can be used to access C4I services by using "generatetoken":*
51 50 Ernst de Vreede
* https://dev.climate4impact.eu/impactportal/tokenapi?service=clipcaccount&request=generatetoken&userid=<userid>
52 50 Ernst de Vreede
53 50 Ernst de Vreede
The token has the following structure:
54 50 Ernst de Vreede
<pre><code class="json">
55 50 Ernst de Vreede
{ 
56 50 Ernst de Vreede
  "creationdate" : "2015-10-23T09:50:05Z",
57 50 Ernst de Vreede
  "notafter" : "2015-10-30T09:50:05Z",
58 50 Ernst de Vreede
  "notbefore" : "2015-10-23T09:50:05Z",
59 50 Ernst de Vreede
  "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>",
60 50 Ernst de Vreede
  "token" : "daad2944-1def-4bee-b032-b08048f88302", 
61 50 Ernst de Vreede
  "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>"
62 50 Ernst de Vreede
}
63 50 Ernst de Vreede
</code></pre>
64 50 Ernst de Vreede
65 5 Maarten Plieger
h2. Basket requests
66 1 Maarten Plieger
67 2 Maarten Plieger
* Get overview of all files in the users basket: service=basket&request=getoverview, 
68 33 Maarten Plieger
** e.g. <pre>https://dev.climate4impact.eu/impactportal/basket?&service=basket&request=getoverview&</pre>
69 33 Maarten Plieger
** For commandline access with an access token: <pre>https://dev.climate4impact.eu/impactportal/basket/<accesstoken>/?&service=basket&request=getoverview&</pre>
70 7 Maarten Plieger
71 14 Maarten Plieger
The result is a json file with nested childs representing te directory structure at C4I
72 10 Maarten Plieger
<pre><code class="json">
73 12 Maarten Plieger
{ "browser" : "/impactportal/data/catalogbrowser.jsp?",
74 12 Maarten Plieger
  "children" : [ { "children" : [  ],
75 12 Maarten Plieger
        "date" : "",
76 12 Maarten Plieger
        "expanded" : true,
77 12 Maarten Plieger
        "iconCls" : "typeFolder",
78 12 Maarten Plieger
        "leaf" : false,
79 12 Maarten Plieger
        "text" : "Remote data"
80 1 Maarten Plieger
      },
81 13 Maarten Plieger
      { "children" : [ { "dapurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
82 12 Maarten Plieger
              "date" : "2014-06-06 14:31:23Z",
83 12 Maarten Plieger
              "filesize" : "611.573K",
84 1 Maarten Plieger
              "hasdap" : true,
85 12 Maarten Plieger
              "hashttp" : true,
86 13 Maarten Plieger
              "httpurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
87 12 Maarten Plieger
              "iconCls" : "typeOF",
88 12 Maarten Plieger
              "id" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
89 12 Maarten Plieger
              "index" : 1,
90 12 Maarten Plieger
              "leaf" : true,
91 1 Maarten Plieger
              "text" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
92 12 Maarten Plieger
              "type" : "file"
93 12 Maarten Plieger
            },
94 13 Maarten Plieger
            { "dapurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
95 1 Maarten Plieger
              "date" : "2015-10-23 09:37:33Z",
96 12 Maarten Plieger
              "filesize" : "47.867M",
97 12 Maarten Plieger
              "hasdap" : true,
98 12 Maarten Plieger
              "hashttp" : true,
99 13 Maarten Plieger
              "httpurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
100 12 Maarten Plieger
              "iconCls" : "typeOF",
101 12 Maarten Plieger
              "id" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
102 12 Maarten Plieger
              "index" : 2,
103 12 Maarten Plieger
              "leaf" : true,
104 12 Maarten Plieger
              "text" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
105 12 Maarten Plieger
              "type" : "file"
106 12 Maarten Plieger
            }
107 12 Maarten Plieger
          ],
108 12 Maarten Plieger
        "date" : "",
109 12 Maarten Plieger
        "expanded" : true,
110 12 Maarten Plieger
        "iconCls" : "typeFolder",
111 1 Maarten Plieger
        "leaf" : false,
112 12 Maarten Plieger
        "text" : "My data"
113 12 Maarten Plieger
      }
114 12 Maarten Plieger
    ],
115 12 Maarten Plieger
  "leaf" : false,
116 13 Maarten Plieger
  "text" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<openid>",
117 11 Maarten Plieger
  "viewer" : "/impactportal/data/datasetviewer.jsp?"
118 11 Maarten Plieger
}
119 10 Maarten Plieger
</code>
120 56 Maarten Plieger
121 57 Maarten Plieger
</pre>
122 57 Maarten Plieger
123 57 Maarten Plieger
124 56 Maarten Plieger
*Remove a file from the basket*
125 56 Maarten Plieger
126 57 Maarten Plieger
<pre>
127 61 Maarten Plieger
GET request:
128 59 Maarten Plieger
https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?&service=basket&request=removeFromList&id[]=<id from basket request>
129 57 Maarten Plieger
</pre>
130 56 Maarten Plieger
e.g.:
131 57 Maarten Plieger
<pre>
132 60 Maarten Plieger
https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?&service=basket&request=removeFromList&id[]=INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc
133 58 Maarten Plieger
</pre>
134 61 Maarten Plieger
135 63 Maarten Plieger
To remove multiple files at once do:
136 63 Maarten Plieger
<pre>
137 63 Maarten Plieger
https://dev.climate4impact.eu/impactportal/ImpactService?&service=basket&request=removeFromList&id[]=WPS_clipc_combine_execute_20160222T150058Z&id[]=WPS_clipc_combine_execute_20160222T150045Z&id[]=WPS_clipc_combine_execute_20160222T145836Z&id[]=WPS_clipc_combine_execute_20160222T145649Z&id[]=WPS_clipc_combine_execute_20160222T145633Z&id[]=WPS_clipc_combine_execute_20160222T145449Z&id[]=WPS_clipc_combine_execute_20160222T145323Z&id[]=WPS_clipc_combine_execute_20160222T145308Z&id[]=WPS_clipc_combine_execute_20160222T145230Z&id[]=WPS_clipc_combine_execute_20160222T145030Z&id[]=WPS_clipc_combine_execute_20160222T144854Z&id[]=WPS_clipc_combine_execute_20160222T144852Z&id[]=WPS_clipc_combine_execute_20160222T144704Z&id[]=WPS_clipc_combine_execute_20160222T144549Z
138 63 Maarten Plieger
</pre>
139 63 Maarten Plieger
140 63 Maarten Plieger
141 61 Maarten Plieger
*Add a file to the basket*
142 61 Maarten Plieger
143 61 Maarten Plieger
<pre>
144 65 Maarten Plieger
GET request to https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?service=basket&mode=add&id=<file basename>&opendap=<opendap url>
145 1 Maarten Plieger
</pre>
146 64 Maarten Plieger
E.g add 
147 66 Maarten Plieger
148 64 Maarten Plieger
* opendap: http://thredds.met.no/thredds/dodsC/osisaf/met.no/ice/conc/2016/04/ice_conc_sh_polstere-100_multi_201604091200.nc
149 64 Maarten Plieger
* id: ice_conc_sh_polstere-100_multi_201604091200.nc
150 64 Maarten Plieger
151 64 Maarten Plieger
Example:
152 61 Maarten Plieger
<pre>
153 65 Maarten Plieger
https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?service=basket&mode=add&id=ice_conc_sh_polstere-100_multi_201604091200.nc&OPENDAP=http%3A%2F%2Fthredds.met.no%2Fthredds%2FdodsC%2Fosisaf%2Fmet.no%2Fice%2Fconc%2F2016%2F04%2Fice_conc_sh_polstere-100_multi_201604091200.nc
154 58 Maarten Plieger
</pre>
155 58 Maarten Plieger
156 58 Maarten Plieger
157 9 Maarten Plieger
158 30 Maarten Plieger
159 30 Maarten Plieger
h2. WPS Requests
160 30 Maarten Plieger
161 36 Maarten Plieger
Climate4impact uses PyWPS, see http://pywps.wald.intevation.org/documentation/course/ for more information
162 36 Maarten Plieger
163 36 Maarten Plieger
* WPS can be used via <pre>https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?</pre>
164 36 Maarten Plieger
165 44 Maarten Plieger
166 44 Maarten Plieger
167 44 Maarten Plieger
*Example Python WPS client on WPS Process "binaryoperatoronnumbers"*
168 36 Maarten Plieger
169 46 Maarten Plieger
The following Python example behaves like a minimal WPS client, using C4I services to perform analysis. The example below makes use of the binaryoperatorfornumbers process, which enables operations on two numbers. The process needs five seconds to run, listing progress in percentages. This is done to ease progress monitoring developments. The process is executed asynchronously and is executed with the GET method. PyWPS also supports POST, this can be used for posting bigger inputs. This is described here: [[WPS]] 
170 46 Maarten Plieger
171 46 Maarten Plieger
The process has three inputs:
172 34 Maarten Plieger
# inputa - a float number
173 34 Maarten Plieger
# inputb - a float number
174 34 Maarten Plieger
# operation -  Can be "add", "substract", "divide" or "multiply"
175 35 Maarten Plieger
176 34 Maarten Plieger
177 30 Maarten Plieger
<pre><code class="python">
178 30 Maarten Plieger
#!/bin/python
179 30 Maarten Plieger
180 30 Maarten Plieger
"""
181 30 Maarten Plieger
Obtain an access token:
182 33 Maarten Plieger
- Login with a google account at https://dev.climate4impact.eu/impactportal/account/OAuth2.jsp 
183 30 Maarten Plieger
- Generate via https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
184 32 Maarten Plieger
- Copy paste the accesstoken value in the "token" variable below.
185 31 Maarten Plieger
"""
186 30 Maarten Plieger
187 30 Maarten Plieger
token="<access_token>" 
188 30 Maarten Plieger
 
189 30 Maarten Plieger
190 33 Maarten Plieger
wpsendpoint="https://dev.climate4impact.eu/impactportal/WPS/"+token+"/?"
191 30 Maarten Plieger
192 30 Maarten Plieger
inputa=11;
193 30 Maarten Plieger
inputb=22;
194 30 Maarten Plieger
operator="divide" # Any of add, substract, multiply, divide
195 30 Maarten Plieger
196 30 Maarten Plieger
print "Calculating ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"]"
197 30 Maarten Plieger
198 30 Maarten Plieger
import xml.etree.ElementTree as et
199 30 Maarten Plieger
import urllib2
200 30 Maarten Plieger
import time
201 30 Maarten Plieger
202 30 Maarten Plieger
""" Parses URL to XML tree """
203 30 Maarten Plieger
def getandparsexml(url):
204 30 Maarten Plieger
  response = urllib2.urlopen(url)
205 30 Maarten Plieger
  html = response.read()
206 30 Maarten Plieger
  #print html
207 30 Maarten Plieger
  return et.fromstring(html)
208 30 Maarten Plieger
  
209 30 Maarten Plieger
# Execute the ultimatequestionprocess aynchronousely and obtain the statusLocation for progress
210 30 Maarten Plieger
datainputs = "inputa="+str(inputa)+";inputb="+str(inputb)+";operator="+str(operator)
211 30 Maarten Plieger
root = getandparsexml(wpsendpoint+"service=WPS&version=1.0.0&request=execute&storeExecuteResponse=true&status=true&identifier=binaryoperatorfornumbers&datainputs="+datainputs)    
212 30 Maarten Plieger
monitorURL = root.attrib['statusLocation']
213 30 Maarten Plieger
214 30 Maarten Plieger
print "Process executed checking " +monitorURL
215 30 Maarten Plieger
216 30 Maarten Plieger
# Get starttime of process, sometimes the monitorURL is not immediately available. Iterate until a valid creationtime is available would be good practice.
217 30 Maarten Plieger
numberOfTries=10
218 30 Maarten Plieger
while numberOfTries>0:
219 30 Maarten Plieger
  numberOfTries = numberOfTries-1 
220 30 Maarten Plieger
  try:
221 30 Maarten Plieger
    root = getandparsexml(monitorURL)
222 30 Maarten Plieger
    numberOfTries = -1
223 30 Maarten Plieger
    print "Process started at "+str(root[1].attrib['creationTime'])
224 30 Maarten Plieger
  except:
225 30 Maarten Plieger
    time.sleep(1)
226 30 Maarten Plieger
    pass
227 30 Maarten Plieger
228 30 Maarten Plieger
# Monitor progress, polling the statuslocation frequently
229 30 Maarten Plieger
percentage=0
230 30 Maarten Plieger
while percentage<100:
231 30 Maarten Plieger
  root = getandparsexml(monitorURL)
232 30 Maarten Plieger
  try:
233 30 Maarten Plieger
    percentage = float(root[1][0].attrib['percentCompleted'])
234 30 Maarten Plieger
  except:
235 30 Maarten Plieger
    percentage=100.0
236 30 Maarten Plieger
    pass
237 30 Maarten Plieger
  print "Progress: "+str(percentage)+ " complete."
238 30 Maarten Plieger
  time.sleep(1)
239 30 Maarten Plieger
240 30 Maarten Plieger
# Obtain the answer  
241 30 Maarten Plieger
root = getandparsexml(monitorURL)
242 30 Maarten Plieger
try:
243 30 Maarten Plieger
  print "The answer for ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"] is "+(root[2][0][2][0].text)
244 30 Maarten Plieger
except:
245 30 Maarten Plieger
  print "Processing failed: "+root[1][0][0][0][0].text
246 30 Maarten Plieger
  pass
247 30 Maarten Plieger
</code>
248 30 Maarten Plieger
</pre>
249 42 Maarten Plieger
250 42 Maarten Plieger
The result is:
251 42 Maarten Plieger
<pre>
252 42 Maarten Plieger
(pyvirtenv)c4m@climate4impact:~$ python c4i_testwps_binaryoperator.py 
253 42 Maarten Plieger
Calculating [11 divide 22]
254 42 Maarten Plieger
Process executed checking http://climate4impact.eu/impactportal/WPS?OUTPUT=/pywps-144559994232.xml
255 42 Maarten Plieger
Process started at 2015-10-23T11:32:23Z
256 42 Maarten Plieger
Progress: 20.0 complete.
257 42 Maarten Plieger
Progress: 40.0 complete.
258 42 Maarten Plieger
Progress: 60.0 complete.
259 42 Maarten Plieger
Progress: 80.0 complete.
260 47 Maarten Plieger
Progress: 100.0 complete.
261 47 Maarten Plieger
The answer for [11 divide 22] is 0.5
262 47 Maarten Plieger
(pyvirtenv)c4m@climate4impact:~$ 
263 47 Maarten Plieger
</pre>
264 52 Maarten Plieger
265 52 Maarten Plieger
h3. WPS JobList requests
266 52 Maarten Plieger
267 52 Maarten Plieger
All submitted jobs are captured and stored in a per user joblist. The joblist can be obtained via:
268 52 Maarten Plieger
269 52 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/ImpactService?service=processor&request=getProcessorStatusOverview
270 52 Maarten Plieger
271 52 Maarten Plieger
A json with array called jobs is returned:
272 52 Maarten Plieger
273 52 Maarten Plieger
<pre>
274 52 Maarten Plieger
{
275 52 Maarten Plieger
  jobs:[{
276 52 Maarten Plieger
    "wpspostdata": ...,
277 52 Maarten Plieger
    "creationdate": "2016-03-30 16:57:03Z",
278 52 Maarten Plieger
    "wpsid": "clipc_combine_execute",
279 52 Maarten Plieger
    "processid": "pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml",
280 52 Maarten Plieger
    "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml",
281 52 Maarten Plieger
    "progress": "ready", // Ready indicated completed job
282 52 Maarten Plieger
    "error": false,
283 52 Maarten Plieger
    "status": "ready" // Ready, running, failed
284 52 Maarten Plieger
  },{
285 52 Maarten Plieger
    "wpspostdata":  ...,
286 52 Maarten Plieger
    "creationdate": "2016-03-30 17:01:36Z",
287 52 Maarten Plieger
    "wpsid": "wps_simple_indice",
288 52 Maarten Plieger
    "processid": "pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml",
289 52 Maarten Plieger
    "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml",
290 52 Maarten Plieger
    "progress": "0 %", //Percentage completed
291 52 Maarten Plieger
    "error": false,
292 52 Maarten Plieger
    "status": "running" // Ready, running, failed
293 52 Maarten Plieger
  }]
294 52 Maarten Plieger
}
295 52 Maarten Plieger
</pre>
296 53 Maarten Plieger
297 53 Maarten Plieger
A job can be removed from the list by doing:
298 53 Maarten Plieger
299 53 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/ImpactService?service=processor&request=removeFromList&id=pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml
300 69 Maarten Plieger
301 69 Maarten Plieger
h2. Search API
302 69 Maarten Plieger
303 69 Maarten Plieger
https://climate4impact.eu/impactportal/esgfsearch?service=search&request=getfacets&query=project%3DCMIP5%26variable%3Dtas%26time_frequency%3Dday%26experiment%3Dhistorical%26&pagelimit=25&pagenumber=0&