Project

General

Profile

API » History » Version 62

Maarten Plieger, 04/12/2016 02:42 PM

1 1 Maarten Plieger
h1. API
2 1 Maarten Plieger
3 48 Maarten Plieger
h2. Clipc combine process
4 48 Maarten Plieger
5 49 Maarten Plieger
The WPS to combine two netcdf files: [[Clipc Combine process]]
6 48 Maarten Plieger
7 28 Maarten Plieger
h2. Access token API - service "account"
8 1 Maarten Plieger
9 28 Maarten Plieger
*1). Generate a token for the current user which can be used to access C4I services by using "generatetoken":*
10 33 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
11 1 Maarten Plieger
12 17 Maarten Plieger
The token has the following structure:
13 17 Maarten Plieger
<pre><code class="json">
14 18 Maarten Plieger
{ 
15 18 Maarten Plieger
  "creationdate" : "2015-10-23T09:50:05Z",
16 17 Maarten Plieger
  "notafter" : "2015-10-30T09:50:05Z",
17 17 Maarten Plieger
  "notbefore" : "2015-10-23T09:50:05Z",
18 22 Maarten Plieger
  "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>",
19 24 Maarten Plieger
  "token" : "daad2944-1def-4bee-b032-b08048f88302", 
20 22 Maarten Plieger
  "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>"
21 17 Maarten Plieger
}
22 17 Maarten Plieger
</code></pre>
23 17 Maarten Plieger
24 28 Maarten Plieger
*2). List all tokens available for the current user by using "listtokens":*
25 33 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=listtokens
26 1 Maarten Plieger
27 1 Maarten Plieger
*Service endpoints which can be used with a token:*
28 1 Maarten Plieger
* WMS can be used via <pre>https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?</pre>
29 1 Maarten Plieger
* WCS can be used via <pre>https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?</pre>
30 1 Maarten Plieger
* WPS can be used via <pre>https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?</pre>
31 1 Maarten Plieger
* File download requests and OpenDAP requests from the basket can be done via <pre>https://dev.climate4impact.eu/impactportal/DAP/<accesstoken>/<userid>/<file>"</pre>
32 29 Maarten Plieger
* Basket requests can be done via <pre>https://dev.climate4impact.eu/impactportal/basket/<accesstoken>?&service=basket&request=getoverview&</pre>
33 55 Maarten Plieger
* ImpactService requests can be done via <pre>https://dev.climate4impact.eu/impactportal/ImpactService/<accesstoken>/?</pre>
34 39 Maarten Plieger
35 41 Maarten Plieger
*3). Crash course token generation:*
36 39 Maarten Plieger
# Login with a google account at https://dev.climate4impact.eu/impactportal/account/OAuth2.jsp 
37 40 Maarten Plieger
# Generate a new token via https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
38 39 Maarten Plieger
# Copy paste the accesstoken value in your URL
39 1 Maarten Plieger
40 50 Ernst de Vreede
h2. CLIP-C Access token API - service "clipcaccount"
41 50 Ernst de Vreede
42 51 Ernst de Vreede
This is not a public API, but a machine to machine service end point, which the CLIPC server can use to request tokens onbehalfof the logged-in user.
43 51 Ernst de Vreede
This token then enables theCLIPC server to
44 51 Ernst de Vreede
 
45 50 Ernst de Vreede
*1). Generate a token for the current user which can be used to access C4I services by using "generatetoken":*
46 50 Ernst de Vreede
* https://dev.climate4impact.eu/impactportal/tokenapi?service=clipcaccount&request=generatetoken&userid=<userid>
47 50 Ernst de Vreede
48 50 Ernst de Vreede
The token has the following structure:
49 50 Ernst de Vreede
<pre><code class="json">
50 50 Ernst de Vreede
{ 
51 50 Ernst de Vreede
  "creationdate" : "2015-10-23T09:50:05Z",
52 50 Ernst de Vreede
  "notafter" : "2015-10-30T09:50:05Z",
53 50 Ernst de Vreede
  "notbefore" : "2015-10-23T09:50:05Z",
54 50 Ernst de Vreede
  "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>",
55 50 Ernst de Vreede
  "token" : "daad2944-1def-4bee-b032-b08048f88302", 
56 50 Ernst de Vreede
  "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>"
57 50 Ernst de Vreede
}
58 50 Ernst de Vreede
</code></pre>
59 50 Ernst de Vreede
60 5 Maarten Plieger
h2. Basket requests
61 1 Maarten Plieger
62 2 Maarten Plieger
* Get overview of all files in the users basket: service=basket&request=getoverview, 
63 33 Maarten Plieger
** e.g. <pre>https://dev.climate4impact.eu/impactportal/basket?&service=basket&request=getoverview&</pre>
64 33 Maarten Plieger
** For commandline access with an access token: <pre>https://dev.climate4impact.eu/impactportal/basket/<accesstoken>/?&service=basket&request=getoverview&</pre>
65 7 Maarten Plieger
66 14 Maarten Plieger
The result is a json file with nested childs representing te directory structure at C4I
67 10 Maarten Plieger
<pre><code class="json">
68 12 Maarten Plieger
{ "browser" : "/impactportal/data/catalogbrowser.jsp?",
69 12 Maarten Plieger
  "children" : [ { "children" : [  ],
70 12 Maarten Plieger
        "date" : "",
71 12 Maarten Plieger
        "expanded" : true,
72 12 Maarten Plieger
        "iconCls" : "typeFolder",
73 12 Maarten Plieger
        "leaf" : false,
74 12 Maarten Plieger
        "text" : "Remote data"
75 1 Maarten Plieger
      },
76 13 Maarten Plieger
      { "children" : [ { "dapurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
77 12 Maarten Plieger
              "date" : "2014-06-06 14:31:23Z",
78 12 Maarten Plieger
              "filesize" : "611.573K",
79 1 Maarten Plieger
              "hasdap" : true,
80 12 Maarten Plieger
              "hashttp" : true,
81 13 Maarten Plieger
              "httpurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
82 12 Maarten Plieger
              "iconCls" : "typeOF",
83 12 Maarten Plieger
              "id" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
84 12 Maarten Plieger
              "index" : 1,
85 12 Maarten Plieger
              "leaf" : true,
86 1 Maarten Plieger
              "text" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
87 12 Maarten Plieger
              "type" : "file"
88 12 Maarten Plieger
            },
89 13 Maarten Plieger
            { "dapurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
90 1 Maarten Plieger
              "date" : "2015-10-23 09:37:33Z",
91 12 Maarten Plieger
              "filesize" : "47.867M",
92 12 Maarten Plieger
              "hasdap" : true,
93 12 Maarten Plieger
              "hashttp" : true,
94 13 Maarten Plieger
              "httpurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
95 12 Maarten Plieger
              "iconCls" : "typeOF",
96 12 Maarten Plieger
              "id" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
97 12 Maarten Plieger
              "index" : 2,
98 12 Maarten Plieger
              "leaf" : true,
99 12 Maarten Plieger
              "text" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
100 12 Maarten Plieger
              "type" : "file"
101 12 Maarten Plieger
            }
102 12 Maarten Plieger
          ],
103 12 Maarten Plieger
        "date" : "",
104 12 Maarten Plieger
        "expanded" : true,
105 12 Maarten Plieger
        "iconCls" : "typeFolder",
106 1 Maarten Plieger
        "leaf" : false,
107 12 Maarten Plieger
        "text" : "My data"
108 12 Maarten Plieger
      }
109 12 Maarten Plieger
    ],
110 12 Maarten Plieger
  "leaf" : false,
111 13 Maarten Plieger
  "text" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<openid>",
112 11 Maarten Plieger
  "viewer" : "/impactportal/data/datasetviewer.jsp?"
113 11 Maarten Plieger
}
114 10 Maarten Plieger
</code>
115 56 Maarten Plieger
116 57 Maarten Plieger
</pre>
117 57 Maarten Plieger
118 57 Maarten Plieger
119 56 Maarten Plieger
*Remove a file from the basket*
120 56 Maarten Plieger
121 57 Maarten Plieger
<pre>
122 61 Maarten Plieger
GET request:
123 59 Maarten Plieger
https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?&service=basket&request=removeFromList&id[]=<id from basket request>
124 57 Maarten Plieger
</pre>
125 56 Maarten Plieger
e.g.:
126 57 Maarten Plieger
<pre>
127 60 Maarten Plieger
https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?&service=basket&request=removeFromList&id[]=INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc
128 58 Maarten Plieger
</pre>
129 61 Maarten Plieger
130 61 Maarten Plieger
*Add a file to the basket*
131 61 Maarten Plieger
132 61 Maarten Plieger
<pre>
133 61 Maarten Plieger
POST request to https://dev.climate4impact.eu/impactportal/ImpactService?service=basket&mode=add
134 61 Maarten Plieger
</pre>
135 61 Maarten Plieger
with body id and opendap:
136 61 Maarten Plieger
<pre>
137 62 Maarten Plieger
id=ice_conc_sh_polstere-100_multi_201604091200.nc&OPENDAP=http%3A%2F%2Fthredds.met.no%2Fthredds%2FdodsC%2Fosisaf%2Fmet.no%2Fice%2Fconc%2F2016%2F04%2Fice_conc_sh_polstere-100_multi_201604091200.nc
138 61 Maarten Plieger
</pre>
139 61 Maarten Plieger
140 61 Maarten Plieger
</pre>
141 61 Maarten Plieger
142 61 Maarten Plieger
143 61 Maarten Plieger
144 61 Maarten Plieger
145 1 Maarten Plieger
146 58 Maarten Plieger
To remove multiple files at once do:
147 58 Maarten Plieger
<pre>
148 58 Maarten Plieger
https://dev.climate4impact.eu/impactportal/ImpactService?&service=basket&request=removeFromList&id[]=WPS_clipc_combine_execute_20160222T150058Z&id[]=WPS_clipc_combine_execute_20160222T150045Z&id[]=WPS_clipc_combine_execute_20160222T145836Z&id[]=WPS_clipc_combine_execute_20160222T145649Z&id[]=WPS_clipc_combine_execute_20160222T145633Z&id[]=WPS_clipc_combine_execute_20160222T145449Z&id[]=WPS_clipc_combine_execute_20160222T145323Z&id[]=WPS_clipc_combine_execute_20160222T145308Z&id[]=WPS_clipc_combine_execute_20160222T145230Z&id[]=WPS_clipc_combine_execute_20160222T145030Z&id[]=WPS_clipc_combine_execute_20160222T144854Z&id[]=WPS_clipc_combine_execute_20160222T144852Z&id[]=WPS_clipc_combine_execute_20160222T144704Z&id[]=WPS_clipc_combine_execute_20160222T144549Z
149 9 Maarten Plieger
</pre>
150 30 Maarten Plieger
151 30 Maarten Plieger
h2. WPS Requests
152 30 Maarten Plieger
153 36 Maarten Plieger
Climate4impact uses PyWPS, see http://pywps.wald.intevation.org/documentation/course/ for more information
154 36 Maarten Plieger
155 36 Maarten Plieger
* WPS can be used via <pre>https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?</pre>
156 36 Maarten Plieger
157 44 Maarten Plieger
158 44 Maarten Plieger
159 44 Maarten Plieger
*Example Python WPS client on WPS Process "binaryoperatoronnumbers"*
160 36 Maarten Plieger
161 46 Maarten Plieger
The following Python example behaves like a minimal WPS client, using C4I services to perform analysis. The example below makes use of the binaryoperatorfornumbers process, which enables operations on two numbers. The process needs five seconds to run, listing progress in percentages. This is done to ease progress monitoring developments. The process is executed asynchronously and is executed with the GET method. PyWPS also supports POST, this can be used for posting bigger inputs. This is described here: [[WPS]] 
162 46 Maarten Plieger
163 46 Maarten Plieger
The process has three inputs:
164 34 Maarten Plieger
# inputa - a float number
165 34 Maarten Plieger
# inputb - a float number
166 34 Maarten Plieger
# operation -  Can be "add", "substract", "divide" or "multiply"
167 35 Maarten Plieger
168 34 Maarten Plieger
169 30 Maarten Plieger
<pre><code class="python">
170 30 Maarten Plieger
#!/bin/python
171 30 Maarten Plieger
172 30 Maarten Plieger
"""
173 30 Maarten Plieger
Obtain an access token:
174 33 Maarten Plieger
- Login with a google account at https://dev.climate4impact.eu/impactportal/account/OAuth2.jsp 
175 30 Maarten Plieger
- Generate via https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
176 32 Maarten Plieger
- Copy paste the accesstoken value in the "token" variable below.
177 31 Maarten Plieger
"""
178 30 Maarten Plieger
179 30 Maarten Plieger
token="<access_token>" 
180 30 Maarten Plieger
 
181 30 Maarten Plieger
182 33 Maarten Plieger
wpsendpoint="https://dev.climate4impact.eu/impactportal/WPS/"+token+"/?"
183 30 Maarten Plieger
184 30 Maarten Plieger
inputa=11;
185 30 Maarten Plieger
inputb=22;
186 30 Maarten Plieger
operator="divide" # Any of add, substract, multiply, divide
187 30 Maarten Plieger
188 30 Maarten Plieger
print "Calculating ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"]"
189 30 Maarten Plieger
190 30 Maarten Plieger
import xml.etree.ElementTree as et
191 30 Maarten Plieger
import urllib2
192 30 Maarten Plieger
import time
193 30 Maarten Plieger
194 30 Maarten Plieger
""" Parses URL to XML tree """
195 30 Maarten Plieger
def getandparsexml(url):
196 30 Maarten Plieger
  response = urllib2.urlopen(url)
197 30 Maarten Plieger
  html = response.read()
198 30 Maarten Plieger
  #print html
199 30 Maarten Plieger
  return et.fromstring(html)
200 30 Maarten Plieger
  
201 30 Maarten Plieger
# Execute the ultimatequestionprocess aynchronousely and obtain the statusLocation for progress
202 30 Maarten Plieger
datainputs = "inputa="+str(inputa)+";inputb="+str(inputb)+";operator="+str(operator)
203 30 Maarten Plieger
root = getandparsexml(wpsendpoint+"service=WPS&version=1.0.0&request=execute&storeExecuteResponse=true&status=true&identifier=binaryoperatorfornumbers&datainputs="+datainputs)    
204 30 Maarten Plieger
monitorURL = root.attrib['statusLocation']
205 30 Maarten Plieger
206 30 Maarten Plieger
print "Process executed checking " +monitorURL
207 30 Maarten Plieger
208 30 Maarten Plieger
# Get starttime of process, sometimes the monitorURL is not immediately available. Iterate until a valid creationtime is available would be good practice.
209 30 Maarten Plieger
numberOfTries=10
210 30 Maarten Plieger
while numberOfTries>0:
211 30 Maarten Plieger
  numberOfTries = numberOfTries-1 
212 30 Maarten Plieger
  try:
213 30 Maarten Plieger
    root = getandparsexml(monitorURL)
214 30 Maarten Plieger
    numberOfTries = -1
215 30 Maarten Plieger
    print "Process started at "+str(root[1].attrib['creationTime'])
216 30 Maarten Plieger
  except:
217 30 Maarten Plieger
    time.sleep(1)
218 30 Maarten Plieger
    pass
219 30 Maarten Plieger
220 30 Maarten Plieger
# Monitor progress, polling the statuslocation frequently
221 30 Maarten Plieger
percentage=0
222 30 Maarten Plieger
while percentage<100:
223 30 Maarten Plieger
  root = getandparsexml(monitorURL)
224 30 Maarten Plieger
  try:
225 30 Maarten Plieger
    percentage = float(root[1][0].attrib['percentCompleted'])
226 30 Maarten Plieger
  except:
227 30 Maarten Plieger
    percentage=100.0
228 30 Maarten Plieger
    pass
229 30 Maarten Plieger
  print "Progress: "+str(percentage)+ " complete."
230 30 Maarten Plieger
  time.sleep(1)
231 30 Maarten Plieger
232 30 Maarten Plieger
# Obtain the answer  
233 30 Maarten Plieger
root = getandparsexml(monitorURL)
234 30 Maarten Plieger
try:
235 30 Maarten Plieger
  print "The answer for ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"] is "+(root[2][0][2][0].text)
236 30 Maarten Plieger
except:
237 30 Maarten Plieger
  print "Processing failed: "+root[1][0][0][0][0].text
238 30 Maarten Plieger
  pass
239 30 Maarten Plieger
</code>
240 30 Maarten Plieger
</pre>
241 42 Maarten Plieger
242 42 Maarten Plieger
The result is:
243 42 Maarten Plieger
<pre>
244 42 Maarten Plieger
(pyvirtenv)c4m@climate4impact:~$ python c4i_testwps_binaryoperator.py 
245 42 Maarten Plieger
Calculating [11 divide 22]
246 42 Maarten Plieger
Process executed checking http://climate4impact.eu/impactportal/WPS?OUTPUT=/pywps-144559994232.xml
247 42 Maarten Plieger
Process started at 2015-10-23T11:32:23Z
248 42 Maarten Plieger
Progress: 20.0 complete.
249 42 Maarten Plieger
Progress: 40.0 complete.
250 42 Maarten Plieger
Progress: 60.0 complete.
251 42 Maarten Plieger
Progress: 80.0 complete.
252 47 Maarten Plieger
Progress: 100.0 complete.
253 47 Maarten Plieger
The answer for [11 divide 22] is 0.5
254 47 Maarten Plieger
(pyvirtenv)c4m@climate4impact:~$ 
255 47 Maarten Plieger
</pre>
256 52 Maarten Plieger
257 52 Maarten Plieger
h3. WPS JobList requests
258 52 Maarten Plieger
259 52 Maarten Plieger
All submitted jobs are captured and stored in a per user joblist. The joblist can be obtained via:
260 52 Maarten Plieger
261 52 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/ImpactService?service=processor&request=getProcessorStatusOverview
262 52 Maarten Plieger
263 52 Maarten Plieger
A json with array called jobs is returned:
264 52 Maarten Plieger
265 52 Maarten Plieger
<pre>
266 52 Maarten Plieger
{
267 52 Maarten Plieger
  jobs:[{
268 52 Maarten Plieger
    "wpspostdata": ...,
269 52 Maarten Plieger
    "creationdate": "2016-03-30 16:57:03Z",
270 52 Maarten Plieger
    "wpsid": "clipc_combine_execute",
271 52 Maarten Plieger
    "processid": "pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml",
272 52 Maarten Plieger
    "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml",
273 52 Maarten Plieger
    "progress": "ready", // Ready indicated completed job
274 52 Maarten Plieger
    "error": false,
275 52 Maarten Plieger
    "status": "ready" // Ready, running, failed
276 52 Maarten Plieger
  },{
277 52 Maarten Plieger
    "wpspostdata":  ...,
278 52 Maarten Plieger
    "creationdate": "2016-03-30 17:01:36Z",
279 52 Maarten Plieger
    "wpsid": "wps_simple_indice",
280 52 Maarten Plieger
    "processid": "pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml",
281 52 Maarten Plieger
    "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml",
282 52 Maarten Plieger
    "progress": "0 %", //Percentage completed
283 52 Maarten Plieger
    "error": false,
284 52 Maarten Plieger
    "status": "running" // Ready, running, failed
285 52 Maarten Plieger
  }]
286 52 Maarten Plieger
}
287 52 Maarten Plieger
</pre>
288 53 Maarten Plieger
289 53 Maarten Plieger
A job can be removed from the list by doing:
290 53 Maarten Plieger
291 53 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/ImpactService?service=processor&request=removeFromList&id=pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml