Project

General

Profile

API » History » Version 54

Maarten Plieger, 03/30/2016 03:08 PM

1 1 Maarten Plieger
h1. API
2 1 Maarten Plieger
3 48 Maarten Plieger
h2. Clipc combine process
4 48 Maarten Plieger
5 49 Maarten Plieger
The WPS to combine two netcdf files: [[Clipc Combine process]]
6 48 Maarten Plieger
7 28 Maarten Plieger
h2. Access token API - service "account"
8 1 Maarten Plieger
9 28 Maarten Plieger
*1). Generate a token for the current user which can be used to access C4I services by using "generatetoken":*
10 33 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
11 1 Maarten Plieger
12 17 Maarten Plieger
The token has the following structure:
13 17 Maarten Plieger
<pre><code class="json">
14 18 Maarten Plieger
{ 
15 18 Maarten Plieger
  "creationdate" : "2015-10-23T09:50:05Z",
16 17 Maarten Plieger
  "notafter" : "2015-10-30T09:50:05Z",
17 17 Maarten Plieger
  "notbefore" : "2015-10-23T09:50:05Z",
18 22 Maarten Plieger
  "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>",
19 24 Maarten Plieger
  "token" : "daad2944-1def-4bee-b032-b08048f88302", 
20 22 Maarten Plieger
  "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>"
21 17 Maarten Plieger
}
22 17 Maarten Plieger
</code></pre>
23 17 Maarten Plieger
24 28 Maarten Plieger
*2). List all tokens available for the current user by using "listtokens":*
25 33 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=listtokens
26 1 Maarten Plieger
27 1 Maarten Plieger
*Service endpoints which can be used with a token:*
28 1 Maarten Plieger
* WMS can be used via <pre>https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?</pre>
29 1 Maarten Plieger
* WCS can be used via <pre>https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?</pre>
30 1 Maarten Plieger
* WPS can be used via <pre>https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?</pre>
31 1 Maarten Plieger
* File download requests and OpenDAP requests from the basket can be done via <pre>https://dev.climate4impact.eu/impactportal/DAP/<accesstoken>/<userid>/<file>"</pre>
32 29 Maarten Plieger
* Basket requests can be done via <pre>https://dev.climate4impact.eu/impactportal/basket/<accesstoken>?&service=basket&request=getoverview&</pre>
33 54 Maarten Plieger
* ImpactService requests can be done via <pre>https://dev.climate4impact.eu/impactportal/ImpactService/<accesstoken>?</pre>
34 39 Maarten Plieger
35 41 Maarten Plieger
*3). Crash course token generation:*
36 39 Maarten Plieger
# Login with a google account at https://dev.climate4impact.eu/impactportal/account/OAuth2.jsp 
37 40 Maarten Plieger
# Generate a new token via https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
38 39 Maarten Plieger
# Copy paste the accesstoken value in your URL
39 1 Maarten Plieger
40 50 Ernst de Vreede
h2. CLIP-C Access token API - service "clipcaccount"
41 50 Ernst de Vreede
42 51 Ernst de Vreede
This is not a public API, but a machine to machine service end point, which the CLIPC server can use to request tokens onbehalfof the logged-in user.
43 51 Ernst de Vreede
This token then enables theCLIPC server to
44 51 Ernst de Vreede
 
45 50 Ernst de Vreede
*1). Generate a token for the current user which can be used to access C4I services by using "generatetoken":*
46 50 Ernst de Vreede
* https://dev.climate4impact.eu/impactportal/tokenapi?service=clipcaccount&request=generatetoken&userid=<userid>
47 50 Ernst de Vreede
48 50 Ernst de Vreede
The token has the following structure:
49 50 Ernst de Vreede
<pre><code class="json">
50 50 Ernst de Vreede
{ 
51 50 Ernst de Vreede
  "creationdate" : "2015-10-23T09:50:05Z",
52 50 Ernst de Vreede
  "notafter" : "2015-10-30T09:50:05Z",
53 50 Ernst de Vreede
  "notbefore" : "2015-10-23T09:50:05Z",
54 50 Ernst de Vreede
  "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>",
55 50 Ernst de Vreede
  "token" : "daad2944-1def-4bee-b032-b08048f88302", 
56 50 Ernst de Vreede
  "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>"
57 50 Ernst de Vreede
}
58 50 Ernst de Vreede
</code></pre>
59 50 Ernst de Vreede
60 5 Maarten Plieger
h2. Basket requests
61 1 Maarten Plieger
62 2 Maarten Plieger
* Get overview of all files in the users basket: service=basket&request=getoverview, 
63 33 Maarten Plieger
** e.g. <pre>https://dev.climate4impact.eu/impactportal/basket?&service=basket&request=getoverview&</pre>
64 33 Maarten Plieger
** For commandline access with an access token: <pre>https://dev.climate4impact.eu/impactportal/basket/<accesstoken>/?&service=basket&request=getoverview&</pre>
65 7 Maarten Plieger
66 14 Maarten Plieger
The result is a json file with nested childs representing te directory structure at C4I
67 10 Maarten Plieger
<pre><code class="json">
68 12 Maarten Plieger
{ "browser" : "/impactportal/data/catalogbrowser.jsp?",
69 12 Maarten Plieger
  "children" : [ { "children" : [  ],
70 12 Maarten Plieger
        "date" : "",
71 12 Maarten Plieger
        "expanded" : true,
72 12 Maarten Plieger
        "iconCls" : "typeFolder",
73 12 Maarten Plieger
        "leaf" : false,
74 12 Maarten Plieger
        "text" : "Remote data"
75 1 Maarten Plieger
      },
76 13 Maarten Plieger
      { "children" : [ { "dapurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
77 12 Maarten Plieger
              "date" : "2014-06-06 14:31:23Z",
78 12 Maarten Plieger
              "filesize" : "611.573K",
79 1 Maarten Plieger
              "hasdap" : true,
80 12 Maarten Plieger
              "hashttp" : true,
81 13 Maarten Plieger
              "httpurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
82 12 Maarten Plieger
              "iconCls" : "typeOF",
83 12 Maarten Plieger
              "id" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
84 12 Maarten Plieger
              "index" : 1,
85 12 Maarten Plieger
              "leaf" : true,
86 1 Maarten Plieger
              "text" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
87 12 Maarten Plieger
              "type" : "file"
88 12 Maarten Plieger
            },
89 13 Maarten Plieger
            { "dapurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
90 1 Maarten Plieger
              "date" : "2015-10-23 09:37:33Z",
91 12 Maarten Plieger
              "filesize" : "47.867M",
92 12 Maarten Plieger
              "hasdap" : true,
93 12 Maarten Plieger
              "hashttp" : true,
94 13 Maarten Plieger
              "httpurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
95 12 Maarten Plieger
              "iconCls" : "typeOF",
96 12 Maarten Plieger
              "id" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
97 12 Maarten Plieger
              "index" : 2,
98 12 Maarten Plieger
              "leaf" : true,
99 12 Maarten Plieger
              "text" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
100 12 Maarten Plieger
              "type" : "file"
101 12 Maarten Plieger
            }
102 12 Maarten Plieger
          ],
103 12 Maarten Plieger
        "date" : "",
104 12 Maarten Plieger
        "expanded" : true,
105 12 Maarten Plieger
        "iconCls" : "typeFolder",
106 1 Maarten Plieger
        "leaf" : false,
107 12 Maarten Plieger
        "text" : "My data"
108 12 Maarten Plieger
      }
109 12 Maarten Plieger
    ],
110 12 Maarten Plieger
  "leaf" : false,
111 13 Maarten Plieger
  "text" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<openid>",
112 11 Maarten Plieger
  "viewer" : "/impactportal/data/datasetviewer.jsp?"
113 11 Maarten Plieger
}
114 10 Maarten Plieger
</code>
115 9 Maarten Plieger
</pre>
116 30 Maarten Plieger
117 30 Maarten Plieger
h2. WPS Requests
118 30 Maarten Plieger
119 36 Maarten Plieger
Climate4impact uses PyWPS, see http://pywps.wald.intevation.org/documentation/course/ for more information
120 36 Maarten Plieger
121 36 Maarten Plieger
* WPS can be used via <pre>https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?</pre>
122 36 Maarten Plieger
123 44 Maarten Plieger
124 44 Maarten Plieger
125 44 Maarten Plieger
*Example Python WPS client on WPS Process "binaryoperatoronnumbers"*
126 36 Maarten Plieger
127 46 Maarten Plieger
The following Python example behaves like a minimal WPS client, using C4I services to perform analysis. The example below makes use of the binaryoperatorfornumbers process, which enables operations on two numbers. The process needs five seconds to run, listing progress in percentages. This is done to ease progress monitoring developments. The process is executed asynchronously and is executed with the GET method. PyWPS also supports POST, this can be used for posting bigger inputs. This is described here: [[WPS]] 
128 46 Maarten Plieger
129 46 Maarten Plieger
The process has three inputs:
130 34 Maarten Plieger
# inputa - a float number
131 34 Maarten Plieger
# inputb - a float number
132 34 Maarten Plieger
# operation -  Can be "add", "substract", "divide" or "multiply"
133 35 Maarten Plieger
134 34 Maarten Plieger
135 30 Maarten Plieger
<pre><code class="python">
136 30 Maarten Plieger
#!/bin/python
137 30 Maarten Plieger
138 30 Maarten Plieger
"""
139 30 Maarten Plieger
Obtain an access token:
140 33 Maarten Plieger
- Login with a google account at https://dev.climate4impact.eu/impactportal/account/OAuth2.jsp 
141 30 Maarten Plieger
- Generate via https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
142 32 Maarten Plieger
- Copy paste the accesstoken value in the "token" variable below.
143 31 Maarten Plieger
"""
144 30 Maarten Plieger
145 30 Maarten Plieger
token="<access_token>" 
146 30 Maarten Plieger
 
147 30 Maarten Plieger
148 33 Maarten Plieger
wpsendpoint="https://dev.climate4impact.eu/impactportal/WPS/"+token+"/?"
149 30 Maarten Plieger
150 30 Maarten Plieger
inputa=11;
151 30 Maarten Plieger
inputb=22;
152 30 Maarten Plieger
operator="divide" # Any of add, substract, multiply, divide
153 30 Maarten Plieger
154 30 Maarten Plieger
print "Calculating ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"]"
155 30 Maarten Plieger
156 30 Maarten Plieger
import xml.etree.ElementTree as et
157 30 Maarten Plieger
import urllib2
158 30 Maarten Plieger
import time
159 30 Maarten Plieger
160 30 Maarten Plieger
""" Parses URL to XML tree """
161 30 Maarten Plieger
def getandparsexml(url):
162 30 Maarten Plieger
  response = urllib2.urlopen(url)
163 30 Maarten Plieger
  html = response.read()
164 30 Maarten Plieger
  #print html
165 30 Maarten Plieger
  return et.fromstring(html)
166 30 Maarten Plieger
  
167 30 Maarten Plieger
# Execute the ultimatequestionprocess aynchronousely and obtain the statusLocation for progress
168 30 Maarten Plieger
datainputs = "inputa="+str(inputa)+";inputb="+str(inputb)+";operator="+str(operator)
169 30 Maarten Plieger
root = getandparsexml(wpsendpoint+"service=WPS&version=1.0.0&request=execute&storeExecuteResponse=true&status=true&identifier=binaryoperatorfornumbers&datainputs="+datainputs)    
170 30 Maarten Plieger
monitorURL = root.attrib['statusLocation']
171 30 Maarten Plieger
172 30 Maarten Plieger
print "Process executed checking " +monitorURL
173 30 Maarten Plieger
174 30 Maarten Plieger
# Get starttime of process, sometimes the monitorURL is not immediately available. Iterate until a valid creationtime is available would be good practice.
175 30 Maarten Plieger
numberOfTries=10
176 30 Maarten Plieger
while numberOfTries>0:
177 30 Maarten Plieger
  numberOfTries = numberOfTries-1 
178 30 Maarten Plieger
  try:
179 30 Maarten Plieger
    root = getandparsexml(monitorURL)
180 30 Maarten Plieger
    numberOfTries = -1
181 30 Maarten Plieger
    print "Process started at "+str(root[1].attrib['creationTime'])
182 30 Maarten Plieger
  except:
183 30 Maarten Plieger
    time.sleep(1)
184 30 Maarten Plieger
    pass
185 30 Maarten Plieger
186 30 Maarten Plieger
# Monitor progress, polling the statuslocation frequently
187 30 Maarten Plieger
percentage=0
188 30 Maarten Plieger
while percentage<100:
189 30 Maarten Plieger
  root = getandparsexml(monitorURL)
190 30 Maarten Plieger
  try:
191 30 Maarten Plieger
    percentage = float(root[1][0].attrib['percentCompleted'])
192 30 Maarten Plieger
  except:
193 30 Maarten Plieger
    percentage=100.0
194 30 Maarten Plieger
    pass
195 30 Maarten Plieger
  print "Progress: "+str(percentage)+ " complete."
196 30 Maarten Plieger
  time.sleep(1)
197 30 Maarten Plieger
198 30 Maarten Plieger
# Obtain the answer  
199 30 Maarten Plieger
root = getandparsexml(monitorURL)
200 30 Maarten Plieger
try:
201 30 Maarten Plieger
  print "The answer for ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"] is "+(root[2][0][2][0].text)
202 30 Maarten Plieger
except:
203 30 Maarten Plieger
  print "Processing failed: "+root[1][0][0][0][0].text
204 30 Maarten Plieger
  pass
205 30 Maarten Plieger
</code>
206 30 Maarten Plieger
</pre>
207 42 Maarten Plieger
208 42 Maarten Plieger
The result is:
209 42 Maarten Plieger
<pre>
210 42 Maarten Plieger
(pyvirtenv)c4m@climate4impact:~$ python c4i_testwps_binaryoperator.py 
211 42 Maarten Plieger
Calculating [11 divide 22]
212 42 Maarten Plieger
Process executed checking http://climate4impact.eu/impactportal/WPS?OUTPUT=/pywps-144559994232.xml
213 42 Maarten Plieger
Process started at 2015-10-23T11:32:23Z
214 42 Maarten Plieger
Progress: 20.0 complete.
215 42 Maarten Plieger
Progress: 40.0 complete.
216 42 Maarten Plieger
Progress: 60.0 complete.
217 42 Maarten Plieger
Progress: 80.0 complete.
218 47 Maarten Plieger
Progress: 100.0 complete.
219 47 Maarten Plieger
The answer for [11 divide 22] is 0.5
220 47 Maarten Plieger
(pyvirtenv)c4m@climate4impact:~$ 
221 47 Maarten Plieger
</pre>
222 52 Maarten Plieger
223 52 Maarten Plieger
h3. WPS JobList requests
224 52 Maarten Plieger
225 52 Maarten Plieger
All submitted jobs are captured and stored in a per user joblist. The joblist can be obtained via:
226 52 Maarten Plieger
227 52 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/ImpactService?service=processor&request=getProcessorStatusOverview
228 52 Maarten Plieger
229 52 Maarten Plieger
A json with array called jobs is returned:
230 52 Maarten Plieger
231 52 Maarten Plieger
<pre>
232 52 Maarten Plieger
{
233 52 Maarten Plieger
  jobs:[{
234 52 Maarten Plieger
    "wpspostdata": ...,
235 52 Maarten Plieger
    "creationdate": "2016-03-30 16:57:03Z",
236 52 Maarten Plieger
    "wpsid": "clipc_combine_execute",
237 52 Maarten Plieger
    "processid": "pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml",
238 52 Maarten Plieger
    "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml",
239 52 Maarten Plieger
    "progress": "ready", // Ready indicated completed job
240 52 Maarten Plieger
    "error": false,
241 52 Maarten Plieger
    "status": "ready" // Ready, running, failed
242 52 Maarten Plieger
  },{
243 52 Maarten Plieger
    "wpspostdata":  ...,
244 52 Maarten Plieger
    "creationdate": "2016-03-30 17:01:36Z",
245 52 Maarten Plieger
    "wpsid": "wps_simple_indice",
246 52 Maarten Plieger
    "processid": "pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml",
247 52 Maarten Plieger
    "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml",
248 52 Maarten Plieger
    "progress": "0 %", //Percentage completed
249 52 Maarten Plieger
    "error": false,
250 52 Maarten Plieger
    "status": "running" // Ready, running, failed
251 52 Maarten Plieger
  }]
252 52 Maarten Plieger
}
253 52 Maarten Plieger
</pre>
254 53 Maarten Plieger
255 53 Maarten Plieger
A job can be removed from the list by doing:
256 53 Maarten Plieger
257 53 Maarten Plieger
* https://dev.climate4impact.eu/impactportal/ImpactService?service=processor&request=removeFromList&id=pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml