Project

General

Profile

API » History » Version 61

« Previous - Version 61/72 (diff) - Next » - Current version
Maarten Plieger, 04/12/2016 02:41 PM


API

Clipc combine process

The WPS to combine two netcdf files: Clipc Combine process

Access token API - service "account"

1). Generate a token for the current user which can be used to access C4I services by using "generatetoken":

The token has the following structure:

{ 
  "creationdate" : "2015-10-23T09:50:05Z",
  "notafter" : "2015-10-30T09:50:05Z",
  "notbefore" : "2015-10-23T09:50:05Z",
  "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>",
  "token" : "daad2944-1def-4bee-b032-b08048f88302", 
  "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>" 
}

2). List all tokens available for the current user by using "listtokens": Service endpoints which can be used with a token:
  • WMS can be used via
    https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?
  • WCS can be used via
    https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?
  • WPS can be used via
    https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?
  • File download requests and OpenDAP requests from the basket can be done via
    https://dev.climate4impact.eu/impactportal/DAP/<accesstoken>/<userid>/<file>"
  • Basket requests can be done via
    https://dev.climate4impact.eu/impactportal/basket/<accesstoken>?&service=basket&request=getoverview&
  • ImpactService requests can be done via
    https://dev.climate4impact.eu/impactportal/ImpactService/<accesstoken>/?
3). Crash course token generation:
  1. Login with a google account at https://dev.climate4impact.eu/impactportal/account/OAuth2.jsp
  2. Generate a new token via https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
  3. Copy paste the accesstoken value in your URL

CLIP-C Access token API - service "clipcaccount"

This is not a public API, but a machine to machine service end point, which the CLIPC server can use to request tokens onbehalfof the logged-in user.
This token then enables theCLIPC server to

1). Generate a token for the current user which can be used to access C4I services by using "generatetoken":

The token has the following structure:

{ 
  "creationdate" : "2015-10-23T09:50:05Z",
  "notafter" : "2015-10-30T09:50:05Z",
  "notbefore" : "2015-10-23T09:50:05Z",
  "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>",
  "token" : "daad2944-1def-4bee-b032-b08048f88302", 
  "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>" 
}

Basket requests

  • Get overview of all files in the users basket: service=basket&request=getoverview,
    • e.g.
      https://dev.climate4impact.eu/impactportal/basket?&service=basket&request=getoverview&
    • For commandline access with an access token:
      https://dev.climate4impact.eu/impactportal/basket/<accesstoken>/?&service=basket&request=getoverview&

The result is a json file with nested childs representing te directory structure at C4I

{ "browser" : "/impactportal/data/catalogbrowser.jsp?",
  "children" : [ { "children" : [  ],
        "date" : "",
        "expanded" : true,
        "iconCls" : "typeFolder",
        "leaf" : false,
        "text" : "Remote data" 
      },
      { "children" : [ { "dapurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
              "date" : "2014-06-06 14:31:23Z",
              "filesize" : "611.573K",
              "hasdap" : true,
              "hashttp" : true,
              "httpurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
              "iconCls" : "typeOF",
              "id" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
              "index" : 1,
              "leaf" : true,
              "text" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
              "type" : "file" 
            },
            { "dapurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
              "date" : "2015-10-23 09:37:33Z",
              "filesize" : "47.867M",
              "hasdap" : true,
              "hashttp" : true,
              "httpurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
              "iconCls" : "typeOF",
              "id" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
              "index" : 2,
              "leaf" : true,
              "text" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
              "type" : "file" 
            }
          ],
        "date" : "",
        "expanded" : true,
        "iconCls" : "typeFolder",
        "leaf" : false,
        "text" : "My data" 
      }
    ],
  "leaf" : false,
  "text" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<openid>",
  "viewer" : "/impactportal/data/datasetviewer.jsp?" 
}

Remove a file from the basket

GET request:
https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?&service=basket&request=removeFromList&id[]=<id from basket request>

e.g.:
https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?&service=basket&request=removeFromList&id[]=INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc

Add a file to the basket

POST request to https://dev.climate4impact.eu/impactportal/ImpactService?service=basket&mode=add

with body id and opendap:
id=ice_conc_sh_polstere-100_multi_201604091200.nc&OPENDAP=http%3A%2F%2Fthredds.met.no%2Fthredds%2FdodsC

%2Fosisaf%2Fmet.no%2Fice%2Fconc%2F2016%2F04%2Fice_conc_sh_polstere-100_multi_201604091200.nc

To remove multiple files at once do:

https://dev.climate4impact.eu/impactportal/ImpactService?&service=basket&request=removeFromList&id[]=WPS_clipc_combine_execute_20160222T150058Z&id[]=WPS_clipc_combine_execute_20160222T150045Z&id[]=WPS_clipc_combine_execute_20160222T145836Z&id[]=WPS_clipc_combine_execute_20160222T145649Z&id[]=WPS_clipc_combine_execute_20160222T145633Z&id[]=WPS_clipc_combine_execute_20160222T145449Z&id[]=WPS_clipc_combine_execute_20160222T145323Z&id[]=WPS_clipc_combine_execute_20160222T145308Z&id[]=WPS_clipc_combine_execute_20160222T145230Z&id[]=WPS_clipc_combine_execute_20160222T145030Z&id[]=WPS_clipc_combine_execute_20160222T144854Z&id[]=WPS_clipc_combine_execute_20160222T144852Z&id[]=WPS_clipc_combine_execute_20160222T144704Z&id[]=WPS_clipc_combine_execute_20160222T144549Z

WPS Requests

Climate4impact uses PyWPS, see http://pywps.wald.intevation.org/documentation/course/ for more information

  • WPS can be used via
    https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?

Example Python WPS client on WPS Process "binaryoperatoronnumbers"

The following Python example behaves like a minimal WPS client, using C4I services to perform analysis. The example below makes use of the binaryoperatorfornumbers process, which enables operations on two numbers. The process needs five seconds to run, listing progress in percentages. This is done to ease progress monitoring developments. The process is executed asynchronously and is executed with the GET method. PyWPS also supports POST, this can be used for posting bigger inputs. This is described here: WPS

The process has three inputs:
  1. inputa - a float number
  2. inputb - a float number
  3. operation - Can be "add", "substract", "divide" or "multiply"
#!/bin/python

""" 
Obtain an access token:
- Login with a google account at https://dev.climate4impact.eu/impactportal/account/OAuth2.jsp 
- Generate via https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
- Copy paste the accesstoken value in the "token" variable below.
""" 

token="<access_token>" 

wpsendpoint="https://dev.climate4impact.eu/impactportal/WPS/"+token+"/?" 

inputa=11;
inputb=22;
operator="divide" # Any of add, substract, multiply, divide

print "Calculating ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"]" 

import xml.etree.ElementTree as et
import urllib2
import time

""" Parses URL to XML tree """ 
def getandparsexml(url):
  response = urllib2.urlopen(url)
  html = response.read()
  #print html
  return et.fromstring(html)

# Execute the ultimatequestionprocess aynchronousely and obtain the statusLocation for progress
datainputs = "inputa="+str(inputa)+";inputb="+str(inputb)+";operator="+str(operator)
root = getandparsexml(wpsendpoint+"service=WPS&version=1.0.0&request=execute&storeExecuteResponse=true&status=true&identifier=binaryoperatorfornumbers&datainputs="+datainputs)    
monitorURL = root.attrib['statusLocation']

print "Process executed checking " +monitorURL

# Get starttime of process, sometimes the monitorURL is not immediately available. Iterate until a valid creationtime is available would be good practice.
numberOfTries=10
while numberOfTries>0:
  numberOfTries = numberOfTries-1 
  try:
    root = getandparsexml(monitorURL)
    numberOfTries = -1
    print "Process started at "+str(root[1].attrib['creationTime'])
  except:
    time.sleep(1)
    pass

# Monitor progress, polling the statuslocation frequently
percentage=0
while percentage<100:
  root = getandparsexml(monitorURL)
  try:
    percentage = float(root[1][0].attrib['percentCompleted'])
  except:
    percentage=100.0
    pass
  print "Progress: "+str(percentage)+ " complete." 
  time.sleep(1)

# Obtain the answer  
root = getandparsexml(monitorURL)
try:
  print "The answer for ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"] is "+(root[2][0][2][0].text)
except:
  print "Processing failed: "+root[1][0][0][0][0].text
  pass

The result is:

(pyvirtenv)c4m@climate4impact:~$ python c4i_testwps_binaryoperator.py 
Calculating [11 divide 22]
Process executed checking http://climate4impact.eu/impactportal/WPS?OUTPUT=/pywps-144559994232.xml
Process started at 2015-10-23T11:32:23Z
Progress: 20.0 complete.
Progress: 40.0 complete.
Progress: 60.0 complete.
Progress: 80.0 complete.
Progress: 100.0 complete.
The answer for [11 divide 22] is 0.5
(pyvirtenv)c4m@climate4impact:~$ 

WPS JobList requests

All submitted jobs are captured and stored in a per user joblist. The joblist can be obtained via:

A json with array called jobs is returned:

{
  jobs:[{
    "wpspostdata": ...,
    "creationdate": "2016-03-30 16:57:03Z",
    "wpsid": "clipc_combine_execute",
    "processid": "pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml",
    "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml",
    "progress": "ready", // Ready indicated completed job
    "error": false,
    "status": "ready" // Ready, running, failed
  },{
    "wpspostdata":  ...,
    "creationdate": "2016-03-30 17:01:36Z",
    "wpsid": "wps_simple_indice",
    "processid": "pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml",
    "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml",
    "progress": "0 %", //Percentage completed
    "error": false,
    "status": "running" // Ready, running, failed
  }]
}

A job can be removed from the list by doing:

run_wps_from_client.py View - Runs a very simple WPS client on one of the services at climate4impact (2.86 KB) Maarten Plieger, 10/31/2016 03:14 PM