Project

General

Profile

API

Clipc combine process

Search GUI

Access token API - service "account"

1). Generate a token for the current user which can be used to access C4I services by using "generatetoken":

The token has the following structure:

{ 
  "creationdate" : "2015-10-23T09:50:05Z",
  "notafter" : "2015-10-30T09:50:05Z",
  "notbefore" : "2015-10-23T09:50:05Z",
  "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>",
  "token" : "daad2944-1def-4bee-b032-b08048f88302", 
  "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>" 
}

2). List all tokens available for the current user by using "listtokens": Service endpoints which can be used with a token:
  • WMS can be used via
    https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?
  • WCS can be used via
    https://dev.climate4impact.eu/impactportal/adagucserver/<accesstoken>/?
  • WPS can be used via
    https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?
  • File download requests and OpenDAP requests from the basket can be done via
    https://dev.climate4impact.eu/impactportal/DAP/<accesstoken>/<userid>/<file>"
  • Basket requests can be done via
    https://dev.climate4impact.eu/impactportal/basket/<accesstoken>?&service=basket&request=getoverview&
  • ImpactService requests can be done via
    https://dev.climate4impact.eu/impactportal/ImpactService/<accesstoken>/?
3). Crash course token generation:
  1. Login with a google account at https://dev.climate4impact.eu/impactportal/account/OAuth2.jsp
  2. Generate a new token via https://dev.climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
  3. Copy paste the accesstoken value in your URL

CLIP-C Access token API - service "clipcaccount"

The accesstoken API is a machine to machine service endpoint, the CLIPC server uses this to request tokens onbehalfof the logged-in user at the CLIPC portal.
This token then enables the CLIPC server to generate a token for the current user which can be used to access C4I services by using "generatetoken".

The token has the following structure:

{ 
  "creationdate" : "2015-10-23T09:50:05Z",
  "notafter" : "2015-10-30T09:50:05Z",
  "notbefore" : "2015-10-23T09:50:05Z",
  "openid" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<identifier>",
  "token" : "daad2944-1def-4bee-b032-b08048f88302", 
  "userid" : "esg-dn1.nsc.liu.se.esgf-idp.openid.<identifier>" 
}

Basket requests

  • Get overview of all files in the users basket: service=basket&request=getoverview,
    • e.g.
      https://dev.climate4impact.eu/impactportal/basket?&service=basket&request=getoverview&
    • For commandline access with an access token:
      https://dev.climate4impact.eu/impactportal/basket/<accesstoken>/?&service=basket&request=getoverview&

The result is a json file with nested childs representing te directory structure at C4I

{ "browser" : "/impactportal/data/catalogbrowser.jsp?",
  "children" : [ { "children" : [  ],
        "date" : "",
        "expanded" : true,
        "iconCls" : "typeFolder",
        "leaf" : false,
        "text" : "Remote data" 
      },
      { "children" : [ { "dapurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
              "date" : "2014-06-06 14:31:23Z",
              "filesize" : "611.573K",
              "hasdap" : true,
              "hashttp" : true,
              "httpurl" : "https://localhost/impactportal/DAP/<userid>/INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
              "iconCls" : "typeOF",
              "id" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
              "index" : 1,
              "leaf" : true,
              "text" : "INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc",
              "type" : "file" 
            },
            { "dapurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
              "date" : "2015-10-23 09:37:33Z",
              "filesize" : "47.867M",
              "hasdap" : true,
              "hashttp" : true,
              "httpurl" : "https://localhost/impactportal/DAP/<userid>/tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
              "iconCls" : "typeOF",
              "id" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
              "index" : 2,
              "leaf" : true,
              "text" : "tasmax_day_CNRM-CM5_historical_r1i1p1_1890.nc",
              "type" : "file" 
            }
          ],
        "date" : "",
        "expanded" : true,
        "iconCls" : "typeFolder",
        "leaf" : false,
        "text" : "My data" 
      }
    ],
  "leaf" : false,
  "text" : "https://esg-dn1.nsc.liu.se/esgf-idp/openid/<openid>",
  "viewer" : "/impactportal/data/datasetviewer.jsp?" 
}

Remove a file from the basket

GET request:
https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?&service=basket&request=removeFromList&id[]=<id from basket request>

e.g.:
https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?&service=basket&request=removeFromList&id[]=INTER_OPER_R___RD3_____L3__20100101T000000_20100201T000000_0004.nc

To remove multiple files at once do:

https://dev.climate4impact.eu/impactportal/ImpactService?&service=basket&request=removeFromList&id[]=WPS_clipc_combine_execute_20160222T150058Z&id[]=WPS_clipc_combine_execute_20160222T150045Z&id[]=WPS_clipc_combine_execute_20160222T145836Z&id[]=WPS_clipc_combine_execute_20160222T145649Z&id[]=WPS_clipc_combine_execute_20160222T145633Z&id[]=WPS_clipc_combine_execute_20160222T145449Z&id[]=WPS_clipc_combine_execute_20160222T145323Z&id[]=WPS_clipc_combine_execute_20160222T145308Z&id[]=WPS_clipc_combine_execute_20160222T145230Z&id[]=WPS_clipc_combine_execute_20160222T145030Z&id[]=WPS_clipc_combine_execute_20160222T144854Z&id[]=WPS_clipc_combine_execute_20160222T144852Z&id[]=WPS_clipc_combine_execute_20160222T144704Z&id[]=WPS_clipc_combine_execute_20160222T144549Z

Add a file to the basket

GET request to https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?service=basket&mode=add&id=<file basename>&opendap=<opendap url>

E.g add

Example:

https://dev.climate4impact.eu/impactportal/ImpactService/<access_token>/?service=basket&mode=add&id=ice_conc_sh_polstere-100_multi_201604091200.nc&OPENDAP=http%3A%2F%2Fthredds.met.no%2Fthredds%2FdodsC%2Fosisaf%2Fmet.no%2Fice%2Fconc%2F2016%2F04%2Fice_conc_sh_polstere-100_multi_201604091200.nc

WPS Requests

Climate4impact uses PyWPS, see http://pywps.wald.intevation.org/documentation/course/ for more information

  • WPS can be used via
    https://dev.climate4impact.eu/impactportal/WPS/<accesstoken>/?

Example Python WPS client on WPS Process "binaryoperatoronnumbers"

The following Python example behaves like a minimal WPS client, using C4I services to perform analysis. The example below makes use of the binaryoperatorfornumbers process, which enables operations on two numbers. The process needs five seconds to run, listing progress in percentages. This is done to ease progress monitoring developments. The process is executed asynchronously and is executed with the GET method. PyWPS also supports POST, this can be used for posting bigger inputs. This is described here: WPS

The process has three inputs:
  1. inputa - a float number
  2. inputb - a float number
  3. operation - Can be "add", "substract", "divide" or "multiply"
#!/bin/python

""" 

Obtain an access token:
- Login with a google account at https://climate4impact.eu/impactportal/account/OAuth2.jsp 
- Generate via https://climate4impact.eu/impactportal/tokenapi?service=account&request=generatetoken
- Copy paste the accesstoken value in the "token" variable below.

Author: Maarten Plieger, KNMI, 2016-10-04
""" 

token="put your access token here" 

wpsendpoint="https://climate4impact.eu/impactportal/WPS/"+token+"/?" 

inputa=11;
inputb=22;
operator="divide" # Any of add, substract, multiply, divide

print "Calculating ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"]" 

import xml.etree.ElementTree as et
import urllib2
import time
from StringIO import StringIO

""" Parses URL to XML tree """ 
def getandparsexml(url):
  response = urllib2.urlopen(url)
  html = response.read()
  it = et.iterparse(StringIO(html))
  for _, el in it:
      if '}' in el.tag:
          el.tag = el.tag.split('}', 1)[1]  # strip all namespaces
  root = it.root
  return root

# Execute the ultimatequestionprocess aynchronousely and obtain the statusLocation for progress
datainputs = "inputa="+str(inputa)+";inputb="+str(inputb)+";operator="+str(operator)
root = getandparsexml(wpsendpoint+"service=WPS&version=1.0.0&request=execute&storeExecuteResponse=true&status=true&identifier=binaryoperatorfornumbers&datainputs="+datainputs)    
monitorURL = root.attrib['statusLocation']

print "Process executed checking " +monitorURL

# Get starttime of process, sometimes the monitorURL is not immediately available. Iterate until a valid creationtime is available would be good practice.
numberOfTries=10
while numberOfTries>0:
  numberOfTries = numberOfTries-1 
  try:
    root = getandparsexml(monitorURL)
    numberOfTries = -1
    print "Process started at "+str(root[1].attrib['creationTime'])
  except:
    time.sleep(1)
    pass

# Wait for process accepted to pass...
ProcessAcceptedFound = True

while ProcessAcceptedFound == True:
  root = getandparsexml(monitorURL)
  ProcessAcceptedFound = False
  for elem in root.iter(tag="ProcessAccepted"):
    ProcessAcceptedFound = True;
    print elem.tag, elem.attrib
  time.sleep(1)

# Now Monitor progress, polling the statuslocation frequently
percentage = 0
while percentage<100:
  root = getandparsexml(monitorURL)
  try:
    for elem in root.iter(tag="ProcessStarted"):
      percentage = float(elem.attrib['percentCompleted'])
    for elem in root.iter(tag="ProcessSucceeded"):
      percentage = 100
      break
  except:
    percentage=100.0
    pass
  print "Progress: "+str(percentage)+ " complete." 
  time.sleep(1)

# Obtain the answer  
root = getandparsexml(monitorURL)
try:
  print "The answer for ["+str(inputa)+" "+str(operator)+" "+str(inputb)+"] is "+(root[2][0][2][0].text)
except:
  print "Processing failed: "+root[1][0][0][0][0].text
  pass

The result is:

(pyvirtenv)c4m@climate4impact:~$ python c4i_testwps_binaryoperator.py 
Calculating [11 divide 22]
Process executed checking http://climate4impact.eu/impactportal/WPS?OUTPUT=/pywps-144559994232.xml
Process started at 2015-10-23T11:32:23Z
Progress: 20.0 complete.
Progress: 40.0 complete.
Progress: 60.0 complete.
Progress: 80.0 complete.
Progress: 100.0 complete.
The answer for [11 divide 22] is 0.5
(pyvirtenv)c4m@climate4impact:~$ 

WPS JobList requests

All submitted jobs are captured and stored in a per user joblist. The joblist can be obtained via:

A json with array called jobs is returned:

{
  jobs:[{
    "wpspostdata": ...,
    "creationdate": "2016-03-30 16:57:03Z",
    "wpsid": "clipc_combine_execute",
    "processid": "pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml",
    "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-a9840c38-f687-11e5-98b2-901b0e19e05f.xml",
    "progress": "ready", // Ready indicated completed job
    "error": false,
    "status": "ready" // Ready, running, failed
  },{
    "wpspostdata":  ...,
    "creationdate": "2016-03-30 17:01:36Z",
    "wpsid": "wps_simple_indice",
    "processid": "pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml",
    "statuslocation": "http://bhw485.knmi.nl:8280/impactportal/WPS?OUTPUT=/pywps-4c323e96-f688-11e5-817a-901b0e19e05f.xml",
    "progress": "0 %", //Percentage completed
    "error": false,
    "status": "running" // Ready, running, failed
  }]
}

A job can be removed from the list by doing:

Search API

https://climate4impact.eu/impactportal/esgfsearch?service=search&request=getfacets&query=project%3DCMIP5%26variable%3Dtas%26time_frequency%3Dday%26experiment%3Dhistorical%26&pagelimit=25&pagenumber=0x%x%

run_wps_from_client.py View - Runs a very simple WPS client on one of the services at climate4impact (2.86 KB) Maarten Plieger, 10/31/2016 03:14 PM