Project

General

Profile

WPS

WPS Installation

Test a pyWPS script from the commandline

With the following shell script and python script you can test any pyWPS process from the commandline. This can be used for debugging and developing new pyWPS processes.

file: testwps.sh:

export PYWPS_CFG=/home/wps/pywps/pywps.cfg
export PYWPS_PROCESSES=/home/wps/pywps/processes
export PYWPS_TEMPLATES=/home/wps/pywps-3.2.1/pywps/Templates
export NCARG_ROOT="/home/wps/software/install" 
export PORTAL_SCRIPTS=/home/wps/pywps/scripts/
export LD_LIBRARY_PATH=/home/wps/software/install/lib:$LD_LIBRARY_PATH
export PATH=/home/wps/software/install/bin:$PATH
export MPLCONFIGDIR=/home/wps/tmp/
. /home/wps/pywps/wps-virtenv/bin/activate
export PYTHONPATH=/usr/lib64/python2.6/site-packages/:$PYTHONPATH
export PYTHONPATH=/home/wps/pywps-3.2.1/:$PYTHONPATH
export PYTHONPATH=$PYWPS_PROCESSES:$PYTHONPATH
python testwps.py

file: testwps.py: (with testcdo_dtdp example)

import testcdo_dtdp as ProcessToTest
from pywps.Process import WPSProcess
from pywps.Process import Status

#Override status class and method in order to print to stdout directly.
class MyStatus(Status):
  def set(self,string,p):
    print(string)
status = MyStatus();

p=ProcessToTest.Process()
p.status=status

# Set input of pyWPS process
#p.Input_models.default="arpege1";
p.execute()

Start WPS job asynchronously

Add to the KVP URL: storeExecuteResponse=true&status=true
You can poll the location pointed with statusLocation to see the progress.

Use POST for asynchronous execute of WPS process

You can post the following XML:

<wps:Execute service="WPS" version="1.0.0" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0/wpsExecute_request.xsd">
      <ows:Identifier>testcdo_dtdp</ows:Identifier>

      <wps:ResponseForm>
        <wps:ResponseDocument storeExecuteResponse="true" status="true"> <!-- status = "false" makes the process synchronous -->
          <wps:Output asReference="false">
            <ows:Identifier>testcdo_dtdp</ows:Identifier>
          </wps:Output>
        </wps:ResponseDocument>
      </wps:ResponseForm>

      <wps:DataInputs>
        <wps:Input>
          <ows:Identifier>models</ows:Identifier>
          <wps:Data>
            <wps:LiteralData>arpege1,arpege2</wps:LiteralData>
          </wps:Data>
        </wps:Input>
        <wps:Input>
          <ows:Identifier>yeare</ows:Identifier>
          <wps:Data>
            <wps:LiteralData>2065</wps:LiteralData>
          </wps:Data>
        </wps:Input>
        <wps:Input>
          <ows:Identifier>input</ows:Identifier>
          <wps:Data>
            <wps:LiteralData>http://opendap.nmdc.eu/knmi/thredds/dodsC/IS-ENES/CERFACS/CERFACS-SCRATCH2010</wps:LiteralData>
          </wps:Data>
        </wps:Input>
        <wps:Input>
          <ows:Identifier>climb</ows:Identifier>
          <wps:Data>
            <wps:LiteralData>1961</wps:LiteralData>
          </wps:Data>
        </wps:Input>
        <wps:Input>
          <ows:Identifier>clime</ows:Identifier>
          <wps:Data>
            <wps:LiteralData>1990</wps:LiteralData>
          </wps:Data>
        </wps:Input>
        <wps:Input>
          <ows:Identifier>yearb</ows:Identifier>
          <wps:Data>
            <wps:LiteralData>2046</wps:LiteralData>
          </wps:Data>
        </wps:Input>
      </wps:DataInputs>
    </wps:Execute>

Patch NCL for unbuffered output

NCL needs to be adjusted in order to allow progress monitoring by a python thread. By default Ncl writes its output to a non-tty stdout in a buffered way, which does not allow a python thread to monitor its progress. By setting this buffer to zero, the output can be monitored by a python thread.

In ncl_ncarg-6.0.0/ni/src/ncl/Ncl.c, line 156 add the following:

/*Unbuffered output for progress monitoring*/
    setvbuf (stdout_fp, NULL, _IONBF, 0 );

Python script for asynchronous output monitoring

The following python script can be used to monitor the stdout of a executing program. This can be used inside pywps to write status reports asynchronousely to the user.

import sys
from subprocess import PIPE, Popen
from threading  import Thread

try:
    from Queue import Queue, Empty
except ImportError:
    from queue import Queue, Empty  # python 3.x

ON_POSIX = 'posix' in sys.builtin_module_names

def enqueue_output(out, queue):
    for line in iter(out.readline, b''):
        queue.put(line)
    out.close()

cmds=['ncl','indir=\"http://opendap.nmdc.eu/knmi/thredds/dodsC/IS-ENES/CERFACS/CERFACS-SCRATCH2010\"','outfile=\"dtdp\"','years=\"2046,2065\"','clim=\"1961,1990\"','models=\"arpege1,arpege2\"','generate_dtdp_clim.ncl']
p = Popen(cmds, stdout=PIPE, bufsize=0, close_fds=ON_POSIX)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True # thread dies with the program
t.start()

# read line without blocking
while True:
  try:  line = q.get_nowait() # or q.get(timeout=.1)
  except Empty:
      if(t.isAlive() == False):
        break;
  else: # got line
      print(line)

References