Project

General

Profile

History

Preinstalled virtual environment with ADAGUC Viewer and ADAGUC Server

For the workshop a preinstalled virtual machine is available. This machine has apache-tomcat, postgres, adaguc-services, adaguc-server, adaguc-viewer and apache httpd with php and cgi support installed.

The installation instructions can be found here: Install_Adaguc_on_Ubuntu

The following repositories are used:

Installation of the virtual image

  1. The virtual image can be downloaded from http://geoservices.knmi.nl/adagucworkshop/adaguc-xubuntu-2017.ova (8.5 Gb, when unpacked 20 Gb)
  2. Install Oracle Virtualbox with the corresponding extension pack from https://www.virtualbox.org/wiki/Downloads
  3. Start Virtualbox, click File -> Import appliance. In the settings, set the memory as high as possible, by default it is set to 2048Mb, but 4096Mb is recommended.
  4. In the VM, just start firefox and the adaguc-viewer will show as default page
  5. From the viewer, the AutoWMS plugin can be opened via clicking on the big gear and selecting AutoWMS from the list. A window on the right opens, displaying all files and folders available.

The VM exposes port 80 and 8080 to localhost, which means that links should work outside of the VM on the host as well.
Use the password "adaguc" to obtain sudo rights

Update the VM to the 2018 version

For the 2018 Workshop, you need to update the ADAGUC software suite. You do not need to upgrade the VM to the latest Ubuntu.

Update adaguc-dataset examples

#
cd /data/adaguc-datasets
git pull
#

Update adaguc-server

#Install libraries for testing purposes
echo "adaguc" | sudo -S apt-get update -y
sudo apt-get install python-lxml python-netcdf4 -y

#Pull and build adaguc-server
cd /src/KNMI/adaguc-server/
git checkout .
git pull
bash compile.sh
bash runtests.sh

#Re-scan all datasets
sudo mkdir /adaguc /servicehealth
sudo chown adaguc: /servicehealth
sudo ln -s /src/KNMI/adaguc-server/ /adaguc/adaguc-server-master
sudo ln -s /src/KNMI/adaguc-server/data/config/adaguc.vm.xml /adaguc/adaguc-server-config.xml

dropdb adaguc
createdb adaguc
bash /src/KNMI/adaguc-server/Docker/adaguc-server-updatedatasets.sh

#

Update adaguc-services

cd /src/KNMI/adaguc-services/
rm -rf target
git pull
mvn package
cp target/adaguc-services-*.war /src/apache-tomcat-8.0.44/webapps/adaguc-services.war 
cd /src/apache-tomcat-8.0.44/bin
. ../setenv.sh 
bash shutdown.sh && sleep 1 && bash startup.sh

#

Update adaguc-viewer which is running in standard APACHE HTTP server with PHP as backend

cd /var/www/html/adaguc-viewer
git checkout .
git pull
echo "xml2jsonrequestURL = 'http://localhost/adaguc-viewer/webmapjs_php/xml2jsonrequest.php?';" \
     "autowmsURL = 'http://localhost:8080/adaguc-services/autowms?';" \
     "getFeatureInfoApplications.push({name:'AutoWMS',iconCls:'button_getfeatureinfo'});" >> ./config.js

firefox http://localhost/adaguc-viewer/

#Apply the same steps for the adaguc-viewer which is running in APACHE TOMCAT with adaguc-services as backend

cd /src/apache-tomcat-8.0.44/webapps/adaguc-viewer/
git checkout .
git pull
echo "xml2jsonrequestURL = '/adaguc-services/xml2json?';" \
     "autowmsURL = '/adaguc-services/autowms?';" \
     "getFeatureInfoApplications.push({name:'AutoWMS',iconCls:'button_getfeatureinfo'});" >> ./config.js

firefox http://localhost:8080/adaguc-viewer/

# Please note: You can add      "getFeatureInfoApplications.open = 'AutoWMS';"  to make the viewe open the AutoWMS viewer by default.

#

Optionally Update docker. Only needed if you would like to build the docker yourself

sudo apt-get update
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable" 
sudo apt-get update
sudo apt install -y docker-ce=17.12.0~ce-0~ubuntu

sudo groupadd docker
sudo usermod -aG docker $USER
sudo systemctl enable docker

sudo curl -L "https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version
# Reboot
docker system prune -a

Install aws-cli

#
sudo apt install python-pip
pip install awscli
aws s3 ls noaa-goes16/ --no-sign-request
#

Important directories and files

  • /src/KNMI/adaguc-server - Git repo where adaguc-server is installed, compiled with gcc, and location where default configuration files are stored. Compiled with "bash compile.sh"
  • /src/KNMI/adaguc-services - Git repo where adaguc-services is installed, build with maven and deployed in webapps dir
  • /src/config/adaguc-services-config-tomcat.xml - Adaguc-services configuration file. Note: this is the tomcat web application to run adaguc-server.
  • /src/KNMI/adaguc-server/data/config/adaguc.vm.xml - Adaguc-server configuration file, this file is in the adaguc-server repository on github
  • /src/log - Log file directory
  • /src/tomcatstarter.sh - Tomcat startup script
  • /src/crontab - Crontab to start tomcat
  • /src/apache-tomcat-8.0.44 - http://localhost:8080/ - Home of apache tomcat
    • /src/apache-tomcat-8.0.44/webapps/adaguc-viewer - Adaguc viewer served with Apache tomcat. Specific configuration settings are appended to config.js:
    • /src/apache-tomcat-8.0.44/webapps/adaguc-services
  • /data/adaguc-autowms - Location where NetCDF files can be stored and are automatically available in a WMS via the source= kvp (http://localhost:8080/adaguc-services/adagucserver?source=<URL Encoded NetCDF file location)
  • /data/adaguc-datasets - Location where Adaguc dataset configurations are stored. Examples below.
  • /var/www/html - http://localhost/ - Documentroot of Apache http server
    • /var/www/html/adaguc-viewer - Adaguc viewer served with Apache HTTP
  • /var/www/cgi-bin
  • http://localhost:8080/adaguc-services/adagucserver? - Adaguc server via adaguc-services and apache tomcat
  • http://localhost/cgi-bin/adaguc-datasets.cgi? - Adaguc server via CGI and apache HTTP server

Adaguc viewer

The viewer is located at:
http://localhost:8080/adaguc-viewer/

The AutoWMS plugin can be opened via clicking on the big gear and selecting AutoWMS from the list. A window on the right opens, displaying all files and folders available.

Example NetCDF files

To update example NetCDF files, please run:

cd /data/adaguc-autowms/testsets
./update.sh

This will load new files from the KDC. (https://data.knmi.nl)

Links to dataset configuration examples:

Radar - RADNL_OPER_R___25PCPRR_L3: Himawari - adaguc.tiled: TU Delft / RWS traffic demo - tud_traffic: Last 100 earthquakes in NL - urn_xkdc_ds_nl.knmi__aardbevingen_nederland_2_ Actual 10 minute data from KNMI automated weather stations (Run cd /data/adaguc-autowms/testsets && ./update.sh first to get actual data from KDC) - urn_xkdc_ds_nl.knmi__Actuele10mindataKNMIstations_1_ Climatology of automated weather stations 1950-present - urn_xkdc_ds_nl.knmi__etmaalgegevensKNMIstations_1_: Demo dataset - dataset_a: Masking via DataPostProcessor - datapostproc_mask:

Viewer examples

The viewer has several examples located at http://localhost/adaguc-viewer/examples/

To see current radar images, go to:

Opendap Server

Besides being an OpenDAP client, adaguc-server has the capability to serve OpenDAP by itself. It can serve a set of files and concatenate those virtually over dimensions, e.g. aggregations are possible. To demonstrate this go to:

ncview http://localhost:8080/adaguc-services/adagucopendap/RADNL_OPER_R___25PCPRR_L3/

You can .das or .dds as extension

You can also dump a header of the file:

ncdump -h  http://localhost:8080/adaguc-services/adagucopendap/RADNL_OPER_R___25PCPRR_L3/

To view an OpenDAP URL with ncview, do:

ncview  http://localhost:8080/adaguc-services/adagucopendap/RADNL_OPER_R___25PCPRR_L3/

To view an OpenDAP URL with adaguc-server, do

http://localhost:8080/adaguc-services/adagucserver?source=http://localhost:8080/adaguc-services/adagucopendap/RADNL_OPER_R___25PCPRR_L3/&service=wms&request=getcapabilities

In the viewer:
http://localhost:8080/adaguc-viewer/?srs=EPSG%3A4326&bbox=-0.10856446,46.581272685142864,10.96501046,58.28762331485714&service=http%3A%2F%2Flocalhost%3A8080%2Fadaguc-services%2Fadagucserver%3Fsource%3Dhttp%3A%2F%2Flocalhost%3A8080%2Fadaguc-services%2Fadagucopendap%2FRADNL_OPER_R___25PCPRR_L3%2F%26service%3Dwms%26request%3Dgetcapabilities&layer=precipitation%24image%2Fpng%24true%24auto%2Fnearest%241%240&selected=0&dims=time$2015-06-05T15:55:00Z&baselayers=naturalearth2$ne_10m_admin_0_countries_simplified

Adaguc as OpenDAP client