Project

General

Profile

Wiki

This repository holds the CLIPC DRS checker (http://www.clipc.eu/).

It can be (1) used as commandline tool or (2) via a browser using the wizard at climate4impact: It is also available as WPS:

Basically the checker does the following:

  1. The checker validates the DRS attributes in the NetCDF file. All needs to be good.
  2. The checker composes the DRS filename based on the NetCDF attributes and compares it with the real filename. They need to be the same.
  3. The checker composes the DRS datasetname name, this can be handy for further data handling.

Obtain the checker as commandline tool

To obtain the DRS checker as commandline tool you can clone it from the repository:

hg clone https://dev.knmi.nl/hg/drschecker

A directory drschecker will be made with corresponding files.

If you do not have hg mercurial, you can download the latest drschecker version from:

Unzip and look at the files in given directory.

Update with latest changes

If you already have the checker but want to update your version, you can do:

cd drschecker
hg pull   # Syncs the repositories
hg update # Updates the files to the latest version or tip.

Running the checker on your NetCDF files

The checker can be used on a local NetCDF file or a remote opendap URL:

python ./drschecker/checkfiledrs.py --checkfile http://opendap.knmi.nl/knmi/thredds/dodsC/CLIPC/storyline_urbanheat/cerfacs/gsi_nco-4-4-8_CERFACS_multi-platform-tier2v1_day_20100111-20101221_1989-2010.nc

It will print a report for you, including the information whether your file is compliant, or otherwise which errors it has.

plieger@bhw485:./drschecker$ python ./drschecker/checkfiledrs.py --checkfile http://opendap.knmi.nl/knmi/thredds/dodsC/CLIPC/storyline_urbanheat/cerfacs/gsi_nco-4-4-8_CERFACS_multi-platform-tier2v1_day_20100111-20101221_1989-2010.nc
Using file [http://opendap.knmi.nl/knmi/thredds/dodsC/CLIPC/storyline_urbanheat/cerfacs/gsi_nco-4-4-8_CERFACS_multi-platform-tier2v1_day_20100111-20101221_1989-2010.nc]
[INFO]  Detected the CLIPC General DRS standard because source_data_id is set to [multi-platform-tier2v1].
[INFO]  Using CLIPCGeneralDRS standard
[OK]    Found           "comment" = "Original approach has been adapted to fit a structured fine-resolution grid of input data. At altitudes above 800 m and over water surfaces, the indicator is set to missing value." 
[OK]    Found           "invar_variable_name" = "tas,tasmin,SIS" 
[OK]    Found           "source_data_id" = "multi-platform-tier2v1" 
[OK]    Found           "package_name" = "nco-4-4-8" 
[OK]    Validated       "invar_tracking_id" = " " 
[OK]    Found           "invar_rcm_model_realization_id" = " " 
[OK]    Found           "references" = "Orlandi et al., 2013: Climatic indices in the interpretation of the phenological phases of the olive in mediterranean areas during its biological cycle. Climatic Change, 116:262-284. DOI:10.1007/s10584-012-0474-9" 
[OK]    Found           "contributor_role" = " " 
[OK]    Validated       "keywords" = "climate, mediterranean, olive, growing, season, index, Orlandi, observations, reanalysis, satellite, 0.05 degree" 
[OK]    Found           "invar_rcm_model_id" = " " 
[OK]    Validated       "geospatial_lat_max" = "47.5 degrees_nord" 
[OK]    Found           "invar_gcm_model_id" = " " 
[OK]    Found           "invar_ensemble_member" = " " 
[OK]    Validated       "title" = "Growing Season Index for olives in Mediterranean basin" 
[OK]    Found           "package_references" = "nco.sourceforge.net" 
[OK]    Found           "variable_name" = "gsi" 
[OK]    Found           "geospatial_bounds" = "Mediterranean basin bounded on the south" 
[OK]    Validated       "date_issued" = "20160613" 
[OK]    Validated       "geospatial_lat_min" = "31.1364 degrees_nord" 
[OK]    Validated       "geospatial_lat_resolution" = "0.05 degrees" 
[OK]    Validated       "time_coverage_start" = "20100111" 
[OK]    Validated       "geospatial_lon_max" = "42.7446 degrees_east" 
[OK]    Found           "institution_url" = "cerfacs.fr" 
[OK]    Validated       "geospatial_lon_min" = "-12.5022 degrees_east" 
[OK]    Validated       "date_modified" = " " 
[OK]    Found           "invar_satellite_sensor" = " " 
[OK]    Found           "invar_bc_method_id" = " " 
[OK]    Found           "summary" = "The GSI after Orlandi et al. (2013) is an indicator describing the combined effect of key climatic factors on the olive phenological development in the Mediterranean basin. The annual cycle of a 21-day moving average of the single daily GSI represents the olive growing season curve and follows a bimodal distribution." 
[OK]    Validated       "tracking_id" = "e38c163b-ab47-4cf8-b4d0-9fd8428d90ef" 
[OK]    Validated       "activity" = "clipc" 
[OK]    Found           "output_frequency" = "day" 
[OK]    Found           "cdm_datatype" = "Grid" 
[OK]    Found           "history" = " " 
[OK]    Found           "invar_reanalysis_id" = "EURO4M-MESAN_v1" 
[OK]    Found           "domain" = "EUR-05" 
[OK]    Found           "invar_platform" = "satellite" 
[OK]    Found           "invar_experiment_name" = " " 
[OK]    Found           "invar_bc_period" = " " 
[OK]    Found           "invar_platform_id" = "EUMETSAT CM-SAF" 
[OK]    Found           "Conventions" = "CLIPC-CCI-v1.0" 
[OK]    Found           "realisation_id" = "v1" 
[OK]    Found           "invar_bc_observation_id" = " " 
[OK]    Validated       "tile" = " " 
[OK]    Found           "product" = "obs_derived" 
[OK]    Found           "invar_rcm_model_driver" = " " 
[OK]    Found           "institution_id" = "CERFACS" 
[OK]    Validated       "reference_period" = "1989-2010" 
[OK]    Validated       "date_created" = "20160217" 
[OK]    Validated       "geospatial_lon_resolution" = "0.05 degrees" 
[OK]    Found           "contributor_name" = " " 
[OK]    Found           "time_coverage_end" = "20101221" 
[OK]    Found           "contact" = "christian.page@cerfacs.fr" 
[OK]    Found           "invar_satellite_algorithm" = " " 
[OK]    Found           "source_data_id_comment" = "Input datasets are issued from two sources. Air temperature input datasets are the regional re-analysis EUR4M-MESAN product while surface downwelling shortwave radiation is published by EUMETSAT CM-SAF" 
[INFO]  Attribute [tilennnnn]/[tile] is not filled in
[INFO]  Used DRS file mapping: <VariableName>_<package>_<institution>_<sourceDataID>_<frequency>_<StartTime-EndTime>_<Reference_period>
[INFO]  DRS Filename: gsi_nco-4-4-8_CERFACS_multi-platform-tier2v1_day_20100111-20101221_1989-2010.nc
[OK]    Filename matches gsi_nco-4-4-8_CERFACS_multi-platform-tier2v1_day_20100111-20101221_1989-2010.nc
[INFO]  Used DRS dataset mapping: <activity>.<product>.<package>.<domain>.<institution>.<sourceDataID>.<frequency>.<Reference_period>.<VariableName>
[INFO]  DRS Dataset: clipc.obs_derived.nco-4-4-8.EUR-05.CERFACS.multi-platform-tier2v1.day.1989-2010.gsi
[INFO]  Nr of found errors: 0
[DRSFilename]   [gsi_nco-4-4-8_CERFACS_multi-platform-tier2v1_day_20100111-20101221_1989-2010.nc]
[DRSDatasetname][clipc.obs_derived.nco-4-4-8.EUR-05.CERFACS.multi-platform-tier2v1.day.1989-2010.gsi]
[GOOD]  File has been validated with zero errors! Well done!

-------------------------
Derived datasetDRS:
clipc.obs_derived.nco-4-4-8.EUR-05.CERFACS.multi-platform-tier2v1.day.1989-2010.gsi
Derived filenameDRS:
gsi_nco-4-4-8_CERFACS_multi-platform-tier2v1_day_20100111-20101221_1989-2010.nc
-------------------------

Checking exit code

Besides the provided report, the exit code can be checked programmatically as well. The exit code will be nonzero if the file is non DRS conformant. In bash this code can be utilized:

printf '%o\n' $?

This can be useful for automated testing of files. It is also possible to scan a directory as shown below:

Running the checker on a directory

You can check a directory with subdirecties, etc.. All files with the extension nc will be scanned. Use -d or --checkdir

  • [!!!!] Indicates that the file is not readable by the NetCDF library
  • [FAIL] Indicates the file is not compliant by the CLIPC DRS
  • [ OK ] All good!
plieger@bhw485:./drschecker$ python ./drschecker/checkfiledrs.py -d ~/mounts/bhlnmis_nmis03/CLIPC/syke/
[FAIL] [/usr/people/plieger/mounts/bhlnmis_nmis03/CLIPC/syke/tier1/MoD/MoD_Matlab-8-1_SYKE_CryoLand-FSC_yr_20010101-20151231.nc]
[ OK ] [/usr/people/plieger/mounts/bhlnmis_nmis03/CLIPC/syke/tier2/MPI/MPI_R-nmle-3-1-121_SYKE_gdd5-133_yr_20010101-20131231.nc]
[ OK ] [/usr/people/plieger/mounts/bhlnmis_nmis03/CLIPC/syke/tier2/MPI/MPI_R-nmle-3-1-121_SYKE_clipc-MoD_yr_20010101-20131231.nc]

PS: sshfs mounts becomes really handy for scanning remote file systems:

mkdir -p <some local place>/bhlnmis_nmis03
sshfs -o Ciphers=arcfour -o Compression=no <user>@<machine>:/data/nmis03 <some local place>/bhlnmis_nmis03

CLIPC DRS checker wizard at climate4impact

The tool can also be used via a browser at https://climate4impact.eu/:

At climate4impact you can upload your own files to your basket and check them there.

Here is a screenshot of a compliant file:

DRS Checker wizard at climate4impact

Here is a screenshot of a file which needed some tuning:

c4i_drschecker.png View - DRS Checker wizard at climate4impact (152 KB) Maarten Plieger, 08/04/2016 12:58 PM

c4i_drschecker-errors.png View (136 KB) Maarten Plieger, 08/04/2016 01:06 PM