Some internal interfaces were changed, and the output format was changed in a few details.
- The keyword parameters for the filter functions are no longer merged per variable. Instead they are taken care of by individual filter objects. This means that the same filter can be used multiple times on a single variable. It also means that keys do not have to be unique. In short: this change probably avoids a lot of headaches.
- Event counters are now stored per granule, rather than aggregated per day. This means that the granularity of the event counters is now finer in the MPC, allowing for a more fine-grained support. Note that this required a change in the output format, previous PyCAMA output files can not be read anymore.
- Doxygen comments have been added, and a Doxygen configuration file has been added (for Doxygen 1.8.13).
- A separate script has been added which can generate a report based on the PyCAMA netCDF file (without requiring a job order file).
The manual has been updated to 4.1.0 and is available in the DMSF.
A short summary of the resolved issue can be found below
- The installation issue #1155 reported by Lidia has been resolved.
- Variables without valid data in a granule lead to a variable length mismatch. This has been resolved.
- An option has been added to exclude parameters from the scatter density plots.
- A division by 0 error has been removed (could be triggered by a run with very few variables.
- Several filters were added, including a Debug filter to print information about variables as they pass through the filter system.
- Comparing different versions of the same product is now fully supported.
- The number of warnings that is allowed before an error is generated has been increased.
- In the
create_pycama_joborder.py script an option has been added to add a label to the output file names.
- A summary line can be added to the joborder by adding a dynamic processing parameter to the job order.
The manual has been updated to release 4.0.0 (see
The update is a response to feedback from the PDGS. parsing of the joborder has been improved, a clearer error message is now generated when unexpected values are encountered or when a job order with the wrong structure is presented.
This update has a new configuration file, which can process the new files generated by the recently updated DLR-L2 UPAS processor. This version can also process NPP-VIIRS files generated by the RAL processor.
The version number update means that the version numbers in hte JobOrder file and the configuration file must be updated (we only check for consistency in the first two digits).
This new release supports OMI L2 files as well as Sentinel 5 precursor files. We use a fair amount of metadata, which means that attributes in either format are required to reconstruct the metadata. This may require some significant amount of code to support in your own files. I'll try to isolate these parts in an upcoming release to make it easier to support PyCAMA in your own files, or for instance the QA4ECV files.
We now have a setup.py script, making it easier to install PyCAMA.
The provided configuration is work in progress. Two events are happening at the same time:
- The L2 processors are using new variable names. The DLR L2 processors have been delivered already, the KNMI processors are currently updated. This requires changes to the configuration file.
- The configuration, in particular for the DLR L2 products has to be refined further with filters to select the desired data.
You are encouraged to experiment with the software and report issues you may find.
I've pushed the current development state out to the repository. I consider this a "working beta" release. Parts of the code are untested, in particular the data transformers (in particular column integration) and some unsynchronized data handling, but it should be good to use on test data (for instance GOME-2 based data from VT-07B).
The installation procedure is not finalized at this point. For the development version it is easiest to add pycama/src to your $PYTHONPATH environment variable. The run.sh script in test/cases/SYS-001-001-FRESCO-OFFL/ gives an example on how to call PyCAMA. I've yet to find a solution to include actual input test data, so the test case currently doesn't work.
The software user manual is available in the DMSF of the project. A more specific question will follow within a few weeks.
You can obtain the new version from your current repository using
hg pull
hg update
If you don't already have a clone of our repository, then you can get a repository using
hg clone https://dev.knmi.nl/hg/pycama
Enjoy.
To give a preview of what PyCAMA can do I provide access to the current development status of PyCAMA. Note that this is very much work in progress. The configuration is currently done directly in a Python script, as a declarative file. See cama.py in the src directory.
The code can be accessed using Mercurial: hg clone https://dev.knmi.nl/hg/pycama