Project

General

Profile

Support #12131

PyCAMA test aborted when processing ALH

Added by Maarten Sneep 3 months ago. Updated 2 months ago.

Status:
Closed
Priority:
High
Assignee:
Target version:
-
Start date:
10/11/2018
Due date:
% Done:

100%


Description

Anomalous behaviour of PyCAMA was detected during the validation test of PyCAMA in the PDGS release 2.2.0 update. This crash happened while extracting data from ALH.

...
2018-10-11T14:34:57.350796 pc170012.knmi.nl PyCAMA 0.8.0 [0000015655]: [W] Variable 'wavelength_calibration_offset' contains no valid data in orbit 01192 (removing granule from all analysis)
2018-10-11T14:34:59.867584 pc170012.knmi.nl PyCAMA 0.8.0 [0000015655]: [W] Orbit 1192 contains no valid data
2018-10-11T14:34:59.867838 pc170012.knmi.nl PyCAMA 0.8.0 [0000015655]: [W] Caught an unexpected exception
2018-10-11T14:34:59.867887 pc170012.knmi.nl PyCAMA 0.8.0 [0000015655]: [E] zero-size array to reduction operation minimum which has no identity
Traceback (most recent call last):
  File "/usr/people/sneep/PyCAMA/pycama/src/pycama/CAMA_call.py", line 159, in main
    data.read()
  File "/usr/people/sneep/PyCAMA/pycama/src/pycama/Reader.py", line 1157, in read
    self.collect_aggregate_data()
  File "/usr/people/sneep/PyCAMA/pycama/src/pycama/Reader.py", line 990, in collect_aggregate_data
    if np.min(v.aggregate_data) == np.max(v.aggregate_data):
  File "/usr/people/sneep/.local/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 2417, in amin
    return amin(axis=axis, out=out, **kwargs)
  File "/usr/people/sneep/.local/lib/python3.6/site-packages/numpy/ma/core.py", line 5605, in min
    axis=axis, out=out, **kwargs).view(type(self))
  File "/usr/people/sneep/.local/lib/python3.6/site-packages/numpy/core/_methods.py", line 29, in _amin
    return umr_minimum(a, axis, None, out, keepdims)
ValueError: zero-size array to reduction operation minimum which has no identity

2018-10-11T14:34:59.941415 pc170012.knmi.nl PyCAMA 0.8.0 [0000015655]: [I] Processing time: 38.4 seconds
2018-10-11T14:34:59.941599 pc170012.knmi.nl PyCAMA 0.8.0 [0000015655]: [I] Errors: 1, warnings: 25
2018-10-11T14:34:59.941737 pc170012.knmi.nl PyCAMA 0.8.0 [0000015655]: [I] Stopping with exit code: 255

The issue could be reproduced, also in the current development version of PyCAMA. While the handling of the error can be improved, the cause is extracting data that is supposed to be empty (the wavelength calibration parameters). The in turn causes all data to be discarded, leading ultimately to the crash.

The solution is simple: a new configuration that does not try to extract empty parameters. This configuration file is attached here. I can make a configuration maintenance request for this file alone, or collect all updates - including the one for methane - and submit that later. Let me know how you want to proceed.

S5P_OPER_CFG_MPC_L2_00000000T000000_99999999T999999_20181011T144817.xml View - MD5 chcksum: 9ee491ff20dfc1339b90ca42787581f5 (177 KB) Maarten Sneep, 10/11/2018 02:54 PM

History

#1 Updated by PDGS Operations 3 months ago

The modified configuration file was tested successfully for OFFL and NRTI processing in the PDGS VAL environment (PDGS-internal tracking: S5P_SCR-180).

#2 Updated by Maarten Sneep 3 months ago

  • Status changed from In Progress to Resolved

Thanks for the feedback. Resolving issue here.

#3 Updated by Maarten Sneep 3 months ago

  • Project changed from Phase E1 to PyCAMA
  • Category deleted (PyCAMA)

#4 Updated by Maarten Sneep 2 months ago

  • Status changed from Resolved to Closed
  • % Done changed from 0 to 100

Also available in: Atom PDF