Bug #28981
[PyCAMA] Hard crash when processing NPP data
Start date:
11/27/2020
Due date:
% Done:
0%
Description
Prepared release 0.10.5 to fix an issue when processing NPP-VIIRS data.
2020-11-26T17:30:47.252852 s5p-val2-off-pn00 PyCAMA 0.10.2 [0000026634]: [E] [Errno -101] NetCDF: HDF error: b'/mnt/data1/storage_offl_l1b/pp_pycama/NPP/S5P_OFFL_L2__NP_BD3_20190510T081608_20190510T095738_08143_01_010100_20201125T083454.nc' Traceback (most recent call last): File "/data/sw/MPC/MPC_PYCAMA_V00.10.02/src/pycama/CAMA_call.py", line 169, in main data.read() File "/data/sw/MPC/MPC_PYCAMA_V00.10.02/src/pycama/Reader.py", line 1220, in read self.read_file(files[p][i], p, scanline_selection=selected_scanlines[p]) File "/data/sw/MPC/MPC_PYCAMA_V00.10.02/src/pycama/Reader.py", line 574, in read_file dim_names = f.dimension_names(v.primary_var) File "/data/sw/MPC/MPC_PYCAMA_V00.10.02/src/pycama/File.py", line 246, in dimension_names with netCDF4.Dataset(self.path, 'r') as ncref: File "netCDF4/_netCDF4.pyx", line 2321, in netCDF4._netCDF4.Dataset.__init__ File "netCDF4/_netCDF4.pyx", line 1885, in netCDF4._netCDF4._ensure_nc_success OSError: [Errno -101] NetCDF: HDF error: b'/mnt/data1/storage_offl_l1b/pp_pycama/NPP/S5P_OFFL_L2__NP_BD3_20190510T081608_20190510T095738_08143_01_010100_20201125T083454.nc'
This could be traced back to the separation between dimensions and the corresponding variables, as introduced by the L1B team. In the crashing code, I used a fallback to find the dimension names using the netCDF4
interface, while most of the reading is done via the h5py interface. I have now figured out how to achieve the same result using the h5py interface only.
Fixed in 0.10.5 (released today).