removing dependence on HDF4
I propose to get rid of the dependence on HDF4, at least for EC-Earth. This is very old software (HDF group recommends to switch to HDF5), and it clutters EC-Earth compilation configurations.
We can already drive the model with netCDF met fields, and still have the option of running with hdf met fields. The only reason we must compile with HDF4 is for few input files. There are only 12 of them distributed with EC-Earth:
cca-login4 ~/perm/ECE3-DATA  >>> find -L . -name "*.hdf" ./tm5/TM5_INPUT/boundary/ODIN/ODIN_CO_O3_ratio_4levels.hdf ./tm5/TM5_INPUT/boundary/ODIN/ODIN_Climatology_HNO3_O3_4levels.hdf ./tm5/TM5_INPUT/boundary/CH4_top/haloe_ch4vmr_91.hdf ./tm5/TM5_INPUT/natural_emissions/tracer/RN222_WMO2004.hdf ./tm5/TM5_INPUT/natural_emissions/reactive_gases/CH4/HYMN/CH4-natural-nonLPJ/CH4-N71-Sanderson-0000-sfc-glb100x100.hdf ./tm5/TM5_INPUT/natural_emissions/reactive_gases/CH4/HYMN/CH4-natural-nonLPJ/CH4-N40-Lambert-0000-sfc-glb100x100.hdf ./tm5/TM5_INPUT/natural_emissions/reactive_gases/CH4/HYMN/CH4-natural-nonLPJ/CH4-N70-Olson-0000-sfc-glb100x100.hdf ./tm5/TM5_INPUT/natural_emissions/reactive_gases/DMS/DMSland.hdf ./tm5/TM5_INPUT/natural_emissions/reactive_gases/DMS/DMSconc.hdf ./tm5/TM5_INPUT/land/lsmlai.hdf ./tm5/TM5_INPUT/land/soilph.hdf ./tm5/TM5_INPUT/land/landfraction.hdf
For the standalone version of the model, there are a lot more files, because of the stratospheric ozone data. Then there is also the case of the old "save file", which we should drop altogether. But for that we just need to get rid of the writing in parallel of the netcdf restart files first, because the only reason to keep old save file is for cases users cannot compile netcdf4 with parallel IO enabled. This can be done in the same time we fix the writing of the timeseries with Tommy.
Ok this is low priority, but to keep in mind. I think we could easily convert to netcdf the few HDF files still needed for EC-Earth.
#2 Updated by Philippe Le Sager over 1 year ago
As of r725, the netcdf restart files are not read/written in parallel anymore. Tommi, the same method can be applied to write your output more efficiently.
New hidden feature: if one uses istart=32 and a tracer is missing in the restart file, the program continues and assigns a very small non-zero value to the missing tracer. Useful when adding tracers, which was one of the two reasons to switch back to "save file", and is irrelevant now.
#4 Updated by Philippe Le Sager 10 months ago
- % Done changed from 20 to 10
My proposal is to keep the use of HDF4 for hdf meteo only, since it is still available. On the long term we may get rid of it, but it should not be a problem to keep it as an option for the time being. For everything else, we should not remove HDF (except few exceptions), but replace it by the equivalent netCDF, since most of the model functionality should remain. We can prioritize and go step-by-step:
- leave hdf meteo available [nothing to do]
- remove the old restart (a/k/a save file) functionality [remove]
- dumpfield (in toolbox.F90) [remove, but somebody will ask for it one day]
- convert MMIX into netCDF (although we do not use it in EC-Earth, so it could be optional and available only if compiled with hdf)
- convert budget into netCDF (this can be optional, i.e. requires HDF4)
- convert all hdf input (except hdf meteo) into netcdf [code and data]
- first limited set for EC-Earth
- extend to all other inputs
I have probably missed some other hdf files (particularly some output), but I think that's the general idea. It requires careful testing and will not happen in one day.
We probably need to rollback r835 and r834. I can do it if you want.