Feature #10241

Feature #10231: ERA5 meteo

fast track ERA5 pre-processing

Added by Philippe Le Sager 12 months ago. Updated 6 days ago.

In Progress
Target version:
Start date:
Due date:
% Done:



Testing / Processing ERA5 meteo at 1x1 / 137layers / 3hourly back to 1990.

comp_with_observations_overall.pdf (347 KB) Philippe Le Sager, 06/14/2019 10:20 AM

WINTER_2006.pdf (45.3 MB) Philippe Le Sager, 06/14/2019 10:27 AM


#1 Updated by Arjo Segers 16 days ago

  • Status changed from New to In Progress

Processed L137 on 1x1 for 1989-2018.
Data available in:
To be tested in TM simulation with full chemistry.

#2 Updated by Philippe Le Sager 15 days ago

  • % Done changed from 0 to 50
I made few changes to get the code working (see r1019 to r1023). I am able to run the full chemistry (CB05+M7) with both ERA-Interim and ERA5 with the ERA-5 branch. Using ERA-Interim I have the same results as in the trunk (good, nothing is broken). Walltime for the step run for a 10-day simulation is:
  • 46 minutes with ERA-Interim
  • 54 minutes with ERA-5

but these are highly variable on my system (even more with short runs due the extra weight given to IO). Longer runs needed for more useful numbers.

I will start a benchmark run.

#3 Updated by Philippe Le Sager 14 days ago

Would it be possible to also retrieve and process the wind speed at 10m instead of (or in addition to) u10m and v10m? We use the latter only to computer the former (see issue #11771).

#4 Updated by Philippe Le Sager 14 days ago

Arjo, I saw you added a "restart.ignore" key. What is it for? I cannot run two or more chunks in a row anymore, I wonder if this is the source of the problem.

#5 Updated by Philippe Le Sager 14 days ago

Philippe Le Sager wrote: added a "restart.ignore" key. What is it for? I cannot run two or more chunks in a row anymore, I wonder if this is the source of the problem.

Ok, found it. That new key defaults to True! Just need to make sure it defaults to False, since it is already set to T for ERA5 preprocessing.

After closer look, this new key looks like a quick way of avoiding writing save file (in case it is not compiled with HDF4) when

restart.write : F

This is ok for now, but we'll have to rethink the all logic around restarts. Note that option 32 for istart provides now all the functionality of a save file but with a restart file: remapping and missing tracers are handled.

#6 Updated by Arjo Segers 14 days ago

Think it was only introduced because I could not get the hdf lib working correctly ..

#7 Updated by Philippe Le Sager 8 days ago

  • % Done changed from 50 to 70

Spinup + benchmark runs with full chemistry (CB05+M7) have finished. First look at the performance:

  • 1y-spinup was 15% slower with ERA5 (spinup has no output)
  • 1y-benchmark was 9% slower with ERA5 (benchmarks have a lot of output)

A closer look at the profiling output shows differences in reading the met fields (in seconds, for one month run):

ERA5 ERA-In. Explanation
tmm readfield 2D 608.29 147.89 more data: every 1h for ERA5, 3 or 6h for ERA-I
tmm readfield 3D 1060.52 887.86 same amount of data, but internally compressed in case of ERA5

#8 Updated by Philippe Le Sager 6 days ago

The benchmark comparison between the two models for 2006 is available. While most of the comparisons look OK, there is a couple of serious issues that need to be resolved. Have a look at the O3 sondes comparisons in the comp_with_observations_overall.pdf (pages 9-14). And why is the solar zenith angle different between the two runs (see p.26 of WINTER_2006.pdf)?

#9 Updated by Arjo Segers 6 days ago

How exactly was sza written out? The internal timestepping in the model might be different, if sza is written out as average over all timesteps instead of instant fields at regular times then the result might be different between ei and ea.

#10 Updated by Philippe Le Sager 6 days ago

Yes indeed, SZA is weighted by the actual timestep:

phot_dat(region)%sza_av  = phot_dat(region)%sza_av  + float(ndyn)/float(ndyn_max) * sza
phot_dat(region)%nalb_av = phot_dat(region)%nalb_av + float(ndyn)/float(ndyn_max)

and sza_av/nalb_av is written to file. Ok the differences are quite small, so it make sense that is due to the different time step. Quite surprised by the pattern...probably logical though.

Also available in: Atom PDF