Project

General

Profile

Support #28421

Prepare for version 2 quality control monitoring.

Added by Maarten Sneep about 1 month ago. Updated 6 days ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Phase E2 - PyCAMA 1.0
Start date:
10/15/2020
Due date:
12/11/2020
% Done:

0%


Description

For some background we refer to the flow chart below. The current level 2 quality control is done on a daily basis only. This provides a lot of detail, but it is also rather time-consuming to look into each report, and monitoring trends is tricky, to say the least.

Processing flow chart.

We (Jacques Claas, Maarten Sneep, Mark ter Linden) have been working to improve this situation, by adding time dependent monitoring facilities. These plots are based on the daily extractions. What isn't extracted can't be monitored. For the offline monitoring we will combine the reprocessed data and the ongoing forward production to have a fully consistent dataset. This also means that adjusting the settings for the daily extractions must be done this year. Any change made later will introduce a break in the record, not ideal for time-dependent monitoring.

For NRTI monitoring we have to rely on whatever was extracted at the time of observation, what wasn't extracted at the time can't be monitored. As the L2 data is no longer avaiable, we can't run the daily extractions again. This means that either not all plots are available, or that plots aren't available for the full phase E2 of the mission. We will likely limit the plots in time, as in many cases the NRT monitoring is likely to be dominated by processor updates, especially where the NRT and OFFL algorithms differ and certainly by the upcoming L1B update. For monitoring the stability of the instrument, the offline dataset is more useful.

For simplicity these plots are generated on a daily basis using a "fixed" configuration. This configuration is different from the configuration of PyCAMA, but local and specific to the time dependent monitoring. It is not possible to request different plots yourself through the website, but for us it is fairly easy to update the configuration to have different plots the next day. However, "fairly easy" doesn't mean we're jumping for joy when a request comes in, we'd like to get this right, especially to make sure that the daily extractions are configured correctly.

One of the important issues has to do with data filtering. At the moment all data is used, provided that all variables contain a value that is not a fill-value. So they are synchronised and not filtered on the qa-value. Either or both can be changed. The synchronisation is per product, the filtering is either per product (when synchronised) or even per variable. The possibilities of PyCAMA are pretty wild, as the same variable can even be read twice: once with and once without QA-value selection. Please get in touch to discuss the options for your product, if you need guidance.

Here are the sample figures we generated for your product, based on 7 days of DDS2B data. This represents the future situation, after reprocessing. Fine-tuning can be done on both sides here: the configuration of the daily extractions and the configuration of the time-dependent plots. If your product isn't there, then we have no idea at the moment what to monitor.

We will upload later is what we can generate from our NRTI archive. Here we have limited room to update the time-dependent monitoring plots, some extractions or plot types may not be available at all.

The issues linked below are to communicate request for changes in specific areas.

S5P-L2-Flow-L2QC.png View - Processing flow chart. (188 KB) Maarten Sneep, 10/15/2020 10:03 AM

AER_AI.zip - AER_AI sample (969 KB) Maarten Sneep, 10/15/2020 10:06 AM

AER_LH.zip - AER_LH sample (158 KB) Maarten Sneep, 10/15/2020 10:06 AM

CH4___.zip - Methane sample (1.13 MB) Maarten Sneep, 10/15/2020 10:06 AM

CLOUD_.zip - Cloud sample (2.25 MB) Maarten Sneep, 10/15/2020 10:06 AM

CO____.zip - CO sample (1.22 MB) Maarten Sneep, 10/15/2020 10:06 AM

FRESCO.zip - FRESCO sample (1.54 MB) Maarten Sneep, 10/15/2020 10:06 AM

O3____.zip - Ozone sample (98.7 KB) Maarten Sneep, 10/15/2020 10:06 AM

HCHO__.zip - HCHO sample (1.76 MB) Maarten Sneep, 10/15/2020 10:06 AM

SO2___.zip - SO2 sample (2.11 MB) Maarten Sneep, 10/15/2020 10:06 AM


Related issues

Related to PyCAMA - Support #26241: [PyCAMA] Supply list of up to 12 parameters you want to follow over time In Progress 06/16/2020 12/11/2020
Related to PyCAMA - Support #27841: [PyCAMA] update filters for HCHO (possibly others) to exclude data with QA value < 0.5 New 09/09/2020 12/11/2020
Related to PyCAMA - Support #26231: [PyCAMA] Verify configuration of PyCAMA for daily extractions New 06/16/2020 12/11/2020
Related to PyCAMA - Support #28431: [PyCAMA] How long should the daily reports be available once the time-dependent monitoring is in place? Feedback 10/15/2020 12/11/2020

History

#1 Updated by Maarten Sneep about 1 month ago

  • Related to Support #26241: [PyCAMA] Supply list of up to 12 parameters you want to follow over time added

#2 Updated by Maarten Sneep about 1 month ago

  • Related to Support #27841: [PyCAMA] update filters for HCHO (possibly others) to exclude data with QA value < 0.5 added

#3 Updated by Maarten Sneep about 1 month ago

  • Related to Support #26231: [PyCAMA] Verify configuration of PyCAMA for daily extractions added

#4 Updated by Maarten Sneep about 1 month ago

  • Due date set to 12/11/2020
  • Assignee set to Jacques Claas

#5 Updated by Maarten Sneep about 1 month ago

  • Related to Support #28431: [PyCAMA] How long should the daily reports be available once the time-dependent monitoring is in place? added

#6 Updated by Maarten Sneep about 1 month ago

Mark already prepared a test page with some sample images http://mpc-l2.tropomi.eu/index-time.html# (click 'timeseries' in the top of the window).

#7 Updated by isabelle de smedt 13 days ago

Dear Maarten and Jacques

Thank you for the HCHO plots.

I find the wish list from issue 26241.
Only the variable "formaldehyde_slant_column_corrected" is not clear to me, because it is called slant column correction in the plot you sent. Did you apply a formula or is it simply the formaldehyde_slant_column_corrected field?

On the line plots; there are probably too many lines, so I would suggest to remove the North and South lines, and keep only the land and sea lines that make more sense for HCHO.

And as we discussed in issue 27841, I would like to apply the qa filtering, but only on the following variables: formaldehyde_tropospheric_vertical_column, formaldehyde_tropospheric_vertical_column_precision, formaldehyde_tropospheric_vertical_column_correction and formaldehyde_tropospheric_air_mass_factor.

This is for the current plots. I will have a look at what is done for the other molecules, to get some ideas.

Kind regards

Isabelle

#8 Updated by Klaus-Peter Heue 12 days ago

The ozone plot looks quite nice.
However, also some line plots as for the other trace gases would be nice to have.
From both scientific and quality control point of view an interesting plot would be lines for:
*Antartica: mean and min ozone columns with latitude south of 60°S
*Artic: comparable
*Norther and southern mid latitude mean ozone columns between 60 to 20
*Tropical band (20 south to 20 north) mean ozone column

Especially for the minimum in the polar regions the qa-filter would be essential.
A separation between land and sea is not needed for ozone
Is this feasible?

Also some plots on the QA value and the RMS as for SO2 or HCHO (figure0) are interesting time series

The surface albedo (effective_albedo for OFFL) as a function for latitude and time, comparable to the O3 columns

The viewing angle dependency (HCHO figure10) for the total ozone columns (ideally only for the tropical band)

#9 Updated by Maarten Sneep 12 days ago

Klaus-Peter Heue wrote:

The ozone plot looks quite nice.
However, also some line plots as for the other trace gases would be nice to have.
From both scientific and quality control point of view an interesting plot would be lines for:
*Antartica: mean and min ozone columns with latitude south of 60°S
*Artic: comparable
*Norther and southern mid latitude mean ozone columns between 60 to 20
*Tropical band (20 south to 20 north) mean ozone column

These parameters are not available as a daily extraction, not even as a module to handle this. I'm afraid that this is not possible. We only have NH/SH as a split, plus the zonal mean.

Especially for the minimum in the polar regions the qa-filter would be essential.

We can add a qa_filter for individual parameters, please be specific to which parameters these should be added (I assume the total vertical column only).

A separation between land and sea is not needed for ozone
Is this feasible?

Some features are one size fits all, so you'll get land/sea in the daily extractions regardless. Of course, for the time-dependent plot we can select what to use.

Also some plots on the QA value and the RMS as for SO2 or HCHO (figure0) are interesting time series

The surface albedo (effective_albedo for OFFL) as a function for latitude and time, comparable to the O3 columns

A zonal mean can be arranged.

The viewing angle dependency (HCHO figure10) for the total ozone columns (ideally only for the tropical band)

Latitude bands are limited, as indicated above.

I have to ask you to be specific. This is a two step process, for a time-series we need to extract the parameter in the daily processor, otherwise there is nothing to plot (issue #26231). Then we have the parameters for the time-dependent monitoring (issue #26241). From your response I understand that the answer for issue #27841 is 'yes, please'. May I ask you to put the requested information into the specific issues, as several parts will be handled by different people.

#10 Updated by Maarten Sneep 12 days ago

isabelle de smedt wrote:

Dear Maarten and Jacques

Thank you for the HCHO plots.

I find the wish list from issue 26241.
Only the variable "formaldehyde_slant_column_corrected" is not clear to me, because it is called slant column correction in the plot you sent. Did you apply a formula or is it simply the formaldehyde_slant_column_corrected field?

I have to double check. Will respond in #26241.

On the line plots; there are probably too many lines, so I would suggest to remove the North and South lines, and keep only the land and sea lines that make more sense for HCHO.

This can be done.

And as we discussed in issue 27841, I would like to apply the qa filtering, but only on the following variables: formaldehyde_tropospheric_vertical_column, formaldehyde_tropospheric_vertical_column_precision, formaldehyde_tropospheric_vertical_column_correction and formaldehyde_tropospheric_air_mass_factor.

Yes, I'm working on that. Still encountering some issues there.

This is for the current plots. I will have a look at what is done for the other molecules, to get some ideas.

Very good, that is part of the idea for these plots here.

#11 Updated by Christophe Lerot 11 days ago

For ozone OFFL, after discussion with Klaus-Peter, we would like to have time series plots for the following variables:
- ozone_total_vertical_column
- ozone_total_vertical_column_precision
- fitted_root_mean_square
- effective_albedo
- ozone_effective_temperature
- ozone_ghost_column

qa_values filtering can be applied to all of them.

It would be nice to have both color zonal/time plots (as the example in the 03.zip) and time lines for all of them. For the time lines, both the land/sea and hemisphere separations can provide complementary information.

Thank you

#12 Updated by Nicolas Theys 6 days ago

SO2:

  • Some graphs have too many lines. I would skip the “mean“ curves and keep only “medians”
  • For the plots where two variables are shown (eg corrected SO2 SCDs and SO2 SCD window1), would it be possible to keep the same colors (eg. North and North, South and South) and use solid and dashed lines instead. It is easier to read and compare.
  • QA > 0.5 filtering should be applied to plots for variables VCD and VCD precision, for the rest it is fine.
  • Figure SO2__timeseties-11.png looks odd. Here the filled values were not filtered out?
  • Figure SO2__timeseties-12.png could be improved. I would use the color scale from 1e-2 to 1e1.
  • Figure SO2__timeseties-13.png and Figure SO2__timeseties-14.png are empty.

Also available in: Atom PDF