Project

General

Profile

Support #26241

[PyCAMA] Supply list of up to 12 parameters you want to follow over time

Added by Maarten Sneep 4 months ago. Updated 16 days ago.

Status:
In Progress
Priority:
Normal
Assignee:
Target version:
Phase E2 - PyCAMA 1.0
Start date:
06/16/2020
Due date:
09/04/2020
% Done:

0%


Description

For the time-dependent QC monitoring, we need a list of up to 12 parameters you want to follow over time. The PyCAMA user manual (https://dev.knmi.nl/projects/pycama/dmsf) has an overview of what is possible, but please don't hesitate to ask if you need examples. Because long term monitoring relies on stability, this list is only needed for the offline products.

The background to this issue will be shown in the progress meeting, on Tuesday 2020-06-16.

aerosol_index_340_380_mean_median_20180701_20200601.png View (66.3 KB) Maarten Sneep, 06/10/2020 11:30 AM

aerosol_index_340_380_histogram_20180701_20200601.png View (63.9 KB) Maarten Sneep, 06/10/2020 11:31 AM

20200616T1130_Msneep_L2QC.pdf (4.06 MB) Maarten Sneep, 07/09/2020 09:00 AM

cloud_fractions.png View (83.3 KB) Maarten Sneep, 08/26/2020 02:18 PM


Related issues

Related to PyCAMA - Support #26231: [PyCAMA] Verify configuration of PyCAMA for daily extractions New 06/16/2020 09/04/2020

History

#1 Updated by Maarten Sneep 4 months ago

  • Related to Support #26231: [PyCAMA] Verify configuration of PyCAMA for daily extractions added

#2 Updated by isabelle de smedt 4 months ago

Dear Maarten
I'm not sure to understand what you need compared to the list we had before.
Do you mean that we have to reduce the list to 12 parameters?
Do we start from scratch or from the excel sheet dating from Dec 2018?
Kind regards
Isabelle

#3 Updated by Maarten Sneep 4 months ago

isabelle de smedt wrote:

Dear Maarten
I'm not sure to understand what you need compared to the list we had before.
Do you mean that we have to reduce the list to 12 parameters?
Do we start from scratch or from the excel sheet dating from Dec 2018?

I will explain this issue in the progress meeting next Tuesday. This is only for the time-dependent display, and is a subset of the parameters available in the extractions. I hope that after the meeting this will be clear, if not ask questions then.

#4 Updated by Maarten Sneep 4 months ago

Examples of the types of figures we want to generate. Note that for line plots multiple traces can be specified, but make sure that they cover a similar range of values, as otherwise you just see horizontal lines. The stats are available for land and sea separately, as wel as for northern and southern hemisphere (so there are 5 variants in total).

Once we receive you suggestion we will prepare samples for you to check and fine-tune.

#5 Updated by Maarten Sneep 3 months ago

  • Assignee set to Jacques Claas

#6 Updated by Steven Compernolle 3 months ago

From Maartens presentation at the PMR it is said that this is an AI "For algorithm leads and algorithm specialists". However, I think that for the validation contributions, this time-dependent monitoring can also be very useful (as proven for AAI at each VAL telecon). So validation contributors should be at least aware of the selection of parameters, and perhaps can add suggestions.

Perhaps it is a good idea to share an excel sheet with suggested parameters per product?

Obvious choices are the main retrieved parameters themselves, and perhaps also the qa_value

HCHO product: HCHO column number density, slant HCHO column number density, qa_value
NO2 product: stratospheric, tropospheric and total column number density (summed), slant NO2 column number density, qa_value
CLOUD product:
- CAL: cloud fraction, cloud top height, cloud optical thickness, qa_value
- CRB: cloud fraction, cloud height, cloud albedo, qa_value
- a priori OCRA cloud fraction
FRESCO: cloud_fraction, cloud_height, qa_value

#7 Updated by Maarten Sneep 3 months ago

Hi Steven: as far as I'm concerned everyone is welcome to join in on the fun, but my primary contact are the people whole perform the weekly checks. I will assume that they maintain contact with the validation team for further suggestions. Note that I need more than just the parameter, I also need the aspect to be looked at (median value over time, time dependent histogram, …). For convenience I've attached the presentation from the PMR here.

#8 Updated by Athina Argyrouli about 1 month ago

  • Status changed from New to In Progress

The suggested 12 parameters (variable names in L2 CLOUD product) that we would like to follow over time are listed below:

1. Cloud fraction apriori (cloud_fraction_apriori) – mean value
2. Cloud fraction CRB (cloud_fraction_crb) – mean value
3. Cloud fraction CAL (cloud_fraction) – mean value
4. Cloud top height CAL (cloud_top_height) – mean value
5. Cloud optical thickness CAL (cloud_optical_thickness) – mean value
6. Cloud height CRB (cloud_height_crb) – mean value
7. Cloud albedo CRB (cloud_albedo_crb) – mean value
8. Continuum Reflectance Oxygen Aband (continuum_reflectance_oxygen_Aband) – mean value
9. Reflectances ocra (reflectances_ocra) 0 (blue color) – mean value
10. Reflectances ocra (reflectances_ocra) 1 (green color) – mean value
11. Fitted RMS CRB (fitted_root_mean_square_crb) – mean value (and if possible maximum value)
12. Fitted RMS CAL (fitted_root_mean_square) – mean value (and if possible maximum value)

#9 Updated by Maarten Sneep about 1 month ago

Hi Athina,

I've modified PyCAMA to allow for multiple variables in a single plot. This will allow us to combine the first three in a single plot, as shown below (this is a limited time, still downloading). The a priori cloud fraction isn't available in the current extractions, we will need the reprocessing for those.

Similarly both cloud heights can be combined as well, and the reflectances, and the RMS values.

The continuum_reflectance_oxygen_Aband parameter isn't extracted in the daily extractions, and can therefore not be plotted, if you want those you'll have to update #26231.

I would encourage you to also include a histogram plot, to monitor the full distribution of values. There is more to clouds than just a mean value.

#10 Updated by Athina Argyrouli about 1 month ago

Hi Maarten,

thank you for the update! The combined cloud fractions plot looks fine.
I thought that a histogram plot is not a possibility. Then, of course including all the moments of the distribution is welcome.

#11 Updated by Maarten Sneep about 1 month ago

Athina Argyrouli wrote:

Hi Maarten,

thank you for the update! The combined cloud fractions plot looks fine.
I thought that a histogram plot is not a possibility. Then, of course including all the moments of the distribution is welcome.

The histogram plot is similar to the plot below for AAI. We don't really extract all moments, we have a few percentiles (1, 5, 10, 16, 25, median, 75, 84, 90, 95 and 99%), mean, min, max, standard deviation, and interquartile range as possibly useful for cloud. These are collected globally, and for north, south, land and sea separately. We also have these for each row separately (for those not separated into north, south, land and sea), these can be plotted in a similar way as the histograms themselves. And we can plot the zonal mean values as a function of time, again on a colour scale.

#12 Updated by Alba Lorente Delgado 22 days ago

For CH4, we thought about the following time-dependent line plots, mean values:

*methane bias corrected
*methane precision
*chi squared NIR
*chi squared SWIR
*qa value

For the qa value, do you think is better the histogram plot rather than just line plot?

Would it be possible to add as well histogram plot on methane zonal mean values? Is it possible to select the latitude ranges or should just be NH and SH?

Thanks in advance,

SRON L2 Team.

#13 Updated by Tobias Borsdorff 22 days ago

For CO we would like to be as comparable as possible with the choices for CH4.
Hence the following line plots with time would be needed:

*mean CO (global)
*mean CO (zonal means NH,SH)
*CO precision
*chi squared SWIR
*qa value

Additionally it would be good to have a plot showing the intensity of the stripes in the CO data with time.
(Maybe the variability of the stripes with swath direction?)

#14 Updated by Maarten Sneep 22 days ago

Thank you.

Alba Lorente Delgado wrote:

For CH4, we thought about the following time-dependent line plots, mean values:

*methane bias corrected
*methane precision
*chi squared NIR
*chi squared SWIR
*qa value

For the qa value, do you think is better the histogram plot rather than just line plot?

Because we only assign 1, 0.4, or 0 from the methane processor (plus a few system values that occur only rarely), I'm not sure either is useful. A plot of the fraction of pixels processed successfully, and the fraction without warnings may be more useful. We may need to tweak the plots to get daily figures rather than by orbit, but this can be done.

Would it be possible to add as well histogram plot on methane zonal mean values? Is it possible to select the latitude ranges or should just be NH and SH?

No, we don't collect that information (histogram/stats for arbitrary latitude bands), just the mean is available. the subset we have is just SH and NH. We can plot a time-dependent zonal mean. We can adjust the resolution of the zonal mean if desired.

#15 Updated by Maarten Sneep 22 days ago

Tobias Borsdorff wrote:

For CO we would like to be as comparable as possible with the choices for CH4.
Hence the following line plots with time would be needed:

*mean CO (global)
*mean CO (zonal means NH,SH)
*CO precision
*chi squared SWIR
*qa value

The same remark applies here, let me know what you think.

Additionally it would be good to have a plot showing the intensity of the stripes in the CO data with time.
(Maybe the variability of the stripes with swath direction?)

Plotting the stripe-correction as a function of time is available. I suggest that we start with that, and evaluate if changes need to be made later.

#16 Updated by Alba Lorente Delgado 22 days ago

Thank you for the feedback.

Maarten Sneep wrote:

Thank you.

Alba Lorente Delgado wrote:

For CH4, we thought about the following time-dependent line plots, mean values:

*methane bias corrected
*methane precision
*chi squared NIR
*chi squared SWIR
*qa value

For the qa value, do you think is better the histogram plot rather than just line plot?

Because we only assign 1, 0.4, or 0 from the methane processor (plus a few system values that occur only rarely), I'm not sure either is useful. A plot of the fraction of pixels processed successfully, and the fraction without warnings may be more useful. We may need to tweak the plots to get daily figures rather than by orbit, but this can be done.

I agree; if the fraction of successes and w/o warnings is possible, that sounds fine.

Would it be possible to add as well histogram plot on methane zonal mean values? Is it possible to select the latitude ranges or should just be NH and SH?

No, we don't collect that information (histogram/stats for arbitrary latitude bands), just the mean is available. the subset we have is just SH and NH. We can plot a time-dependent zonal mean. We can adjust the resolution of the zonal mean if desired.

Then I think SH and NH histogram plot should be fine.

#17 Updated by isabelle de smedt 16 days ago

Hi
Here is the wish list for HCHO, from the most important to the less, depending on the possibilites.

  • qa_value: histogram (if possible, one for lands, one for oceans)
  • formaldehyde_tropospheric_vertical_column: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • formaldehyde_slant_column_corrected: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • formaldehyde_tropospheric_vertical_column_precision: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • formaldehyde_slant_column_density_window1: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • formaldehyde_tropospheric_vertical_column_correction: mean, 4 horizontal lines for NH, SH, lands and oceans
  • fitted_root_mean_square_win1: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • formaldehyde_tropospheric_air_mass_factor: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • formaldehyde_clear_air_mass_factor: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • integrated_formaldehyde_profile_apriori: mean, 4 horizontal lines for NH, SH, lands and oceans
in PRODUCT/SUPPORT_DATA/INPUT_DATA/BACKGROUND_CORRECTION, there are 2 variables with dimension 450 that I would like to monitor over time.
These fields change only once a day, and they are already averaged, but if possible, we need to keep the row dimension in a colorplot (x: time, y: row, color values).
  • PRODUCT/SUPPORT_DATA/INPUT_DATA/BACKGROUND_CORRECTION/offsets: can be called formaldehyde_slant_column_correction
  • PRODUCT/SUPPORT_DATA/INPUT_DATA/BACKGROUND_CORRECTION/offsets_scd0: can be called formaldehyde_model_background_slant_column_correction

Kind regards

Isabelle

#18 Updated by Nicolas Theys 16 days ago

Hi Maarten,

Inspired by HCHO, here is the list for SO2:
• qa_value: histogram
• sulfurdioxide_tropospheric_vertical_column: mean+std, 6 horizontal lines for 30°zonal bands
• sulfurdioxide _slant_column_corrected: mean+std, 6 horizontal lines for 30°zonal bands
• sulfurdioxide _tropospheric_vertical_column_precision: mean+std, 6 horizontal lines for 30°zonal bands
• fitted _slant_columns_wind1 (1st variable): mean+std, 6 horizontal lines for 30°zonal bands
• fitted_root_mean_square_win1: mean+std, 6 horizontal lines for 30°zonal bands
• sulfurdioxide _total_air_mass_factor_polluted: mean+std, 6 horizontal lines for 30°zonal bands
• sulfurdioxide _clear_air_mass_factor_polluted: mean+std, 6 horizontal lines for 30°zonal bands

If possible, equatorial Pacific (10°S-10°N, 160°W-240°W) timeseries colorplots (x: time, y: row, color values) for:

• fitted_root_mean_square: mean
• sulfurdioxide_slant_column_corrected : mean+std

in PRODUCT/SUPPORT_DATA/INPUT_DATA/BACKGROUND_CORRECTION, there are 2 variables with dimension 450 that I would like to monitor over time.
These fields change only once a day, and should be averaged over the o3_grid dimension, and displayed as colorplot (x: time, y: row, color values).
• PRODUCT/SUPPORT_DATA/INPUT_DATA/BACKGROUND_CORRECTION/window1_north can be called sulfurdioxide_slant_column_correction_north
• PRODUCT/SUPPORT_DATA/INPUT_DATA/BACKGROUND_CORRECTION/window1_south can be called sulfurdioxide_slant_column_correction_south

Best regards,
Nicolas

#19 Updated by Maarten Sneep 16 days ago

isabelle de smedt wrote:

Hi
Here is the wish list for HCHO, from the most important to the less, depending on the possibilites.

  • qa_value: histogram (if possible, one for lands, one for oceans)
  • formaldehyde_tropospheric_vertical_column: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • formaldehyde_slant_column_corrected: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • formaldehyde_tropospheric_vertical_column_precision: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • formaldehyde_slant_column_density_window1: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • formaldehyde_tropospheric_vertical_column_correction: mean, 4 horizontal lines for NH, SH, lands and oceans
  • fitted_root_mean_square_win1: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • formaldehyde_tropospheric_air_mass_factor: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • formaldehyde_clear_air_mass_factor: mean+std, 4 horizontal lines for NH, SH, lands and oceans
  • integrated_formaldehyde_profile_apriori: mean, 4 horizontal lines for NH, SH, lands and oceans

I'm going to see how we combine these, as values with similar order of magnitude will give a more readable result.

in PRODUCT/SUPPORT_DATA/INPUT_DATA/BACKGROUND_CORRECTION, there are 2 variables with dimension 450 that I would like to monitor over time.
These fields change only once a day, and they are already averaged, but if possible, we need to keep the row dimension in a colorplot (x: time, y: row, color values).
  • PRODUCT/SUPPORT_DATA/INPUT_DATA/BACKGROUND_CORRECTION/offsets: can be called formaldehyde_slant_column_correction
  • PRODUCT/SUPPORT_DATA/INPUT_DATA/BACKGROUND_CORRECTION/offsets_scd0: can be called formaldehyde_model_background_slant_column_correction

I have to check that we extract these variables, but this is otherwise a standard feature.

#20 Updated by Maarten Sneep 16 days ago

Nicolas Theys wrote:

Hi Maarten,

Inspired by HCHO, here is the list for SO2:
• qa_value: histogram
• sulfurdioxide_tropospheric_vertical_column: mean+std, 6 horizontal lines for 30°zonal bands
• sulfurdioxide _slant_column_corrected: mean+std, 6 horizontal lines for 30°zonal bands
• sulfurdioxide _tropospheric_vertical_column_precision: mean+std, 6 horizontal lines for 30°zonal bands
• fitted _slant_columns_wind1 (1st variable): mean+std, 6 horizontal lines for 30°zonal bands
• fitted_root_mean_square_win1: mean+std, 6 horizontal lines for 30°zonal bands
• sulfurdioxide _total_air_mass_factor_polluted: mean+std, 6 horizontal lines for 30°zonal bands
• sulfurdioxide _clear_air_mass_factor_polluted: mean+std, 6 horizontal lines for 30°zonal bands

30 degree zonal bands aren't extracted, so we can't do that. We have SH/NH only.

If possible, equatorial Pacific (10°S-10°N, 160°W-240°W) timeseries colorplots (x: time, y: row, color values) for:

• fitted_root_mean_square: mean
• sulfurdioxide_slant_column_corrected : mean+std

This isn't extracted, nor do we have facilities to extract this at the moment. Do you have this data available via the background correction in some way?

in PRODUCT/SUPPORT_DATA/INPUT_DATA/BACKGROUND_CORRECTION, there are 2 variables with dimension 450 that I would like to monitor over time.
These fields change only once a day, and should be averaged over the o3_grid dimension, and displayed as colorplot (x: time, y: row, color values).
• PRODUCT/SUPPORT_DATA/INPUT_DATA/BACKGROUND_CORRECTION/window1_north can be called sulfurdioxide_slant_column_correction_north
• PRODUCT/SUPPORT_DATA/INPUT_DATA/BACKGROUND_CORRECTION/window1_south can be called sulfurdioxide_slant_column_correction_south

At last, this is possible. I have to verify that we read these variables, but otherwise this shouldn't be an issue.

#21 Updated by Nicolas Theys 16 days ago

Hi Maarten,

30°bands: sorry I forgot it was not possible. Hemispherical (N and S) timeseries are fine.
Equatorial Pacific: no problem, I can live with it.

Nicolas

#22 Updated by Maarten Sneep 16 days ago

Nicolas Theys wrote:

30°bands: sorry I forgot it was not possible. Hemispherical (N and S) timeseries are fine.
Equatorial Pacific: no problem, I can live with it.

I think I can add a module to extract the mean value (row-independent) for a specified region, this might be useful for other processors as well as a background monitor. Doing the rows correctly for a region can be done, but requires quite some care (and might need to move with the seasons, as is done with offline analysis for NO2.

If more products are going to use this, then I can add such a module.

Also available in: Atom PDF