Project

General

Profile

Task #616

annoying warnings from udunits2 clog log

Added by Philippe Le Sager over 5 years ago. Updated almost 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
06/30/2016
Due date:
% Done:

100%


Description

On some system, udunits v1.x is not available and we implemented an interface to udunits v2.x. Some annoying warning messages are repeated and filling the output log. Excerpt:

Definition of "kt" in "/usr/local/apps/udunits/2.2.17/share/udunits/udunits2-common.xml", line 90, overrides prefixed-unit "1000000 kilogram" 
Definition of "microns" in "/usr/local/apps/udunits/2.2.17/share/udunits/udunits2-common.xml", line 336, overrides prefixed-unit "1e-15 second" 
Definition of "ft" in "/usr/local/apps/udunits/2.2.17/share/udunits/udunits2-common.xml", line 433, overrides prefixed-unit "1e-12 kilogram" 
Definition of "yd" in "/usr/local/apps/udunits/2.2.17/share/udunits/udunits2-common.xml", line 441, overrides prefixed-unit "8.64e-20 second" 
Definition of "pt" in "/usr/local/apps/udunits/2.2.17/share/udunits/udunits2-common.xml", line 655, overrides prefixed-unit "1e-09 kilogram" 
Definition of "at" in "/usr/local/apps/udunits/2.2.17/share/udunits/udunits2-common.xml", line 1052, overrides prefixed-unit "1e-15 kilogram" 
Definition of "ph" in "/usr/local/apps/udunits/2.2.17/share/udunits/udunits2-common.xml", line 1571, overrides prefixed-unit "3.6e-09 second" 
Definition of "nt" in "/usr/local/apps/udunits/2.2.17/share/udunits/udunits2-common.xml", line 1578, overrides prefixed-unit "1e-06 kilogram" 

Besides making a large log (15M for a 1-month full chemistry run), it may impend the performance of the model by eating some of the resources on the node. Note that this affects only standalone runs driven by netcdf files. I have the problem on the XC30 and XC40 Cray machines at ECMWF. A quick lookup brought me to:
http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg09715.html

which has some suggestion, but are not easy to implement here because it is about interfacing fortran with C. Anybody familiar with this? If not, I will mute the usage of udunits in TM5-MP.


Related issues

Related to TM5-MP - Bug #10191: Aquabug Closed 06/29/2018

History

#1 Updated by Philippe Le Sager over 5 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 90

I have commented the code that calls udunits to scale data, when reading the met fields produced by TM5. See r359. This was not needed.

#2 Updated by Philippe Le Sager over 3 years ago

  • Status changed from Closed to In Progress
  • % Done changed from 90 to 0

Need to go back to this issue. The land sea mask from netcdf met field is not correctly scaled anymore.

#3 Updated by Arjo Segers over 3 years ago

These warnings that UDUNITS produces look like:

  Definition of "kt" in "/path/to/udunits2-common.xml", line   90, overrides prefixed-unit "1000000 kilogram" 

To get rid of these messages, the database with unit information needs to be editted.
The original database is a set of xml files provided with the distribution, for example on sara/cartesius:

  /hpc/sw/udunits-2.1.24-intel/share/udunits/*.xml

The set of files should be copied to a local directory, for example:

  /projects/0/tm54dvar/admin/opt/udunits/2.1.24_shut-up/share/udunits/

To avoid the warning above, the alias settings in "udunits2-common.xml" at line 90:

            <aliases>
                <symbol>kt</symbol>
                <symbol>kts</symbol>
            </aliases>

should be commented to:

            <!-- <aliases>
                <symbol>kt</symbol>
                <symbol>kts</symbol>
            </aliases> -->

Repeat this for all error messages.

To use this new database, define the following environment variable and let it point to the top-level file of the units database:

  UDUNITS2_XML_PATH=opt/udunits/2.1.24_shut-up/share/udunits/udunits2.xml

On sara/cartesius, a version with appropriate comments is already available and could be loaded with the following settings in your '~/.bash_profile<' :

  # shared settings:
  . /projects/0/tm54dvar/admin/bash_profile

  # load default library:
  module load udunits/intel/2.1.24
  # replace units database by less annoying local version:
  module load udunits/2.1.24_shut-up

#4 Updated by Philippe Le Sager over 3 years ago

#5 Updated by Philippe Le Sager over 3 years ago

See #10191-2 for an updated estimate of the performance hit we got from udunits as it is implemented.

The Steering Committee on June 29th, 2018, has decided to make udunits an option, which could be used to test but not for production runs. This is part of the task 8.4 of the SC actions list.

A branch to cleanup the code, including implementing the optional udunits has been created: cleanup-udunits-hdf-ncep

#6 Updated by Philippe Le Sager over 3 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 90

Reintegration (r858) of the cleanup-udunits-hdf-ncep solves the issue at hand (see #8931-5): udunits is now optional. Need to update the User manual accordingly.

#7 Updated by Philippe Le Sager over 3 years ago

  • Status changed from Resolved to Closed
  • % Done changed from 90 to 100

Doc (r867) and rc files templates (r865) have been adapted for optional udunits.

#8 Updated by Philippe Le Sager almost 2 years ago

  • Target version set to TM5-MP 3.0

Also available in: Atom PDF