Project

General

Profile

Feature #945

Switch from netCDF4 to h5py

Added by Maarten Sneep about 5 years ago. Updated almost 5 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
Target version:
-
Start date:
10/07/2016
Due date:
% Done:

100%


Description

To avoid serious performance instability, switch away from the netCDF4 python module and use h5py instead.

After upgrading to Fedora 24, with python 3.5.1, numpy 1.11.0 and netCDF4 1.2.2 performance of PyCAMA dropped significantly compared to python 3.4.3, numpy 1.9.2 and netCDF4 1.1.5. Further investigation showed that the performance drop occurred in the write modules.

Performance of the netCDF4 library has been iffy at times, and this is a good excuse to switch to a more stable system.

History

#1 Updated by Maarten Sneep about 5 years ago

  • Status changed from New to In Progress
  • Priority changed from Normal to Urgent

#2 Updated by Maarten Sneep about 5 years ago

  • % Done changed from 0 to 50

Writing is complete. Next up: ingesting what I just wrote.

#3 Updated by Maarten Sneep about 5 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 50 to 100

Both writing an reading is now completely transferred from netCDF4 to h5py. Note that netCDF4 is still imported. Only the date2num and num2date functions are used now.

#4 Updated by Maarten Sneep almost 5 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF