The HDF Group Introduction to netCDF-4 Elena Pourmal The HDF Group 110/17/2015

Overview 10/17/20152 What is netCDF? Data model and formats Ecosystem

WHAT IS NETCDF? 310/17/2015

Background 10/17/20154 Developed and maintained by Unidata Mission: provide data services, tools, and community leadership to advance Earth system science, enhance educational opportunities, and broaden participation Funded by NSF through UCAR Open source software

What is netCDF? 10/17/20155 netCDF = network Common Data Form Data model Scientific data (especially suitable for gridded data) Widely used in ocean and atmospheric science, climate and weather modeling Other disciplines: molecular dynamics, fusion research, medical imaging File FORMAT Portable, self-describing, etc. Not just one file format (netcdf, HDF4, HDF5, OPeNDAP, etc.) Application programming interfaces (APIs) C, Java, C++, Fortran Python, Ruby, Perl, MATLAB, IDL

History of netCDF 10/17/ – first release of netCDF 1991 – netCDF – HDFv 3.3 reads netCDF files 2004 – netCDF 3.6 supports 64-bit offsets 2008 – netCDF-4 based on HDF and later – parallel, OPeNDAP, HDF4

NETCDF DATA MODEL AND FORMATS 710/17/2015

netCDF classic data model 10/17/20158 Variables Name, shape, type N-dim arrays Dimension Name, length Attributes Name, type, value

netCDF classic data model Data model elements: Variables Name, shape, type N-dim arrays Dimensions Name, length Can be shared Attributes Name, type, value Model limitations: Atomic types only One extendible dimension Flat structure Model advantages: Widely adopted Best practices and conventions 10/17/20159

netCDF classic format Pros: Simple (header with metadata, raw data) Suitable for parallel access (pnetcdf from Argonne) Aside: could easily support SWMR Cons: No support for compression No extensibility in multiple dimension XDR-based (big-endian) Costly to add more variables 10/17/201510

netCDF enhanced data model Data model elements: Variables Dimensions Attributes Groups Model limitations: Complex Harder to use and not widely adopted Emerging best practices and conventions Model advantages: Rich collection of datatypes Hierarchical data organization 10/17/201511

Example of ncdump output netcdf OMI-Aura_L2-example { dimensions: PRESSURE = 18 ; DATETIME = 2 ; independent_465 = 465 ; independent_22 = 22 ; independent_7 = 7 ; independent_2 = 2 ; variables: float PRESSURE(PRESSURE) ; PRESSURE:MissingValue = -999.f ; PRESSURE:standard_name = "air_pressure" ; PRESSURE:VAR_NAME = "PRESSURE" ; PRESSURE:VAR_DESCRIPTION = "Pressure at retrieval layer" ; PRESSURE:units = "hPa" ; PRESSURE:valid_range = -Infinityf, Infinityf ; PRESSURE:VAR_DEPEND = "PRESSURE" ; short APrioriCovarianceMatrix(DATETIME, independent_465) ; APrioriCovarianceMatrix:MissingValue = s ; APrioriCovarianceMatrix:UniqueFieldDefinition = "OMI-Specific" ; APrioriCovarianceMatrix:Offset = 0. ; 10/17/201512

Example of h5dump output /PRESSURE Dataset {18/18} Attribute: CLASS scalar Type: 16-byte null-terminated ASCII string Data: "DIMENSION_SCALE" Attribute: MissingValue {1} Type: native float Data: -999 Attribute: NAME scalar Type: 9-byte null-terminated ASCII string Data: "PRESSURE" Attribute: REFERENCE_LIST {9} Type: struct { "dataset" +0 object reference "dimension" +8 native int } 16 bytes Data: (0) {DATASET-1:14192, 1}, {DATASET-1:14192, 2}, {DATASET-1:30064, 1}, {DATASET-1:32126, 1}, {DATASET-1:32472, 1}, {DATASET-1:34237, 1}, (6) {DATASET-1:49807, 1}, {DATASET-1:55542, 1}, {DATASET-1:62105, 1} 10/17/201513

netCDF-4 - netCDF extended format and library netCDF-4 uses HDF5 as a storage layer Pros: Compression Chunking storage Parallel I/O Extensibility (more objects can be added easily) No limitation on number objects Cons: Complex as HDF5 Clunky support for dimensions Not easy to tune for performance since some HDF5 parameters are hidden 10/17/201514

netCDF-4 Architecture 10/17/ HDF5 Library netCDF-4Library netCDF-3 Interface netCDF-3 applications netCDF-3 applications netCDF-4 applications netCDF-4 applications HDF5 applications HDF5 applications netCDF files netCDF files netCDF-4 HDF5 files HDF5 files

NETCDF ECOSYSTEM Real Strength 1610/17/2015

Ecosystem 10/17/201517

Power of CF conventions 10/17/ NPP data visualized with IDV Without CF metadata With CF metadata and dimensions

19

Questions 10/17/ ? Thank you!