NetCDF: Data Model, Programming Interfaces, Conventions and Format Adapted from Presentations by Russ Rew Unidata Program Center University Corporation for Atmospheric Research Sponsored by NSF and NASA

2 Overview l What is Unidata? l What is netCDF? l Data model l Programming interfaces l Format l Limitations l Standardization Possibilities l Summary

3 Unidata: A Community Endeavor l Community of educators and researchers at 120 universities, 30 other institutions, international in scope l Mission: providing data, tools, support, and community leadership for enhanced earth-system education and research l Funded primarily by the National Science Foundation l Atmospheric science community, expanding to oceanography, hydrology, other geosciences l Unidata Program Center: 25 staff, 15 developers

4 Unidata Provides... l User support, software development, maintenance, and community advocacy l Near real time “push” data dissemination: Internet Data Distribution among 170 institutions for use in research and education (IDD, LDM) l Tools for analysis and visualization (Gempak, IDV, McIDAS) l Data access and collections infrastructure (netCDF, udunits, THREDDS,...)

5 NetCDF Overview l Model for scientific data: variables, dimensions, attributes, coordinates l Libraries for data access: C, Fortran, C++, Java, Perl, Python, Ruby, Matlab, IDL,... l Associated Conventions convey semantics l Format for portable binary data that supports direct access, metadata, appending new data, shared access (note that users need not know anything about format)

6 Some Applications for netCDF l Gridded output from models: forecast, climate, ocean, atmospheric chemistry,... l Observational data: surface, soundings, satellite, lightning, aircraft, radar,... l Various scientific data archives l Chromatography, mass spectrometry, medical imaging,... l In general: platform-independent, language- independent interface to insulate data providers from data consumers

NetCDF’s Niche l Simple data model for scientific datasets l Portable, self-describing data l Appendable, sharable, archivable l Direct access for efficient subsetting l Metadata via attribute conventions such as CF l Flexible remote access via OPeNDAP, HTTP, WCS l Lots of applications: NCO, ncbrowse, ncview, IDV, IDL, MATLAB, ArcGIS,... l Language interfaces include C, Java, Fortran, C++, Perl, Python, Ruby,...

8 NetCDF Data Model l Variables: name, shape (list of Dimensions), type, Attributes, values l Dimensions: name, length l Attributes: name, type, value(s) l Coordinate Variables: values associated with Dimensions

9 NetCDF Data Model Variable name: String shape: Dimension[ ] type: DataType Dataset location: String Dimension name: String length: int isUnlimited: boolean Attribute name: String value: Vector type: DataType CoordinateVariable name: String shape: Dimension[ 1 ] type: DataType same name ValuesArray type: primitive shape: int[ ]

10 Availability of netCDF l Freely available source, binaries: u l Unidata-supported interfaces: u C, C++, Fortran-77, Fortran-90, Java, Perl l Platforms: u Unix, Windows, Mac OS X,... l License: u X Windows/MIT style open source license u Used in several commercial applications l Status: u version released February 2004 u netCDF-4 under development

Widespread netCDF Use l 500 entries representing 32 in netcdfgroup mailing list: academic institutions, government agencies, commercial firms l Downloaded by more than 2000 distinct hosts in 55 countries in last 10 years l Software for manipulating/displaying netCDF: more than u 15 commercial packages, u 56 freely available package l Adopted by Analytical Instrument Association [AIA] to implement the Analytical Data Interchange Protocols l Neuro-Imaging Laboratory of the Montreal Neurological Institute

12 netCDF Language Interfaces l Conventions of each language supported by interfaces l C: core implementation l C++, Fortran-77, Fortran-90, Perl: call C library l Java: independent pure-Java implementation l Python, Ruby, IDL, Matlab, Ada: third-party layers over C library l Data is language-neutral

13 Java Interface l Currently most advanced netCDF interface, used to test new ideas l Client access to data servers: u HTTPD u OPeNDAP/DODS l Advanced coordinate system support for general and georeferenced coordinates l Virtual datasets to aggregate, subset, redefine, using NcML (XML representation for netCDF)

14 NetCDF Format l Purpose: insulate users, applications, and data from machine architectures, format details, additions to data l Provides data that is u self-describing u portable u directly accessible u appendable l Still support original XDR-based format (1988) l 32-bit file offset limits file size l New format with 64-bit offsets for huge files

NetCDF File Format Records grow in the UNLIMITED dimension l Header u Dimensions u Attributes u Variable metadata l Fixed-size variables l Record variables u Use UNLIMITED dimension l XDR-like represenations for all data and metadata netCDF Header 1st non-record variable 2nd non-record variable... nth non-record variable 1st record for 1st record variable 1st record for 2nd record variable... 1st record for rth record variable 2nd record for 1st record variable 2nd record for 2nd record variable... 2nd record for rth record variable

16 Some Current netCDF Limitations l Not a database system: no keyed access, transactions, multiple concurrent writers l Strings are just character arrays l No nested data structures l No “ragged arrays”, variable length records l Only one growable dimension per dataset l No built-in compression or packing l Georeferencing requires agreement on conventions l...

17 NetCDF CF Conventions l Climate and Forecast metadata conventions u Metadata for use, not discovery  u Adopted by: PCMDI, PRISM, ESMF, NCAR, Hadley Centre, GFDL, various EU projects u Developed by volunteers, active list l Includes georeferencing conventions for multidimensional coordinate systems l Includes a “Standard Name Table” for quantities u Google for “CF standard name table” u 76 of 416 entries (as of June 3, 2004) relate to hydrology

Taking Advantage of Web Services for Data System Interoperability GIS Client Applications FES Client Applications OpenGIS Protocols: WMS, WFS, WCS OGC or proprietary GIS protocols OGC or OPeNDAP ADDE. FTP… protocols GIS Server GIS Servers Demographic, infrastructure, societal impacts, … datasets THREDDS Server FES Servers Satellite, radar, forecast model output, … datasets

19 Obstacles to GIS-netCDF Connection l netCDF data model (finite sampling of real functions) differs from relational, GIS datamodels l Some netCDF data not spatial l Much netCDF data multidimensional, > 2D l Georeferencing for netCDF data uses multiple conventions l Typical netCDF datasets may be very large l Large netCDF datasets may be on a remote server

20 Unidata GIS-netCDF Efforts l Prototype netCDF to geoTIFF converters l Parser for georeferencing conventions implemented in Java l Unidata GeoGrid from georeferenced netCDF variable supported in Java l GridDataset: collection of GeoGrids in a netCDF dataset with methods for getting volumes and slices l NcML for wrapping any new conventions around existing data

netCDF-related Changes Proposed in WCS 1.1 l WCS encoding “profiles” instead of fixed list of encoding formats l CF-netCDF is first proposed WCS encoding profile l Multiple “variables” or “parameters” in a coverage (e.g., pressure, temperature, etc.) l Coverages with 3 spatial dimensions l Coverages with multiple time dimensions (e.g. forecast time in model output) l Non-spatial “height” dimension, (e.g., atmospheric pressure, ocean density) l Irregularly-spaced grids l Technical committee votes on entirety of WCS 1.1 this summer.

Possible netCDF Standardization l File format l Applications programming interfaces l Conventions (e.g., CF, Climate and Forecast) l Release level: u netCDF 3 u netCDF 4 later this year (will include access to HDF 5 files) l Common Data Access Model (with elements of netCDF, HDF, and OPeNDAP data models) l (Relationship to WCS encoding profile specification?)

23 Summary l NetCDF is a simple data model, different from Relational or GIS data models l NetCDF format and interfaces provide a portable way to store and access multidimensional array-oriented data l Conventions exist for georeferencing netCDF data l May be a bridge to link GIS and scientific users, applications, and data

More Information l NetCDF documentation and code l CF Conventions metadata/index.html metadata/index.html l Russ Rew