© Crown copyright Met Office CF-netCDF in WMO Bruce Wright (Senior IT Architect), 7 th GO-ESSP Community Workshop, September 2008

© Crown copyright Met Office Contents This presentation covers the following areas WMO ET-ADRS ET-ADRS Initial Meeting Since the ET-ADRS Initial Meeting Some Thoughts Discussion

© Crown copyright Met Office WMO (World Meteorological Organization)

© Crown copyright Met Office World Meteorological Organization (WMO) Specialized agency of the United Nations (UN) Membership of National Meteorological and Hydrological Services (NMHSs) Facilitates the free and unrestricted exchange of data and information, products and services in real- or near-real time on matters relating to safety and security of society, economic welfare and the protection of the environment Recognized as international standardization body by ISO

© Crown copyright Met Office WMO Structure Commission for Basic Systems (CBS) Information Systems & Services (ISS) Data Representation Codes (DRC) Assessment of Data Representation Systems (ADRS) Technical Commission (8) Open Programme Area Group (OPAG) (4) - Supported by Implementation Coordination Team (ICT) Expert Teams (ET) - Permanent: Maintains the WMO code forms (e.g. GRIB, BUFR) - Temporary: Deliver recommendations

© Crown copyright Met Office ET-ADRS (Expert Team on Assessment of Data Representation Systems)

© Crown copyright Met Office ET-ADRS Formation Extraordinary session of CBS (Seoul, Nov 2006): Agreed to study the implications of using data forms such as XML or netCDF for meteorological data, especially for operational real time exchanges Requested the establishment of the ET-ADRS within the OPAG-ISS for this purpose All technical commissions were invited to participate Charged the ET-ADRS to provide recommendations, including a proposal on CBS Policy ET-ADRS to submit outcomes to the ICT of the OPAG-ISS for Nov 2008 meeting (ahead of next CBS session in Mar 2009)

© Crown copyright Met Office ET-ADRS Aim Assess advantages and disadvantages of different data representation systems, e.g.: GRIB / BUFR / CREX XML NetCDF / HDF For use in: Real time operational international exchanges between NMHSs Transmission of information to users outside the NMHSs Including implications: Need for defining standardization Data processing development and integration Costs and benefits: flexibility, compression, feasibility of implementation, etc

© Crown copyright Met Office ET-ADRS Membership Co-Chairs Simon Elliott (Eumetsat), Fred Branski (NOAA) Core members (6 – 4 at meeting) Also presented on data representation systems (i.e. GRIB/BUFR/CREX, XML, netCDF, HDF5 & ASN.1) Associate members (31 – 3 at meeting) Technical Commissions (4 reps at meeting) Invited Experts (8 at meeting) International Organizations (1 ICAO rep at meeting)

© Crown copyright Met Office ET-ADRS Initial Meeting Silver Springs, Washington, April

© Crown copyright Met Office ET-ADRS Initial Meeting Review of data representation systems: Presentations & synopsis documents (Core team members) Including SWOT analysis & recommendations Discussion Comparison of data representation systems Sub-groups Particular criteria Plan for future work

© Crown copyright Met Office CBS Policy Proposals (Initial Meeting Report, version 7) Data Representation Systems (DRSs) should be driven by user requirements Consider the need for interoperability when different DRSs are being utilized Application of ISO191xx series of geographic information standards to the development of a WMO conceptual model of data representation: Standard approach -> WMO core profile, including feature catalogues, application schemata and data specifications Facilitating interoperability and data exchange between applications based on DRSs associated to BUFR, CREX, GRIB, XML, netCDF and HDF

© Crown copyright Met Office Recommendations 1 (Initial Meeting Report, version 7) Adoption of netCDF where fit for purpose Adoption of NetCDF-4 (rather than NetCDF-3): Improved performance Convergence of NetCDF and HDF (effectively adopting 2 standards though one API). It was considered important do distinguish between recommending netCDF as a format and recommending the use of the API from Unidata

© Crown copyright Met Office Recommendations 2 (Initial Meeting Report, version 7) Noting that: NetCDF itself provides very limited metadata CF metadata convention has been developed for sharing climate model and NWP forecast data. Although used in some operational meteorological implementations, there are shortcomings with the CF Engagement with the NetCDF and CF community: Agree on a coordination mechanism to drive forward the NetCDF format (e.g. enhanced packing/compression) and the CF metadata standards to be fit for operational meteorology (WMO requirement) Consider the resources (e.g. at NMHS level) required to support the data format and metadata standard development Develop a model for the governance

© Crown copyright Met Office Since the ET-ADRS Initial Meeting

© Crown copyright Met Office Feedback on Meeting Report Suggested change to netCDF recommendation: “…WMO should look to adopt the netCDF4 API …but initially with the netCDF-4 classic model 'virtual format' (a combination of a real binary format and data model), as this will offer a high degree of interoperability, whilst addressing many of the shortcomings of netCDF3.” Raised concerns over the interpretation of phrase: “…important do distinguish between recommending it as a format and recommending the use of the API from Unidata.” Proposed various changes / additions to the Annex on the comparison of DRSs

© Crown copyright Met Office Further progress …it’s all gone rather quiet

© Crown copyright Met Office Some Thoughts

© Crown copyright Met Office Déjà vu? Meeting of Expert Team on the Development of GRIB2, Silver Springs, 2-5 Dec 1997 Included Russ Rew (Unidata) & Mike Folk (NCSA) to present on netCDF & HDF Recommended: Develop Roadmap for merge project with netCDF, HDF and GRIB WMO to write to netCDF and HDF authorities expressing interest in closer collaboration and the potential for a standard common API Noted: GRIB, netCDF and HDF complementary in many ways NetCDF and HDF could adopt GRIB2 as a basic physical format

© Crown copyright Met Office Déjà vu? (2) …and then it all went rather quiet

© Crown copyright Met Office Changing Times? NetCDF now in very wide use: High profile – IPCC climate data Bastion of GRIB – ECMWF for ensemble data Likely that WMO CBS will provide some kind of endorsement of netCDF Will be driven forward by NMHS usage anyway E.g. Met Office looking to move away from bespoke format to CF-netCDF, GRIB-2 and BUFR

© Crown copyright Met Office Opportunities / Threats WMO involvement with CF… Opportunity to get additional resources allocated from WMO member states to develop / support the standard How will a slow-moving standards organisation like WMO work with a (relatively) quickly evolving standard like CF? Threat to the standard / process – currently both responsive to user requirements and very well thought-out, due to the fairly small, dedicated community involved

© Crown copyright Met Office Questions and answers Discussion