GSICS Baseline Review: Product meta-data and structures Masaya Takahashi - JMA CMA, CNES, EUMETSAT, ISRO, IMD, JMA, KMA, NASA, NIST, NOAA, ROSHYDROMET, USGS, WMO

Overview of Purpose of the Review Overview - GDWG has collaboration on the following Data Management Guidelines, Conventions and Standards for GSICS products and data server structures: NetCDF as the Data Exchange format; classic data model NetCDF global attributes WMO file naming convention Climate and Forecast (CF) meta-data convention CF geo-location guideline; lon, lat indexing of arrays ISO 8601 date and time format standard THREDDS as the GSICS server Purpose of the Review is to: Propose justifiable updates; for current limitations and future needs Remove obsolete / unused versions

Achievements in 2015 – Product (netCDF) File naming Convention Reference satellite/instrument names have been updated https://gsics.nesdis.noaa.gov/wiki/Development/FilenameConvention Recommendation - use the following reference satellite and instrument names which are used by the owner agency or used on the WMO-OSCAR even though file naming is not case sensitive (No need to change filename of existing GSICS Correction)

Achievements in 2015 – GEO-LEO-IR netCDF Convention 5 Global Attributes were added to the netCDF template https://gsics.nesdis.noaa.gov/wiki/Development/NetcdfConvention :monitored_instrument_code = "WMO_SATELLITE_CODE WMO_INSTRUMENT_CODE"; (e.g. :monitored_instrument_code = 57 207 means “MSG3 SEVIRI”) :reference_instrument_code = "WMO_SATELLITE_CODE WMO_INSTRUMENT_CODE"; - from WMO common code table C-5 and C-8: :window_period = "P-14D+14D" FOR GEO-LEO-IR RAC, "P-14D+0D" FOR GEO-LEO - represents the period of time for statistics (e.g. from :time_coverage_start to :time_coverage_end) :atbd_doi = "DOI_NAME" // Note, there are no GSICS specific DOI names. :product_doi = "DOI_NAME" // DOI are assigned by the product owners's organisation and must satisfy the constraints of Talk on DOIs – Thursday pm (agenda item 6i) by Peter Miu

Achievements in 2015 – THREDDS configuration EUMETSAT, NOAA and CMA agreed to update THREDDS configuration https://gsics.nesdis.noaa.gov/wiki/Development/DataServerConf

Requirements Summary GRWG Requirements for change: Adding new attribute to netCDF variables – Let’s discuss now! Adding new algorithm type to file naming – To be discussed at 6g Use of groups to categorised variables (enhance data model) – by Masaya Takahashi, agenda item 6e Proposed VNIR Product Convention – by Masaya Takahashi, 6g Proposed MW Product Convention - by Manik Bali, 6b GPAT Requirements for change: None at present User Requirements for change: None at present WMO, Unidata, CF requirements: None at represent We might need to do in future (e.g. proposing new standard_name for CF Convention)… (Just to be introduced here)

Discussion on the GRWG requirement Decision at the 2015 GDWG session in Delhi To add “comment” attribute to netCDF variables in order to: provide users with more accurate indication of what the variable is providing e.g. storing time information for valid period of time float slope [date, chan] _FillValue: -999999 long_name: Regression Slope units: 1 comment: “valid hour range: 00:00 UTC and 03:59 UTC” “invalid hour range: 15:00 UTC and 17:59 UTC” Is free format comment allowed? Do we need to use ISO 8601 time format in the attribute?

Alternative solution To add attribute which is specific to represent validity time range variable slope [date, chan] _FillValue: -999999 long_name: Regression slope units: 1 invalid_hours: T03:00:00Z/PT12H 12 hours from 0300UTC (i.e., 0300-1500UTC) ISO8601 which GSICS products follow Expressions should be checked... We also have “validity_period” variable, so we need to consider new attribute name to avoid user’s confusion double validity_period(date, validity) ; validity_period:long_name = "correction validity period" ; validity_period:standard_name = "time" ; validity_period:calendar = "gregorian" ; validity_period:units = "seconds since 1970-01-01T00:00:00Z" ;

Agenda item summary; assign action identifier, make EP recommendations and propose a lead for the action (to be updated in the discussion) Action Identifier: GDWG.2016.5b.1: XX to … Substantial effort if required by the GDWG, GDWG chair should inform this activity to GSICS Executive Panel, and ask for feedback regarding: The GDWG estimates XX week of resources is needed to support this action. Identify the Working Group Member Taking the Lead on this Action: XXXX

NetCDF-4 Data Model New features: Groups and User-Defined Types For more information, see: https://www.unidata.ucar.edu/presentations/Rew/rew-egu.pdf New features: Groups and User-Defined Types http://www.unidata.ucar.edu/software/netcdf/workshops/2011/datamodels/Nc4-uml.html

Reasons to use the classic model: When to Use the Enhanced Data Model – from 2011 Unidata NetCDF Workshop http://www.unidata.ucar.edu/software/netcdf/workshops/2011/datamodels/Nc4-WhyUse.html Enhanced data model: offers rich features for structuring data, but breaks backward compatibility Classic model: simple, well-understood, and had been around for a long time a Reasons to use the classic model: Data using the classic model can be read by all existing netCDF software Writing programs for classic model data is easier Most or all existing netCDF conventions are targeted at the classic model Many great features, like compression, parallel I/O, large data sizes, etc., are available within the classic model Reasons to use the enhanced model: Complex data structures can be represented very easily in the data, leading to easier programming Description in the 2011 Unidata NetCDF Workshop website... If existing HDF5 applications produce or use these data, and depend on user-defined types, unsigned types, strings, or groups, then the enhanced model is required. In performance-critical applications, the enhanced model may provide significant benefits.

Potential use of the enhanced data model in GSICS products GEO-LEO-VNIR Products (netCDF) will contain multiple calibration methods’ results in ONE FILE Current proposal by GRWG Grouping could be useful char method_name(method, method_strlen) ; :long_name = "method identifier" ; float weight_method(date, chan, method) ; :long_name = "weight of the methods used for the blend in each channel" ; :units = "1" ; SEVIRI vs. Aqua/MODIS VISNIR product Combined result (DCC+Moon+…) Global attributes, Dimensions, Variables Method1 (Deep Convective Cloud) Global attributes, Dimensions, Variables Advantage: No need to use enhanced data model Disadvantage: Variables have multiple dimensions (>=3) All the methods have to use the same variables Method-specific variables are not allowed Method2 (Moon) Global attributes, Dimensions, Variables Method3 (Desert) Global attributes, Dimensions, Variables