1. GSICS Baseline Review: Product meta-data and structures
Masaya Takahashi - JMA
CMA, CNES, EUMETSAT, ISRO, IMD, JMA, KMA, NASA, NIST, NOAA, ROSHYDROMET, USGS, WMO

2. Overview of Purpose of the Review
Overview - GDWG has collaboration on the following Data Management Guidelines, Conventions and Standards for GSICS products and data server structures:
NetCDF as the Data Exchange format; classic data model
NetCDF global attributes
WMO file naming convention
Climate and Forecast (CF) meta-data convention
CF geo-location guideline; lon, lat indexing of arrays
ISO 8601 date and time format standard
THREDDS as the GSICS server
Purpose of the Review is to:
Propose justifiable updates; for current limitations and future needs
Remove obsolete / unused versions

3. Achievements in 2015 – Product (netCDF) File naming Convention
Reference satellite/instrument names have been updated
Recommendation - use the following reference satellite and instrument names which are used by the owner agency or used on the WMO-OSCAR even though file naming is not case sensitive (No need to change filename of existing GSICS Correction)

4. Achievements in 2015 – GEO-LEO-IR netCDF Convention
5 Global Attributes were added to the netCDF template
:monitored_instrument_code = "WMO_SATELLITE_CODE WMO_INSTRUMENT_CODE";
(e.g. :monitored_instrument_code = means “MSG3 SEVIRI”)
:reference_instrument_code = "WMO_SATELLITE_CODE WMO_INSTRUMENT_CODE";
- from WMO common code table C-5 and C-8:
:window_period = "P-14D+14D" FOR GEO-LEO-IR RAC, "P-14D+0D" FOR GEO-LEO
- represents the period of time for statistics (e.g. from :time_coverage_start to :time_coverage_end)
:atbd_doi = "DOI_NAME" // Note, there are no GSICS specific DOI names.
:product_doi = "DOI_NAME" // DOI are assigned by the product owners's organisation and must satisfy the constraints of
Talk on DOIs – Thursday pm (agenda item 6i) by Peter Miu

5. Achievements in 2015 – THREDDS configuration
EUMETSAT, NOAA and CMA agreed to update THREDDS configuration

6. Requirements Summary GRWG Requirements for change:
Adding new attribute to netCDF variables – Let’s discuss now!
Adding new algorithm type to file naming – To be discussed at 6g
Use of groups to categorised variables (enhance data model) – by Masaya Takahashi, agenda item 6e
Proposed VNIR Product Convention – by Masaya Takahashi, 6g
Proposed MW Product Convention - by Manik Bali, 6b
GPAT Requirements for change: None at present
User Requirements for change: None at present
WMO, Unidata, CF requirements: None at represent
We might need to do in future (e.g. proposing new standard_name for CF Convention)…
(Just to be introduced here)

7. Discussion on the GRWG requirement
Decision at the 2015 GDWG session in Delhi
To add “comment” attribute to netCDF variables in order to:
provide users with more accurate indication of what the variable is providing
e.g. storing time information for valid period of time
float slope [date, chan]
_FillValue:
long_name: Regression Slope
units: 1
comment: “valid hour range: 00:00 UTC and 03:59 UTC”
“invalid hour range: 15:00 UTC and 17:59 UTC”
Is free format comment allowed?
Do we need to use ISO 8601 time format in the attribute?

8. Alternative solution
To add attribute which is specific to represent validity time range
variable slope [date, chan]
_FillValue:
long_name: Regression slope
units: 1
invalid_hours: T03:00:00Z/PT12H
12 hours from 0300UTC (i.e., UTC)
ISO8601 which GSICS products follow
Expressions should be checked...
We also have “validity_period” variable, so we need to consider new attribute name to avoid user’s confusion
double validity_period(date, validity) ;
validity_period:long_name = "correction validity period" ;
validity_period:standard_name = "time" ;
validity_period:calendar = "gregorian" ;
validity_period:units = "seconds since T00:00:00Z" ;

9. Agenda item summary; assign action identifier, make EP recommendations and propose a lead for the action (to be updated in the discussion)
Action Identifier:
GDWG b.1: XX to …
Substantial effort if required by the GDWG, GDWG chair should inform this activity to GSICS Executive Panel, and ask for feedback regarding:
The GDWG estimates XX week of resources is needed to support this action.
Identify the Working Group Member Taking the Lead on this Action:
XXXX

10. NetCDF-4 Data Model New features: Groups and User-Defined Types
For more information, see:
New features: Groups and User-Defined Types

11. Reasons to use the classic model:
When to Use the Enhanced Data Model – from 2011 Unidata NetCDF Workshop
Enhanced data model: offers rich features for structuring data, but breaks backward compatibility
Classic model: simple, well-understood, and had been around for a long time a
Reasons to use the classic model:
Data using the classic model can be read by all existing netCDF software
Writing programs for classic model data is easier
Most or all existing netCDF conventions are targeted at the classic model
Many great features, like compression, parallel I/O, large data sizes, etc., are available within the classic model
Reasons to use the enhanced model:
Complex data structures can be represented very easily in the data, leading to easier programming
Description in the 2011 Unidata NetCDF Workshop website...
If existing HDF5 applications produce or use these data, and depend on user-defined types, unsigned types, strings, or groups, then the enhanced model is required.
In performance-critical applications, the enhanced model may provide significant benefits.

12. Potential use of the enhanced data model in GSICS products
GEO-LEO-VNIR Products (netCDF) will contain multiple calibration methods’ results in ONE FILE
Current proposal by GRWG
Grouping could be useful
char method_name(method, method_strlen) ;
:long_name = "method identifier" ;
float weight_method(date, chan, method) ;
:long_name = "weight of the methods used for the blend in each channel" ;
:units = "1" ;
SEVIRI vs. Aqua/MODIS VISNIR product
Combined result (DCC+Moon+…)
Global attributes, Dimensions, Variables
Method1 (Deep Convective Cloud)
Global attributes, Dimensions, Variables
Advantage:
No need to use enhanced data model
Disadvantage:
Variables have multiple dimensions (>=3)
All the methods have to use the same variables
Method-specific variables are not allowed
Method2 (Moon)
Global attributes, Dimensions, Variables
Method3 (Desert)
Global attributes, Dimensions, Variables