1. GDWG Agenda Item Existing netCDF Format Updates
Masaya Takahashi, JMA
CMA, CNES, EUMETSAT, ISRO, IMD, JAXA, JMA, KMA, NASA, NIST, NOAA, ROSHYDROMET, USGS, WMO

2. Overview and Purpose Overview Purpose of this presentation
Existing GSICS products
GEO-LEO-IR (demo/pre-op phase, some are planned to be operational in 2015)
LEO-LEO-VIS (PATMOS-x for AVHRR, demo phase, no progress in the last few years)
GRWG discussion in 2014 annual/web meetings
Some variables and their attributes should be modified in order to allow the expandability of the inter-calibration results
Masaya Takahashi and Tim Hewison proposed the new convention
Should be done before GEO-LEO-IR products move to the operational phase
Purpose of this presentation
To introduce the GRWG proposal
To discuss the feasibility, and accept the proposal or provide alternative solution to GRWG

3. Requirements for updating the current GEO-LEO-IR netCDF convention
GRWG Requirements
For GEO-LEO-IR products, the following variables to be modified:
1) Calibration coefficients (slope, slope_se, offset, offset_se, covariance)
Many variables are required in the processing (e.g. Implementation of GSICS correction, calculation of uncertainty)
Based on linear regression = no expandability for non-linear regression (might be needed in future)
2) Validity period
Impossible to specify “valid hour range” in the current variable
“valid hour range” is required for some products, in order to
Boost the confidence of the product
Avoid the incorrect use of the product
Note: Once these variables are changed, the bias plotting tool should also be modified

4. Group discussion to proposed the following solution
GDWG Proposed Solution for updating the current GEO-LEO-IR netCDF convention
Group discussion to proposed the following solution
1) GDWG to accept GRWG proposal.
Pros of the proposal
can reduce number of variables
easy-to-expand into polynominal calibration coefficients
Cons of the proposal
3-D (or higher) array may NOT be supported in the plotting tool
Attributes like “units” might be a problem (1-D attributes are allowed, but need to check)
2) GDWG to provide alternative netCDF convention (or reject the proposal), and the reason why GRWG proposal is not accepted.

5. GDWG discussion on the calibration coefficients – comparison of the current/proposed variables
Current GRWG proposal
float offset(date, chan) ;
offset:valid_max = 200.f ;
offset:valid_min = -200.f ;
offset:_FillValue = " " ;
offset:long_name = "Regression Offset" ;
offset:units = "mW m-2 sr-1(cm-1)-1" ;
float offset_se(date, chan) ;
offset_se:valid_max = 200.f ;
offset_se:valid_min = -200.f ;
offset_se:_FillValue = " " ;
offset_se:long_name = "Standard Error of Regression Offset" ;
offset_se:units = "mW m-2 sr-1(cm-1)-1" ;
float slope(date, chan) ;
slope:valid_max = 2.f ;
slope:valid_min = -2.f ;
slope:_FillValue = " " ;
slope:long_name = "Regression Slope" ;
slope:units = "1" ;
float slope_se(date, chan) ;
slope_se:valid_max = 2.f ;
slope_se:valid_min = -2.f ;
slope_se:_FillValue = " " ;
slope_se:long_name = "Standard Error of Regression Slope" ;
slope_se:units = "1" ;
float covariance(date, chan) ;
covariance:valid_max = 200.f ;
covariance:valid_min = -200.f ;
covariance:_FillValue = " " ;
covariance:long_name = "Regression Coefficients Covariance" ;
covariance:units = "mW m-2 sr-1(cm-1)-1" ;
float coeff(ncoeff, date, chan) ;
offset:valid_max = 200.f ;
offset:valid_min = -200.f ;
offset:_FillValue = " " ;
offset:long_name = "Regression coefficients" ;
offset:units = "mW m-2 sr-1(cm-1)-1" ;
float covar(ncoeff, ncoeff, date, chan) ;
covariance:valid_max = 200.f ;
covariance:valid_min = -200.f ;
covariance:_FillValue = " " ;
covariance:long_name = "Regression coefficients covariance matrix" ;
covariance:units = "mW m-2 sr-1(cm-1)-1" ;
GRWG proposal is a general form of the regression

6. (need for the uncertainty calculation)
GDWG discussion on the calibration coefficients – comparison of the current/proposed variables
Linear regression (existing GEO-LEO-IR products adopt this way)
Y = slope * X + offset
Covariance matrix
(need for the uncertainty calculation)
float coeff(ncoeff, date, chan)
float covar(ncoeff, ncoeff, date, chan)
ncoeff = 2 (i.e., [offset, slope] )
covar(2, 2, date, chan)
Quadratic regression (future GSICS products might use this regression)
Y = a * X2 + b * X + c
Existing variables cannot support quadratic (polynomial) regression equation, but the proposed one can!
Future GSCIS products: not only new GEO-LEO-IR, but also GEO-LEO-VISNIR, GEO-LEO-MW, …
Is the proposal acceptable? No problem on the plotting tool / THREDDS?

7. GDWG discussion on the validity period – problem on the current variable
“2”, this means start/end time for each result
double validity_period(date, validity) ;
validity_period:long_name = "correction validity period" ;
validity_period:standard_name = "time" ;
validity_period:calendar = "gregorian" ;
validity_period:units = "seconds since T00:00:00Z" ;
Example
validity_period[783][2]
[0], E9, E9
[1], E9, E9
[2], E9, E9
...
Problem: Impossible to specify “valid hour range” in that variable
Necessary for instruments which have diurnal calibration biases (e.g., NOAA/JMA GEO-LEO-IR products)
Should be specified for all instruments (In definition, GSICS Correction is derived from only night-time observation)

8. GDWG discussion on the validity period – solutions
S1: Add new attribute to each variable
pros: good to avoid complexity of netCDF contents
cons: redundant because many variables might have the same attributes
S2: Add new variable (e.g., diurnal_validity_range)
pros: can avoid redundant attributes on the variables
cons: similar variable (i.e., validity_range) already exists, redundant
S3: Expand existing validity_period to contain hour range
pros: simple
cons: need to change the variable type from double to strings
Example
variable coeffs [ncoeff, date, chan]
_FillValue:
long_name: inter calibration coefficients
validity_hours: T03:00:00Z/PT12H
12 hours from 0300UTC (i.e., UTC)
ISO8601 which GSICS products follow
Expressions should be checked...

9. GDWG discussion on the details on the proposed solution (to be updated in the discussion)
The GDWG accepted the GRWG proposal for updating the current GSICS netCDF convention. According to this decision, GWDG will update the bias plotting tool.
Resources Needed:
Man Days Effort:
Prepare a documentation to update the GSICS bias plotting tool– 1 week.
Implementation – 1 week.
HW & SW Procurements: not applicable
Input Data Acquisition Effort: not applicable
Proposed Time Scale:
Before the first GSCIS OPERATIONAL product (EUMETSAT) becomes available (to be planned in mid 2015)
Deliverables:
New GSCIS netCDF convention for GEO-LEO-IR (and future GSCIS products)
Updated bias plotting tool
Priority of the Task: High – by mid 2015

10. Agenda item summary; assign action identifier, make EP recommendations and propose a lead for the action (to be updated in the discussion)
Action Identifier: GDWG_15.XX
Substantial effort if required by the GDWG, GDWG chair should inform this activity to GSICS Executive Panel, and ask for feedback regarding:
Should GDWG resources be used to update the GSICS bias plotting tool for GSICS activities.
The GDWG estimates 2 weeks of resources is needed to support this action.
Identify the Working Group Member Taking the Lead on this Action:
EUMETSAT for updating the bias plotting tool.
The EP is invited to comment on this if the action is to be performed.

11. End of Presentation: Thank you for your attention
WMO GSICS Portal
GSICS Coordination Centre -
GSICS Product Catalog -
EUMETSAT’s Data and Management Server