1. GDWG Agenda Item Repository for GSICS Work - Clarification on Archiving
Masaya Takahashi, JMA
CMA, CNES, EUMETSAT, ISRO, IMD, JAXA, JMA, KMA, NASA, NIST, NOAA, ROSHYDROMET, USGS, WMO

2. Overview and Purpose Overview Purpose of this presentation
Existing datasets for the ATBDs
1) SRF netCDFs for all the GRWG activities
Now archived on the NOAA THREDDS and the GISCS wiki
2) Source dataset (lunar observation netCDFs) for GRWG VIS/NIR sub-group
Now archived on the EUMETSAT internal Linux server
Possible datasets for the ATBDs
3) Intermediate dataset (Deep Convective Cloud) for GRWG VIS/NIR sub-group
To share those data online is very important not only for maintaining the data but also for accelerating/encouraging the GRWG activities
Purpose of this presentation
To introduce the GRWG requirements
To discuss the proper way to archive those datasets

3. GRWG Requirements for archiving the dataset for the ATBDs – 1) SRF netCDFs
Spectral Response Function (SRF)
Instrument specific (fundamental) information -> necessary for GRWG activities
File size: small in general (~MB), not so frequently updated
Each GPRC has their official SRF(s)
A variety of formats (.csv, .txt, .xls, .hdf, etc.)
In 2013, GDWG (A. Jelenak) converted official SRFs to netCDFs for user convenience
GRWG VIS/NIR sub-group is the first user (after minor modifications by M. Takahashi)
More and more GRWG members will use the dataset for the ATBDs in the near future
How to store the data at present?
On the NOAA THREDDS server as “Source Data”
GSICS THREDDS servers are NOT the data archiving system,
even though that is not clarified
On the GSICS wiki (
Commas/plus signs: automatically removed from filenames
E.g., W_CN-CMA-NSMC,VIS+IR+SRF,FY2D+SVISSR_C_BABJ.nc
-> W_CN-CMA-NSMCVISIRSRFFY2DSVISSR_C_BABJ.nc

4. GRWG Requirements for archiving the dataset for the ATBDs – 2) Lunar observation netCDFs
Source dataset for the ATBD (lunar calibration, GRWG VIS/NIRS sub-group)
To collaborate with the lunar calibration community, the lunar calibration workshop (LCW, Dec decided to share some datasets
File size: depends on the instrument (e.g., ~5MB for MTSAT-2, ~20MB for Himawari-8)
# of files: depends on the instrument (e.g., ~5/month for MTAST-2, ~500/month for Himawari-8)
NO need to share all the data, so we can adjust data volume depending to the storage capacity
Password protection would be required to limit the access within the community
Action assigned at the LCW
GLCWS_14.7: GSICS Data Working Group to identify a suitable mechanism for sharing the lunar calibration dataset (Main: M.Takahashi)
How to store the data at present?
On the EUMETSAT Linux server
Only EUMETSAT people can access the data

5. GDWG Proposed Solution for issued on the data archiving
Group discussion to proposed the following solution
S1: GDWG to define GSICS THREDDS servers as the archiving system
S2: GDWG to recommend to Use GSICS wiki
S3: GDWG to find online storage services such as Dropbox

6. GDWG discussion on the Pros and Cons of the presented solutions
S1: GDWG to define GSICS THREDDS servers as the archiving system
pros: good to store the netCDF files
cons: internal issues in EUMETSAT/NOAA/CMA?
S2: GDWG to recommend to use the GSICS wiki
pros: easy to put small / few files
cons: difficult to store big volume data, problem on the automatic change of the netCDF file name
S3: GDWG to find online storage services such as Dropbox
pros: easy-to-use
cons: THREDDS/OpenDAP is not supported?, the service might not free, security issue?
I would like to recommend to invest resources to implementing S1.

7. GDWG discussion on the details on the proposed solution (to be updated in the discussion)
The GDWG proposed …
Resources Needed:
Man Days Effort:
Identify a system (service) and evaluate its suitable for GSICS dataset archiving – 1 week
Specify how the system is to be used and the tasks to be implemented – 2 weeks
Implementation – 1 week.
HW & SW Procurements: unknown
Input Data Acquisition Effort: not applicable
Proposed Time Scale:
To be available as soon as possible for collaboration. Latest implementation is for the next joint meeting
Deliverables:
Data archiving server (or service)
Dataset on the server
Developer’s documentation for using the server
Priority of the Task: High – by the next GSCIS annual meeting

8. Agenda item summary; assign action identifier, make EP recommendations and propose a lead for the action (to be updated in the discussion)
Action Identifier: GDWG_15.XX: XX to …
Substantial effort if required by the GDWG, GDWG chair should inform this activity to GSICS Executive Panel, and ask for feedback regarding:
Should GDWG resources be used to investigate how to archive the dataset for GSICS activities (not products) and provide solutions to GRWG.
The GDWG estimates 4 weeks of resources is needed to support this action.
Identify the Working Group Member Taking the Lead on this Action:
XXXX
The EP is invited to comment on this if the action is to be performed.

9. End of Presentation: Thank you for your attention
WMO GSICS Portal
GSICS Coordination Centre -
GSICS Product Catalog -
EUMETSAT’s Data and Management Server