2. GSICS Data and Products Servers Volker Gärtner (EUMETSAT) For GSICS Data Working Group

3. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 3 EUMETSAT GSICS Data and Products Servers GSICS Data Working Group (GDWG)‏ V. Gaertner (Chair), Y. Hwang, A. Jelenak, J. Lafeuille, Y. Li, P. Miu, R. Nakayama, H. Owada, Z. Rong, H. Rothfuss CMA, EUMETSAT, JMA, KMA, NOAA/NESDIS, WMO

4. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 4 GDWG and the GSICS Data and Products Servers What is the GDWG?  The GSICS Data Working Group (GDWG) is a body of the GSICS organisation which consists of data management experts from the participating GSICS Processing and Research Centres (GPRC’s).  In close cooperation with the GSICS Research Working Group (GRWG) it advises the GSICS Executive Panel on the planning and implementation of data management activities, infrastructure to provide data, data formats, data retention etc. What are the GSICS Data and Products Servers?  These are data management servers, to be implemented by several GPRC’s (CMA, JMA, NOAA, EUMETSAT), which follow a defined set of architecture principles and provide a set of services for the GDWG and the wider GSICS user community to support exchange and access to relevant data.

5. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 5 GSICS Data and Products Servers Concept  EUMETSAT took the lead in the development of this server as a result of an agreed GSICS Action (Ref: GDWG-II WG2/01) from the annual Joint Meeting of the GSICS Research and Data Working Groups.  The purpose of the server is to provide a platform where the GRWG can access and exchange comparable source data sets for their research and upload resultant products back to the server for validation, publishing and/or archiving.  To facilitate data handling,  the netCDF format was chosen for both the comparable data sets and the resultant products  the WMO Global Telecommunication System (GTS) filename convention was selected  Once the development has been finally approved, it is envisaged that federation of these servers is created to ensure data redundancy as well as offer a platform for different calibration activities (e.g. CNES SADE desert site calibration project).

6. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 6 GSICS Data and Products Servers Services  The server provides an upload service for the GSICS partners to upload their comparable netCDF data sets in the WMO file naming convention.  These data sets form a rolling archive accessible via HTTP or FTP download.  Resulting products from the GRWG are also uploaded to the server for validation, publishing and/or archiving.  For the EUMETSAT Data and Products Server, these products are archived in the EUMETSAT Data Centre Archive (URL - http://archive.eumetsat.int).  The EUMETSAT Data and Product Server is implemented on a Sun Fire X4440 x64 server under the Sun Solaris v10 OS.  The Unidata THREDDS Data Server, deployed in the Apache Tomcat web- server, catalogues the uploaded data sets and offers the HTTP and FTP download services. A ProFTP server is installed for FTP file transfers.

7. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 7 GSICS Data and Products Servers High Level Design  GEO-AIRS/IASI is one of many  GSICS supports inter-calibration on other orbits, for other instruments, in other spectral regions  All are similar in some ways  Each pair is unique in some ways  Instrument characteristics  Agency priority  Hierarchical Structure  General ATBD  Purpose, Options, Selection, Implementation  Details for individual pairs on http://www.star.nesdis.noaa.gov/smcd/spb/calibration/icvs/GSIC S/index.php

8. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 8 EUMETSAT GSICS Data and Products Server  Comparable Data Sets available on the EUMETSAT Data and Products Server:  MFG15 netCDF data set  MSG15 netCDF data set  IASI 1c netCDF data set  Members of GRWG perform comparisons between MFG15 and IASI 1c data sets as well as MSG15 and IASI 1c data sets  Currently uploaded products to the Server are the prototype GSICS Correction Coefficients for the inter- calibration of the IR channels of Meteosat-9 using IASI as a reference

9. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 9 Access to the EUMETSAT GSICS Data and Products Server Downloading source data and products The EUMETSAT Data and Product Server went live on the 1st April, 2009. It can be accessed via any web browser using the following URL: http://gsics.eumetsat.int Uploading source data and products  FTP accounts have been created for the upload of data and products to the server. Only validated and accepted products – according to GSICS Procedure for Product Acceptance – can be uploaded.  Requests for ftp accounts can be forwarded to the EUMETSAT User Service Helpdesk (Email: ops@eumetsat.int).ops@eumetsat.int  The server is currently pre-operational awaiting finalisation of the first set of GSICS data and products, i.e. changes can be made without user announcement.  More GSICS information can be found on the GSICS website: http://gsics.wmo.int/

10. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 10 GSICS Data and Products Servers Documentation  The following documents have been presented on the GSICS mailing lists (http://groups.google.com/group/gsics-data-wg?hl=en):  GSICS Data Management Server User Guide, Draft.  GSICS Data Management Server Operations Service Specification (and supporting documentation) Action GDWG 03_22, Draft.  EUMETSAT Archive NetCDF Format Proposal, Draft.  All documents are in draft form but will be released once the Data and Products server is operational.

11. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 11 Data and Products Servers: Planned Activities Planned activities:  Other GPRC’s to fully implement Data and Products Server  Implement the mechanism for transfer of source data sets between GPRC data and product servers.  Support to the GRWG in creation of the first validated calibration products.  Analyse how best to improve the server’s services in terms of the GSICS products created from the source data sets (e.g. automated processes for data access).  Examine the porting of EUMETSAT NetCDF data sets from version 3 to version 4.

12. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 12 B A C K U P

13. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 13 EUMETSAT GSICS Data and Products Server - Screenshots Access: http://gsics.eumetsat.inthttp://gsics.eumetsat.int

14. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 14 EUMETSAT GSICS Data and Products Server - Screenshots Source data Uploaded products

15. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 15 EUMETSAT GSICS Data and Products Server - Screenshots Source data EUMETSAT

16. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 16 EUMETSAT GSICS Data and Products Server - Screenshots Source data EUM MFG15

17. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 17 EUMETSAT GSICS Data and Products Server - Screenshots Access to Dataset Metadata of source data collection

18. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 18 EUMETSAT GSICS Data and Products Server - Screenshots Access: dataset Dataset Metadata

19. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 19 EUMETSAT GSICS Data and Products Server - Screenshots First 10 cols / rows of MFG1.5, channel 1