1. Integrated Model Data Management S.Hankin ESMF July ‘04 Integrated data management in the ESMF (ESME) Steve Hankin (NOAA/PMEL & IOOS/DMAC) ESMF Team meeting July 2004

2. Integrated Model Data Management S.Hankin ESMF July ‘04 The growing importance of data integration to modelers: We no longer fund “modeling”. Today we fund “climate prediction” or “coastal processes” … the science topics. Modeling is just a component. a program manager (anonymous)

3. Integrated Model Data Management S.Hankin ESMF July ‘04 Model outputs need to be made useful to many classes of users model run project modeling community research community education

4. Integrated Model Data Management S.Hankin ESMF July ‘04 Data and products need to be made more usable for modelers modeling validation assimilation Real time and delayed-modeobservations data products comparison boundary & initial conditions

5. Integrated Model Data Management S.Hankin ESMF July ‘04 An ESME must include a plan for data management. But, how ? Funds are limited …

6. Integrated Model Data Management S.Hankin ESMF July ‘04 Partnership: A community of data managers has formed … GO-ESSP – Global Organization of Earth System Science Portals (http://esportal.gfdl.noaa.gov) Unidata Unidata ESG (NCAR, LLNL) ESG (NCAR, LLNL) OPeNDAP (a.k.a. ‘DODS’) OPeNDAP (a.k.a. ‘DODS’) COLA COLA NOMADS (GFDL, PMEL, NCDC, NCEP) NOMADS (GFDL, PMEL, NCDC, NCEP) NASA/GCMD NASA/GCMD BADC, BODC BADC, BODC WMO WMO …

7. Integrated Model Data Management S.Hankin ESMF July ‘04 Ocean data systems following similar approaches … National Virtual Ocean Data System (NVODS) National Virtual Ocean Data System (NVODS) US Integrated Ocean Observing System US Integrated Ocean Observing System GODAE (US and International) GODAE (US and International) OCMIP, AOMIP, … OCMIP, AOMIP, …

8. Integrated Model Data Management S.Hankin ESMF July ‘04 Data portal components: Workflow Data discovery Data discovery Data access/transport Data access/transport On-line browse and comparison On-line browse and comparison (Segue to analysis) (Segue to analysis)

9. Integrated Model Data Management S.Hankin ESMF July ‘04 Data discovery == Metadata + search  Mature standards do not exist today. A task for ESMF: define and utilize metadata standards for modelers

10. Integrated Model Data Management S.Hankin ESMF July ‘04 Metadata for modelers Reviewed & “published” (a “standard”) Reviewed & “published” (a “standard”) Structured (XML) Structured (XML) Generated automatically in conjunction with setting up model runs Generated automatically in conjunction with setting up model runs Standardized parameter names (“controlled vocabularies”) Standardized parameter names (“controlled vocabularies”) Hierarchical Hierarchical –components, grids, fields and attributes

11. Integrated Model Data Management S.Hankin ESMF July ‘04 Others are working hard on search: 1. Traditional metadata partners (e.g. GCMD) 2. Semantic Web (Google on steroids) (3 years off?) Data discovery == Metadata + search

12. Integrated Model Data Management S.Hankin ESMF July ‘04 Data portal components: Data discovery Data discovery Data access/transport Data access/transport On-line browse and comparison On-line browse and comparison (Segue to analysis) (Segue to analysis) Workflow

13. Integrated Model Data Management S.Hankin ESMF July ‘04 Data transport OPeNDAP (a.k.a. “DODS”) –Network data access –Format-independence –Subsetting –Aggregation (GDS, Unidata) –Compression –Security: Grid-enabled OPeNDAPg

14. OPeNDAP: distributed access to data and semantic metadata

15. Integrated Model Data Management S.Hankin ESMF July ‘04 CF (“climate and forecast”) CF 1.0 is now a “standard” “use” metadata – e.g. units, coords. “use” metadata – e.g. units, coords. curvilinear, hybrid-Z, time-dependent curvilinear, hybrid-Z, time-dependent great applicability beyond modeling, too great applicability beyond modeling, too Discussion question: As the use of the CF standard widens how should the community support it? Not enough to “endorse” it. Need a partnership.

16. Integrated Model Data Management S.Hankin ESMF July ‘04 Data portal components: Data discovery Data discovery Data access/transport Data access/transport On-line browse and comparison On-line browse and comparison (Segue to analysis) (Segue to analysis) Workflow

17. Live Access Server

18. netCDF

19. Integrated Model Data Management S.Hankin ESMF July ‘04 Metadata (XML) contains the “intelligence” Back end applications do the real work OPeNDAP provides remote data access LAS -- an “Information Product Server” Ferret,CDAT or other

21. Information access Uniform data access

22. Integrated Model Data Management S.Hankin ESMF July ‘04 A “home page” Information access Uniform data access Live Access Server home Area 1 Project 1 Exp 1 Exp 2 Project 2 Area 2 Area 3 Project 3 Project 3 Exp 1 Exp 2 Project 4

23. Integrated Model Data Management S.Hankin ESMF July ‘04 The UI talks to LAS through an XML “web service”

24. For example …

25. On-line comparison

27. Example: Average over a lat-long box

28. Integrated Model Data Management S.Hankin ESMF July ‘04 Data portal components: Data discovery Data discovery Data access/transport Data access/transport On-line browse and comparison On-line browse and comparison (Segue to analysis) (Segue to analysis) Workflow

30. discover  browse  access NASA Global Change Master Directory (GCMD) Metadata Standards

31. Integrated Model Data Management S.Hankin ESMF July ‘04 LAS “sisters” share metadata to form a unified (virtual) site. OPeNDAP allows LAS to difference distributed fields. Collaborating groups of modelers

32. v6.2 customizable interfaces

33. Integrated Model Data Management S.Hankin ESMF July ‘04 “Web Service” access to products XML request protocol implemented and documented XML “package” out implemented and documented XML query protocol under development formal SOAP interface under development

34. A wealth of data products are available through the National Virtual Ocean Data System (NVODS)

35. Example: Zebiac model output “live” from IRI/LDEO

39. Access to observations (WODB -- 9 million ocean profiles)

42. configurable constraints

44. Integrated Model Data Management S.Hankin ESMF July ‘04 Query available data sets: Query variables in data set “model_1”: Query space-time domain: Request a subset of data as a file: (“asc” for ASCII format) “Batch” access to products >lasls http://cpu/LAS >lasls http://cpu/LAS model_1 >lasls http://cpu/LAS model_1 sst >lasget -x 20:60 -y 20:60 -t 11-Dec-2000 -f asc http://cpu/LAS model_1 sst

48. Integrated Model Data Management S.Hankin ESMF July ‘04 IOOS US Integrated Ocean Observing System 1.Detect and Predict Change 2.Mitigate natural hazards 3.Improve safety and efficiency of marine ops 4.Ensure national security 5.Reduce public health risks 6.Protect and restore marine ecosystems 7.Sustain marine resources

49. IOOS Data Management and Communications Subsystem Users Archive Centers Modeling On-line Browse ProductsForecasts Maps Regional Data Management Systems International Data Management Systems Terrestrial and Atmospheric Data Management Systems Metadata, Data Discovery and Data Transport Standards and Protocols Satellites Ships Primary Data Assembly & QC Hand Measurements Moorings Floats

50. IOOS Data Management and Communications Subsystem Users Archive Centers Modeling On-line Browse Products Forecasts Maps Regional Data Management Systems International Data Management Systems Terrestrial and Atmospheric Data Management Systems Metadata, Data Discovery and Data Transport Standards and Protocols Satellites Ships Primary Data Assembly & QC Hand Measurements Moorings Floats

51. Recommendations for ESMF 1. Define ESMF metadata standard and use it Consider a GCMD modelers’ portal, too Consider a GCMD modelers’ portal, too 2. Endorse partnership with GO-ESSP and the emerging tools Data available through OPeNDAP(g) Data available through OPeNDAP(g) Live Access Servers for on-line collaborations Live Access Servers for on-line collaborations

52. Questions? NVODS LAS: http://www.ferret.noaa.gov/nvods http://www.ferret.noaa.gov/nvods GCMD “DODS” Portal: http://gcmd.gsfc.nasa.gov/Data/portals/dods