1. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 1 GO-ESSP at LLNL Livermore, June 19th – 21st, 2006 World Data Center Climate: Status and Portal Integration WDCC Home: www.wdcc-climate.de / WDCC Contact: data@dkrz.dewww.wdcc-climate.dedata@dkrz.de Michael Lautenschlager, Hannes Thiemann and Frank Toussaint ICSU World Data Center Climate Model and Data / Max-Planck-Institute for Meteorology Hamburg, Germany

2. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 2 Content: WDCC Status CERA Concept Portal Integration

3. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 3 WDCC Content ERA40 IPCC CEOP BALTEX HOAPS CARIBIC WOCE ERA15/40 NCEP GEBCO COSMOS Simulations @ MPI, GKSS,… Data from Earth System Modelling and Related Observations EH5/MPI-OM IPCC-AR4 Start: Approved in January 2003 Maintenance: Model and Data (M&D/MPI-M) and German Climate Computing Centre (DKRZ) June 2006: 590 Experiments / 79.000 Data Sets

4. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 4 Data Export from WDC Climate Corresponds to 2 – 10 TB/month

5. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 5 Geographical Distribution of WDCC Users Total number of registered users: 750 (Mai 2006)

6. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 6 Data Import into WDC Climate 6 * 10**9 BLOBs ECHAM5/MPI-OM IPCC AR4 Scenarios (ca. 110 TB)

7. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 7 (I) Data catalogue and Pointer to Unix files  Enable search and identification of data  Allow for data access as they are (coarse granularity raw data files) (II) Application-oriented data storage in BLOB tables  Time series of individual variables are stored as BLOB entries in DB Tables (fine granularity data products) Allow for fast and selective data access  Storage in standard data format (GRIB, NetCDF/CF) Allow for application of standard data processing routines (PINGOs, CDOs) CERA 1) Concept: Semantic Data Management 1) Climate and Environmental data Retrieval and Archiving

8. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 8 Level 1 - Interface: Metadata entries (XML, ASCII) + Data Files Level 2 – Interf.: Separate files containing BLOB table data in application adapted structure (time series of single variables) Experiment Description Pointer to Unix-Files Dataset 1 Description Dataset n Description BLOB Data Table BLOB Data Table WDCC Data Topology BLOB DB Table corresponds to scalable, virtual file at the operating system level.

9. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 9 CERA Data Model Entry Reference Status Distribution Contact Coverage Parameter Spatial Reference Local Adm. Data Access Data Org

10. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 10 Data matrix of model experiment Model variables Model Run Time 2 D: small BLOBS (180 KB) 3 D: large BLOBS (3 MB) Raw data file: direct model output (1.3 – 16.2 GB) Each columm is one BLOB Table in CERA-DB Raw data file inDKRZ Archive

11. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 11 Preferred DB-storage structure for web-based access: single variable single level time series of 2D gridded data records Formats: GRIB-1 – NetCDF/CF (- GRIB-2) Climate Model Data Structures Application related data structure (2-D)original data structure (4-D)

12. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 12 TX7: Intel Itanium-2 with Linux DKRZ Architecture

13. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 13 Portal Integration Two strategies: One way integration: discovery and use metadata are integrated in a central data portal in one step Example: C3Grid data catalogue (refer to presentation from Heinrich Widmann) Two way integration: discovery metadata are integrated in central data portal, use metadata are extracted from remote archive when they are needed for data download and processing Example: Primary data publication in TIB library catalogue (STD-DOI) WDCC integration in NDG (NERC Data Grid)

14. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 14 Primary data publication (STD-DOI) URL: http://www.std-doi.de/ http://www.std-doi.de/ Data Review Primary Data Publication Process ISO 690-2: Metadata for citation of electronic media

15. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 15 Example: Publ.-DOI from WDCC

16. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 16 DOI URN

17. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 17 Publ.-DOI

18. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 18 830 GB

19. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 19 Data retrieval procudure is given at the end (user identification is required) Ident.-DOI

20. WDCC Metadaten und OAI-PMH O p e n A r c h i v e s I n i t i a t i v e Protocol for Metadata Harvesting

21. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 21 WDCC support of OAI-PMH requests Ü 1.Identify get information about a repository 2.ListMetadataFormats list of available metadata formats 3.ListSets list the structure of a repository (sets,...) 4.ListIdentifiers list of all identifiers of a set 5.GetRecord retrieve one individual metadata record 6.ListRecords list records of a set (used for harvesting)

22. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 22 OAI-PMH http Ü http request: - base URL - list of keyword arguments Form: key=value pairs - Request type GET or POST (URI syntax) http response: - responseDate (format: UTCdatetime) - request (request that generated a response) - error (incl. request that generated the error) http://www.openarchives.org/OAI/openarchivesprotocol.html

23. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 23 Ü WDCC OAI server at: (Software: dlese (www.dlese.org) + apache-tomcat 5.5.12 + Java 1.5)www.dlese.org http://uranus.dkrz.de:8080/oai/provider - 35 IPCC experiments with more than 11000 datasets Metadata Format: ISO 19115 C3Grid (http://gsphere.awi.de:8080/gridsphere/gridsphere) - 40 STD-DOI experiments with more than 1700 datasets Metadata Format: DIF GO-ESSP (NDG, http://ndg.badc.rl.ac.uk/)

24. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 24 NDG Ü DIF XMLs WDCC OAI Server WDCC (Software: dlese) OAI Client NDG (dlese) DIF XMLs Provider 2 OAI Server 2 OAI Server n Discovery Portal NDG OAI Harvesting (Pull or Notification) Catalog NDG record 1...n Process Delivery

25. M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 25 URL: http://glue.badc.rl.ac.uk/discovery/http://glue.badc.rl.ac.uk/discovery/ Keyword: ECHAM4