M.Lautenschlager (WDCC / MPI-M) / / 1 GO-ESSP at LLNL Livermore, June 19th – 21st, 2006 World Data Center Climate: Status and Portal Integration WDCC Home: / WDCC Contact: Michael Lautenschlager, Hannes Thiemann and Frank Toussaint ICSU World Data Center Climate Model and Data / Max-Planck-Institute for Meteorology Hamburg, Germany

M.Lautenschlager (WDCC / MPI-M) / / 2 Content: WDCC Status CERA Concept Portal Integration

M.Lautenschlager (WDCC / MPI-M) / / 3 WDCC Content ERA40 IPCC CEOP BALTEX HOAPS CARIBIC WOCE ERA15/40 NCEP GEBCO COSMOS MPI, GKSS,… Data from Earth System Modelling and Related Observations EH5/MPI-OM IPCC-AR4 Start: Approved in January 2003 Maintenance: Model and Data (M&D/MPI-M) and German Climate Computing Centre (DKRZ) June 2006: 590 Experiments / Data Sets

M.Lautenschlager (WDCC / MPI-M) / / 4 Data Export from WDC Climate Corresponds to 2 – 10 TB/month

M.Lautenschlager (WDCC / MPI-M) / / 5 Geographical Distribution of WDCC Users Total number of registered users: 750 (Mai 2006)

M.Lautenschlager (WDCC / MPI-M) / / 6 Data Import into WDC Climate 6 * 10**9 BLOBs ECHAM5/MPI-OM IPCC AR4 Scenarios (ca. 110 TB)

M.Lautenschlager (WDCC / MPI-M) / / 7 (I) Data catalogue and Pointer to Unix files  Enable search and identification of data  Allow for data access as they are (coarse granularity raw data files) (II) Application-oriented data storage in BLOB tables  Time series of individual variables are stored as BLOB entries in DB Tables (fine granularity data products) Allow for fast and selective data access  Storage in standard data format (GRIB, NetCDF/CF) Allow for application of standard data processing routines (PINGOs, CDOs) CERA 1) Concept: Semantic Data Management 1) Climate and Environmental data Retrieval and Archiving

M.Lautenschlager (WDCC / MPI-M) / / 8 Level 1 - Interface: Metadata entries (XML, ASCII) + Data Files Level 2 – Interf.: Separate files containing BLOB table data in application adapted structure (time series of single variables) Experiment Description Pointer to Unix-Files Dataset 1 Description Dataset n Description BLOB Data Table BLOB Data Table WDCC Data Topology BLOB DB Table corresponds to scalable, virtual file at the operating system level.

M.Lautenschlager (WDCC / MPI-M) / / 9 CERA Data Model Entry Reference Status Distribution Contact Coverage Parameter Spatial Reference Local Adm. Data Access Data Org

M.Lautenschlager (WDCC / MPI-M) / / 10 Data matrix of model experiment Model variables Model Run Time 2 D: small BLOBS (180 KB) 3 D: large BLOBS (3 MB) Raw data file: direct model output (1.3 – 16.2 GB) Each columm is one BLOB Table in CERA-DB Raw data file inDKRZ Archive

M.Lautenschlager (WDCC / MPI-M) / / 11 Preferred DB-storage structure for web-based access: single variable single level time series of 2D gridded data records Formats: GRIB-1 – NetCDF/CF (- GRIB-2) Climate Model Data Structures Application related data structure (2-D)original data structure (4-D)

M.Lautenschlager (WDCC / MPI-M) / / 12 TX7: Intel Itanium-2 with Linux DKRZ Architecture

M.Lautenschlager (WDCC / MPI-M) / / 13 Portal Integration Two strategies: One way integration: discovery and use metadata are integrated in a central data portal in one step Example: C3Grid data catalogue (refer to presentation from Heinrich Widmann) Two way integration: discovery metadata are integrated in central data portal, use metadata are extracted from remote archive when they are needed for data download and processing Example: Primary data publication in TIB library catalogue (STD-DOI) WDCC integration in NDG (NERC Data Grid)

M.Lautenschlager (WDCC / MPI-M) / / 14 Primary data publication (STD-DOI) URL: Data Review Primary Data Publication Process ISO 690-2: Metadata for citation of electronic media

M.Lautenschlager (WDCC / MPI-M) / / 15 Example: Publ.-DOI from WDCC

M.Lautenschlager (WDCC / MPI-M) / / 16 DOI URN

M.Lautenschlager (WDCC / MPI-M) / / 17 Publ.-DOI

M.Lautenschlager (WDCC / MPI-M) / / GB

M.Lautenschlager (WDCC / MPI-M) / / 19 Data retrieval procudure is given at the end (user identification is required) Ident.-DOI

WDCC Metadaten und OAI-PMH O p e n A r c h i v e s I n i t i a t i v e Protocol for Metadata Harvesting

M.Lautenschlager (WDCC / MPI-M) / / 21 WDCC support of OAI-PMH requests Ü 1.Identify get information about a repository 2.ListMetadataFormats list of available metadata formats 3.ListSets list the structure of a repository (sets,...) 4.ListIdentifiers list of all identifiers of a set 5.GetRecord retrieve one individual metadata record 6.ListRecords list records of a set (used for harvesting)

M.Lautenschlager (WDCC / MPI-M) / / 22 OAI-PMH http Ü http request: - base URL - list of keyword arguments Form: key=value pairs - Request type GET or POST (URI syntax) http response: - responseDate (format: UTCdatetime) - request (request that generated a response) - error (incl. request that generated the error)

M.Lautenschlager (WDCC / MPI-M) / / 23 Ü WDCC OAI server at: (Software: dlese ( + apache-tomcat Java 1.5) IPCC experiments with more than datasets Metadata Format: ISO C3Grid ( STD-DOI experiments with more than 1700 datasets Metadata Format: DIF GO-ESSP (NDG,

M.Lautenschlager (WDCC / MPI-M) / / 24 NDG Ü DIF XMLs WDCC OAI Server WDCC (Software: dlese) OAI Client NDG (dlese) DIF XMLs Provider 2 OAI Server 2 OAI Server n Discovery Portal NDG OAI Harvesting (Pull or Notification) Catalog NDG record 1...n Process Delivery

M.Lautenschlager (WDCC / MPI-M) / / 25 URL: Keyword: ECHAM4