1. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 1 Data Discovery and Basic Processing within the German Collaborative Climate Community Data and Processing Grid (C3Grid) Project Heinrich Widmann and Stephan Kindermann Model and Data / DKRZ / Max-Planck-Institute for Meteorology Hamburg, Germany GO-ESSP at LLNL Livermore, June 19th – 21st, 2006 C3Grid Home: www.c3grid.de

2. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 2 Overview C3Grid Background Data Analysis Workflows C3Grid Architecture and Interfaces Data Discovery and Metadata in C3- Grid Data Information Service with Lucene Data Access and Preprocessing Summary

3. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 3 C3Grid Background C3Grid –Status : month 10 of 36 (phase 1) –is the earth system science community grid within the German D-Grid initiative –D-Grid includes five further community grid projects ( AstroGrid, HEP-Grid, InGrid, MediGrid, TextGrid ) –is a community driven grid  Goal is to develop a grid infrastructure appropriate for typical climate analysis workflows  Stepwise introduction and integration

4. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 4 Requirements Metadata Discovery Data access (+ preprocessing) Security Scheduling Complex processing  Grid technologies  ISO19115 / ISO19139  OAI-PMH + Lucene  community webservice  Shibboleth  Globus Toolkit 4  WS-GRAM C3Grid Data Analysis Workflow Requirements

5. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 5 C3Grid Architecture and Interfaces Data Discovery Data Access and Basic Processing

6. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 6 C3Grid Data Discovery and Data Access workspace data Scheduling  Data Management Service Portal - Discovery Data Access Web Service oids time/space constraints processing constraints Data request preprocessing data DB Files Prop. Xml Prop. Rel. World Data Centers (Climate,Mare,RSAT), DWD PIK, IFM-Geomar,.. ISO 19115 / 19139 Discovery Use Web server / OAI provider OAI harvester OAI- PMH C3 Metadata catalog workspace resource provider - Workflow composition WS-GRAM Grid Infrastructure Metadata job submission analysis job

7. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 7 C3 ISO 19139 Metadata “Profile” Data Items: gridded data Metadata Database “implicit” Metadata Archive Database Postprocessed Experiment Data 2D single variable time series Post-processing Raw Experiment Data 3D multi variable files

8. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 8 C3Grid Data Information Service with Lucene full-text index harvesting backend Web service frontend Apache Axis + Servlet Container Apache Lucene Portal CERAPangaeaArchiv Webserver OAI-PMH DIS... FieldTermDocument identifierABC:1232 identifierXYZ:2236 identifierMI6:00712 abstractregion2,23,112 abstractpressure3,23 abstracthumid4,33,215,6,4 min_lat030.431 min_lat-023.232 localfile://path/4 inverted index cache for ISO19139 documents indexing of selected fields [T. Langhammber, ZIB, Berlin]

9. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 9 C3Grid Portal – Simple search

10. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 10 C3Grid Portal – Advanced search

11. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 11 C3Grid Data Access and Preprocessing Data access interface – Community-specific webservice (WSDL) – Solutions of the individual institutes will be adapted to support the webservice e.g. triggering of local data processing tools – Support data base and file based storage types –More detailed use metadata will be provided during the extraction process with the data

12. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 12 C3Grid Data Access/Preprocessing Interface data DB Files Data Access Web service Access CDO processing Stage file webservice request contains : ObjectList of OIDs requested CFList of standard names Space constraints Time constraints Target directory File format, e.g. netCDF or grib … SOAP-XML StageFile Request Constraints  necessary processing CF standard names  Local variable names data

13. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 13 Summary Grid development is application driven Discovery is based on –ISO 19115/19139 based metadata catalog –Hierarchical, two-leveled metadata scheme –Text based search in the catalog Data access is implemented by Proprietary C3Grid data access interface (webservice) Part of the use data are provided along with the data extraction

14. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 14 The end

15. H. Widmann (M&D) Data Discovery and Processing within C3Grid GO-ESSP/LLNL / June, 19 th 2006 / 15 C3Grid Architecture DBMS/ File Available Resources Distributed Processing Resources Distributed Data Archives Meta Data Job Data DMS (local) Site C3Grid Components OAI / WS Pre-Proc Grid Workspace Resource Scheduler Base Data & Meta Data File Management Archive Interface Data Transfer Service Distributed Grid Infrastructure GT4 based new Metadata-Service DMS (global) Workflow Scheduler Resource Information Service DIS Staging Search Harvesting Task Execution Matchmaking User Job Submission User Interface API (Web Services)GUI Monitoring