1. INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org ESR Database Access K. Ronneberger,DKRZ, Germany H. Schwichtenberg, SCAI, Germany S. Kindermann, DKRZ, Germany J. Kraus, SCAI, Germany J. Biercamp, DKRZ, Germany

2. Enabling Grids for E-sciencE INFSO-RI-031688 ESR Data access- Genf 28.09.06 2 Structure Data Requirements of ESR Example Climate workflow: –Access via Webservice-interface/Amga –Missing pieces –Future challenge Example Satellite Data: –Access via OGSA-DAI –Implementation –Evaluation

3. Enabling Grids for E-sciencE INFSO-RI-031688 ESR Data access- Genf 28.09.06 3 ESR Data Requirements Metadata and data bases are commonly large data sets, handled by different teams. The RDBMS generally used are MySQL, PostgreSQL or Oracle Many databases already exist the aim is the implementation of an interface with EGEE or at least to access a copy of them. If new bases are created on EGEE they need to be accessible outside Grid. Some metadata and data are only accessible to authorized persons. Others available on web site have rules for publications (acknowledgement, co-author). Many queries concern matching in time and/or space, expressed in geographical coordinates.

4. Enabling Grids for E-sciencE INFSO-RI-031688 ESR Data access- Genf 28.09.06 4 Typical climate workflow Collect & Prepare Visualize 4 Analyse Find & Select Distributed Climate Data Model Data Observation Data Analysis Dataset Result Dataset Scenario data 3 2 Data description 1 What is needed A central metadata catalog based on common and standardized metadata schema Uniform data access interfaces with transparent AA policies

5. Enabling Grids for E-sciencE INFSO-RI-031688 ESR Data access- Genf 28.09.06 5 Grid-enabled climate data access EGEE UI CE (1) Find & Select (Amga Java API) Data Resource Metadata C3Grid data interface Metadata Server Climate Data Workspace Webservice Interface (a) Publish (ISO 19115/19139) (c) Request (jdbc or archive) (d) Retrieve and Preprocess (2) Collect & Prepare (webservice request) (b) Harvest (OAI-PMH) AMGA Metadata Catalog SE WN (f) Register & Store data (gLite) (3) Analyse (jdl job) sh LFC Catalog (4) Visualize (grads) (g) Process (cdo-tools) (e) Transfer (gridftp)

6. Enabling Grids for E-sciencE INFSO-RI-031688 ESR Data access- Genf 28.09.06 6 Potential Impact Offering an alternative to current solutions for the daily workflows Additionally a common platform is provided to share data, tools and resources, supporting collaboration The common metadata scheme, based on international standards can be adapted/extended – by other disciplines – by International partners (discussion with NDG (GB) and ESG (USA) are ongoing)

7. Enabling Grids for E-sciencE INFSO-RI-031688 ESR Data access- Genf 28.09.06 7 Next steps Registering of uploaded and processed files in Amga Grid-enabling the remaining data Data Centers Current Volume Grid enabled DKRZ Archive~4 PB~3 TB WDCs (Climate/Mare) ~200 TB~5 TB IFM Geomar~1 TB~500 GB DWD~200 GB The rest is coming soon… FUB~1 TB PIK~700 GB AWI~300 GB DLR~60 GB

8. Enabling Grids for E-sciencE INFSO-RI-031688 ESR Data access- Genf 28.09.06 8 Future challenges Feedback from EGEE to C3 (publish updated metadata of AMGA for the C3 portal) Mapping and interoperability of the AA infrastructures of EGEE, C3 and DBs Direct and transparent transfer of external files to, and registration in EGEE – That is, automatic selection of a close and free SE for storage

9. Enabling Grids for E-sciencE INFSO-RI-031688 ESR Data access- Genf 28.09.06 9 Validation of GOME/ERS experiment with Lidar data The goal is to develop for a specific case a prototype that includes the needed tools: Example: Two different instruments : Ground-based Lidar, spectrometer aboard the satellite, ERS. The satellite data stored by orbit or pixel; different algorithms The Lidar data stored in monthly files with one profile/night

10. Enabling Grids for E-sciencE INFSO-RI-031688 ESR Data access- Genf 28.09.06 10 OGSA-DAI Installed Environment at SCAI SL 4.1 Web-Service Container: Tomcat 4.1.31 OGSA-DAI OGSI 6.0 with GLOBUS 3.2.1 (TLS by Port 8443) Three different resources today - MySQL 4.1.10 MySQL spatial extensions only support convex polygons - PostgreSQL 7.4.8 + PostGIS (production) PostGIS adds support for geographic objects to Postgres: http://postgis.refractions.net/http://postgis.refractions.net/ - Oracle 10g (also for Bio Applications)

11. Enabling Grids for E-sciencE INFSO-RI-031688 ESR Data access- Genf 28.09.06 11 ES meta data clients : query OGSA-DAI Service SE EGEE UI ES meta data client query lfns data X 509 User Proxy use ES meta data client EGEE Job on WN X 509 User Proxy submits lfns use

12. Enabling Grids for E-sciencE INFSO-RI-031688 ESR Data access- Genf 28.09.06 12 ES meta data clients straight forward installation by SCAI no integration fat client on nodes -- only for Authorisation (Globus ) User Authentication - with grid proxy certificates - mapping to db roles for every user

13. Enabling Grids for E-sciencE INFSO-RI-031688 ESR Data access- Genf 28.09.06 13 Evaluation Advantage: access to existing databases - nothing to convert out-of-the box installation easy to extend by own classes “quasi industrial standard” multiple resources with multiple services Disadvantage: not fast scalable over the resources ? not integrated in gLite