Presentation is loading. Please wait.

Presentation is loading. Please wait.

18.09.08 / S. Kindermann / DKRZ GO-ESSP 20081 C3-Grid * Federation System for Climate Data Handling Stephan Kindermann German Climate Computing Center.

Similar presentations


Presentation on theme: "18.09.08 / S. Kindermann / DKRZ GO-ESSP 20081 C3-Grid * Federation System for Climate Data Handling Stephan Kindermann German Climate Computing Center."— Presentation transcript:

1 18.09.08 / S. Kindermann / DKRZ GO-ESSP 20081 C3-Grid * Federation System for Climate Data Handling Stephan Kindermann German Climate Computing Center – DKRZ * Collaborative Climate Community Grid Project (Part of D-Grid Initiative)

2 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 2  C3Grid Overview: Architecture, Partners, Goals..  C3Grid Federation System Components:  C3Grid ISO Discovery Metadata and Metadata Catalog A short interop. study: C3Grid ISO Metadata / Geonetwork  Data Access and Preprocessing  C3Grid Security Overview  C3Grid / IPCC ?

3 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 3 C3Grid Data and Job Management Middleware D-Grid (SRM, d-cache,..) D-Grid (SRM, d-cache,..) C3Grid: Overview World Data Centers Research Institutes ClimateMareRSATPIKGKSSAWIMPI-M Universities FU Berlin Uni Köln Data Access Interface DWD ISO Discovery Metadata Data + Metadata Workflow Data + Metadata Grid Data / Job Interface ISO 19139 Discovery Catalog Result Data Products + Metadata C3Grid Data Providers Collaborative Grid Workspace (A)(B) ? ! IFM- Geomar DKRZ Portal C3RC

4 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 4 (A) Metadata for Data Discovery: Design and Implementation Data Access InterfaceISO Discovery Metadata ISO 19139 Discovery Catalog C3Grid Data Providers (A) ?

5 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 5 (A) Metadata – harvesting and lookup components Fast Range Queries Java API + Web Service Interface made available on sourceforge.net see also: http://www.panfmp.org Technology  ISO 19115/19139 metadata profile  OAI-PMH harvesting  catalogue  lucene based catalogue search  GridSphere based portal

6 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 6 (A) C3Grid ISO 19139 profile Design criteria:  no schema extensions, profiling by restriction  restriction using schematron constraints  „the granularity of the discovery metadata should reflect the logical organization of the data repository at a sufficiently coarse grained level“ (1)  CF based content description  Link to resource metadata infrastructure (GT4-MDS based) (1) Inspire: DT Metadata – Draft Implementing Rules for Metadata (version 2, 02/02/2007)

7 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 7 (A) C3Grid ISO Profile Description at aggregate level (e.g. experiment)  Aggregate extent description with multiple verticalExtent sections  Sub-selection in data request

8 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 8 (A) C3Grid ISO Profile: CF usage air_temperature K sea_surface_temperature K Reference to vertical CRS Content description based on (extended) CF names  Link to corresponding vertical CRS

9 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 9 (A) C3Grid ISO profile Data Distributor Info: reference to C3Grid resource metadata catalog (MDS) (names  service endpoints) (optional: service endpoints)

10 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 10 (A) C3Grid ISO profile Data provenance description: by now (data staging output): simple sequence of ProcessStep descriptions later (c3grid processed data): combined Source/ProcessStep blocks + external data provenance store

11 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 11

12 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 12 C3Grid ISO Profile: A short geonetwork experiment Federation building: OAI-PMH, WebDAV, Z39.50, geonet Full ISO metadata support (ISO19139/19119) OGC CSW 2.0 reference impl. RSS and GeoRSS newsfeeds SKOS based thesauri adaptable to new schema`s schematron constraint checking On roadmap: flexible ISO profile support shibboleth integration

13 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 13 C3Grid ISO Profile: A short geonetwork experiment

14 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 14 Building complex metadata federations … Harvesting via: CSW OAI-PMH Geneonet Web-Dav

15 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 15 C3Grid ISO Profile: A short geonetwork experiment Import / Edit / Search: ok Missing: content (CF) search vertical search temporal BBox search data staging

16 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 16

17 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 17

18 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 18  complete portal protoype to seach, access (pre-process) data described by C3Grid ISO profile in 3 weeks based on geonetwork open source solution..

19 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 19

20 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 20 (B) Data Access and Preprocessing World Data CentersResearch InstitutesUniversity Partners Data Access InterfaceISO Discovery Metadata Data + Metadata Data Analysis Workflow Data + Metadata ISO Discovery Catalog Result Data Products + Metadata C3Grid Data Providers Collaborative Grid Workspace (A) (B) ? !

21 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 21 Data Staging Request Data IDs Output Properties Selection: lon, lat, alt time content: CF Data Staging Web Service DB Flat File Archive Distributed C3Grid Work Space (B) Data Access and Pre-Processing: Implementation Offer Time / resource estimation skeleton impl status.. Provider staging jars Provider staging scripts MD DB WS GRAM JSDL based description Processing jobs Local resource manager C3Grid Generation 1: secured plain web services (status) C3Grid Generation 2: WSRF service interfaces (scheduled november 08) Generation 2+: full PKI/SAML security stack

22 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 22 C3Grid Middleware Components Scheduler: Globus WSRF based, accepts WSL workflow description: compute tasks + data staging tasks Datamanagement: Globus WSRF based, offer negotiation with scheduler, consistent view to distributed data, (later: replica management, caching) Globus MDS Resource Metadata Catalog: service registry, resource status  Dependency on Globus SW stack, no high level impl. support tools, impl. Globus 4.1.x migration ??, problems with delegation impl. (insufficient docu. and guidance)

23 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 23 C3Grid Workflow Analysis workflow-related task-related analysis and preparation of workflows monitoring and management of workflow execution (individual) scheduling strategy to optimize the management Handler to facade single/ specific Tasks interaction an moitoring via WS Notification standard

24 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 24 (C) Security Infrastructure Identity Provider Home Organisation Attribute Provider Virtual Organisation MyProxy Delegation Service Grid Service Grid Resource GRAM / DataRAM C3Grid Middleware GridShib SAML tools wflow client SLCS (CA) SLCS (CA) X509 Grid- proxy GridShib for GT policy Portal <..SAML Assertions..> SAML Personal / Group Account „Home attributes + VO attributes“ DFN Browser Webstart app Shibb. login

25 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 25 (C) Security Infrastructure Status: Shibb IdP`s running at core C3Grid partners Online CA for short-lived credentials tested, set up & operated by DFN (the German NREN) Online CA (DFN-SLCS) accreditation process with EUGridPMA started SLC contain campus attributes as SAML assertion Java Webstart app to bootstrap SLCS in development at DFN GridShib SAML Tools (v0.6.0) tested Prototype of shibbolized GridSphere portal tested open issues with GT4 proxy-delegation implementation Next: Integration of components Virtual home organization for C3 users without a Shibboleth IdP Integration of VO attributes (shibbolized VOMS)

26 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 26 C3Grid / IPCC Use Case (0) IPCC Metadata harvested / mirrored in CERA DB (WDCC) (1) Metadata visible in C3 Portal (2) User issues IPCC data import from external repository (3) User  OpenID IdP / + IPCC_Access role  external repos (4) Download  ??  C3 Repository (5) C3Grid grants access to users with IPCC_Access role ‘grant procedure ?’: before each wflow exec. contact to IdP/AttributeService ?? or more offline method ? C3RC / C3 Workspace IPCC data import Wflow result publication Analysis wflow

27 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 27 Appendix

28 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 28 C3Grid Content Info (Version 2) CF_name_with_attribute air_temperature with a cell_methods attribute including time:mean (interval: 1 day) K

29 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 29 Security Aspect: C3Grid step 0  step 1

30 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 30

31 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 31

32 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 32 Lucene+ Index (C) Data Reuse of Analysis Results: Metadata Generation OAI-Harvester WS Interface C3Grid Workspace wflow m_tool OAI-PMH Server Portal p_data parent process step source collection Time stamp Description Citation info Description * + + 0..* 0..1 is_part_of has_parent is_generated_by has_input “quality check” API Prototype (Python) Context description of Analysis Data: Aggregation Processing history


Download ppt "18.09.08 / S. Kindermann / DKRZ GO-ESSP 20081 C3-Grid * Federation System for Climate Data Handling Stephan Kindermann German Climate Computing Center."

Similar presentations


Ads by Google