Presentation is loading. Please wait.

Presentation is loading. Please wait.

18.09.08 / S. Kindermann / DKRZ GO-ESSP 20081 C3-Grid * Federation System for Climate Data Handling Stephan Kindermann German Climate Computing Center.

Similar presentations


Presentation on theme: "18.09.08 / S. Kindermann / DKRZ GO-ESSP 20081 C3-Grid * Federation System for Climate Data Handling Stephan Kindermann German Climate Computing Center."— Presentation transcript:

1 / S. Kindermann / DKRZ GO-ESSP C3-Grid * Federation System for Climate Data Handling Stephan Kindermann German Climate Computing Center – DKRZ * Collaborative Climate Community Grid Project (Part of D-Grid Initiative)

2 / S. Kindermann / DKRZ GO-ESSP  C3Grid Overview: Architecture, Partners, Goals..  C3Grid Federation System Components:  C3Grid ISO Discovery Metadata and Metadata Catalog A short interop. study: C3Grid ISO Metadata / Geonetwork  Data Access and Preprocessing  C3Grid Security Overview  C3Grid / IPCC ?

3 / S. Kindermann / DKRZ GO-ESSP C3Grid Data and Job Management Middleware D-Grid (SRM, d-cache,..) D-Grid (SRM, d-cache,..) C3Grid: Overview World Data Centers Research Institutes ClimateMareRSATPIKGKSSAWIMPI-M Universities FU Berlin Uni Köln Data Access Interface DWD ISO Discovery Metadata Data + Metadata Workflow Data + Metadata Grid Data / Job Interface ISO Discovery Catalog Result Data Products + Metadata C3Grid Data Providers Collaborative Grid Workspace (A)(B) ? ! IFM- Geomar DKRZ Portal C3RC

4 / S. Kindermann / DKRZ GO-ESSP (A) Metadata for Data Discovery: Design and Implementation Data Access InterfaceISO Discovery Metadata ISO Discovery Catalog C3Grid Data Providers (A) ?

5 / S. Kindermann / DKRZ GO-ESSP (A) Metadata – harvesting and lookup components Fast Range Queries Java API + Web Service Interface made available on sourceforge.net see also: Technology  ISO 19115/19139 metadata profile  OAI-PMH harvesting  catalogue  lucene based catalogue search  GridSphere based portal

6 / S. Kindermann / DKRZ GO-ESSP (A) C3Grid ISO profile Design criteria:  no schema extensions, profiling by restriction  restriction using schematron constraints  „the granularity of the discovery metadata should reflect the logical organization of the data repository at a sufficiently coarse grained level“ (1)  CF based content description  Link to resource metadata infrastructure (GT4-MDS based) (1) Inspire: DT Metadata – Draft Implementing Rules for Metadata (version 2, 02/02/2007)

7 / S. Kindermann / DKRZ GO-ESSP (A) C3Grid ISO Profile Description at aggregate level (e.g. experiment)  Aggregate extent description with multiple verticalExtent sections  Sub-selection in data request

8 / S. Kindermann / DKRZ GO-ESSP (A) C3Grid ISO Profile: CF usage air_temperature K sea_surface_temperature K Reference to vertical CRS Content description based on (extended) CF names  Link to corresponding vertical CRS

9 / S. Kindermann / DKRZ GO-ESSP (A) C3Grid ISO profile Data Distributor Info: reference to C3Grid resource metadata catalog (MDS) (names  service endpoints) (optional: service endpoints)

10 / S. Kindermann / DKRZ GO-ESSP (A) C3Grid ISO profile Data provenance description: by now (data staging output): simple sequence of ProcessStep descriptions later (c3grid processed data): combined Source/ProcessStep blocks + external data provenance store

11 / S. Kindermann / DKRZ GO-ESSP

12 / S. Kindermann / DKRZ GO-ESSP C3Grid ISO Profile: A short geonetwork experiment Federation building: OAI-PMH, WebDAV, Z39.50, geonet Full ISO metadata support (ISO19139/19119) OGC CSW 2.0 reference impl. RSS and GeoRSS newsfeeds SKOS based thesauri adaptable to new schema`s schematron constraint checking On roadmap: flexible ISO profile support shibboleth integration

13 / S. Kindermann / DKRZ GO-ESSP C3Grid ISO Profile: A short geonetwork experiment

14 / S. Kindermann / DKRZ GO-ESSP Building complex metadata federations … Harvesting via: CSW OAI-PMH Geneonet Web-Dav

15 / S. Kindermann / DKRZ GO-ESSP C3Grid ISO Profile: A short geonetwork experiment Import / Edit / Search: ok Missing: content (CF) search vertical search temporal BBox search data staging

16 / S. Kindermann / DKRZ GO-ESSP

17 / S. Kindermann / DKRZ GO-ESSP

18 / S. Kindermann / DKRZ GO-ESSP  complete portal protoype to seach, access (pre-process) data described by C3Grid ISO profile in 3 weeks based on geonetwork open source solution..

19 / S. Kindermann / DKRZ GO-ESSP

20 / S. Kindermann / DKRZ GO-ESSP (B) Data Access and Preprocessing World Data CentersResearch InstitutesUniversity Partners Data Access InterfaceISO Discovery Metadata Data + Metadata Data Analysis Workflow Data + Metadata ISO Discovery Catalog Result Data Products + Metadata C3Grid Data Providers Collaborative Grid Workspace (A) (B) ? !

21 / S. Kindermann / DKRZ GO-ESSP Data Staging Request Data IDs Output Properties Selection: lon, lat, alt time content: CF Data Staging Web Service DB Flat File Archive Distributed C3Grid Work Space (B) Data Access and Pre-Processing: Implementation Offer Time / resource estimation skeleton impl status.. Provider staging jars Provider staging scripts MD DB WS GRAM JSDL based description Processing jobs Local resource manager C3Grid Generation 1: secured plain web services (status) C3Grid Generation 2: WSRF service interfaces (scheduled november 08) Generation 2+: full PKI/SAML security stack

22 / S. Kindermann / DKRZ GO-ESSP C3Grid Middleware Components Scheduler: Globus WSRF based, accepts WSL workflow description: compute tasks + data staging tasks Datamanagement: Globus WSRF based, offer negotiation with scheduler, consistent view to distributed data, (later: replica management, caching) Globus MDS Resource Metadata Catalog: service registry, resource status  Dependency on Globus SW stack, no high level impl. support tools, impl. Globus 4.1.x migration ??, problems with delegation impl. (insufficient docu. and guidance)

23 / S. Kindermann / DKRZ GO-ESSP C3Grid Workflow Analysis workflow-related task-related analysis and preparation of workflows monitoring and management of workflow execution (individual) scheduling strategy to optimize the management Handler to facade single/ specific Tasks interaction an moitoring via WS Notification standard

24 / S. Kindermann / DKRZ GO-ESSP (C) Security Infrastructure Identity Provider Home Organisation Attribute Provider Virtual Organisation MyProxy Delegation Service Grid Service Grid Resource GRAM / DataRAM C3Grid Middleware GridShib SAML tools wflow client SLCS (CA) SLCS (CA) X509 Grid- proxy GridShib for GT policy Portal <..SAML Assertions..> SAML Personal / Group Account „Home attributes + VO attributes“ DFN Browser Webstart app Shibb. login

25 / S. Kindermann / DKRZ GO-ESSP (C) Security Infrastructure Status: Shibb IdP`s running at core C3Grid partners Online CA for short-lived credentials tested, set up & operated by DFN (the German NREN) Online CA (DFN-SLCS) accreditation process with EUGridPMA started SLC contain campus attributes as SAML assertion Java Webstart app to bootstrap SLCS in development at DFN GridShib SAML Tools (v0.6.0) tested Prototype of shibbolized GridSphere portal tested open issues with GT4 proxy-delegation implementation Next: Integration of components Virtual home organization for C3 users without a Shibboleth IdP Integration of VO attributes (shibbolized VOMS)

26 / S. Kindermann / DKRZ GO-ESSP C3Grid / IPCC Use Case (0) IPCC Metadata harvested / mirrored in CERA DB (WDCC) (1) Metadata visible in C3 Portal (2) User issues IPCC data import from external repository (3) User  OpenID IdP / + IPCC_Access role  external repos (4) Download  ??  C3 Repository (5) C3Grid grants access to users with IPCC_Access role ‘grant procedure ?’: before each wflow exec. contact to IdP/AttributeService ?? or more offline method ? C3RC / C3 Workspace IPCC data import Wflow result publication Analysis wflow

27 / S. Kindermann / DKRZ GO-ESSP Appendix

28 / S. Kindermann / DKRZ GO-ESSP C3Grid Content Info (Version 2) CF_name_with_attribute air_temperature with a cell_methods attribute including time:mean (interval: 1 day) K

29 / S. Kindermann / DKRZ GO-ESSP Security Aspect: C3Grid step 0  step 1

30 / S. Kindermann / DKRZ GO-ESSP

31 / S. Kindermann / DKRZ GO-ESSP

32 / S. Kindermann / DKRZ GO-ESSP Lucene+ Index (C) Data Reuse of Analysis Results: Metadata Generation OAI-Harvester WS Interface C3Grid Workspace wflow m_tool OAI-PMH Server Portal p_data parent process step source collection Time stamp Description Citation info Description * * 0..1 is_part_of has_parent is_generated_by has_input “quality check” API Prototype (Python) Context description of Analysis Data: Aggregation Processing history


Download ppt "18.09.08 / S. Kindermann / DKRZ GO-ESSP 20081 C3-Grid * Federation System for Climate Data Handling Stephan Kindermann German Climate Computing Center."

Similar presentations


Ads by Google