Leading the pervasive adoption of grid computing for research and industry © 2006 Global Grid Forum The information contained herein is subject to change without notice The OGSA Data Architecture Services and Scenarios Dave Berry Allen Luniewski OGSA F2F 17 th July 2006

Goals Provide update of status and plans Motivate OGSA WG to read and comment −Overview −Five sample scenarios Action items for OGSA WG −What has OGSA done for us? −Gap analysis

Progress since GGF16 More scenarios −E,g, Provenance, Grid File System More integration −Particularly between scenarios and architecture document −Also raising some issues from individual chapters to cross-cutting concerns More refinement

Current Status Two documents ripe for review −Read Now! −Submission after GGF18 OGSA Data Architecture − −70+ pages −Most sections complete OGSA Data Scenarios − −50+ pages −Active development (E.g. GFS, Metadata catalog)

Current Scope Yes: files, databases & storage No: streams, sessions, … Services and interfaces −Storage, Access, Transfer −Replication, Caching, Federation, Metadata catalogues Cross-cutting themes −Security, Policies, … Scenarios −Data Transfer, Data Integration, Data Staging, Personal Data Profile, Data Discovery, Data Storage, Data Federation, Data Provenance Grid File Systems Part of the bigger OGSA picture − E.g. Naming, Workflow, Transactions, Scheduling, Provisioning, …

Questions for reviewers Does our approach mesh properly with OGSA? Do our scenarios match with EMS architecture, where relevant? Do EMS scenarios match our architecture? Which items can be raised from the OGSA Data Architecture to the main OGSA architecture? Do OGSA capabilities need to be modified to support Data? −E.g. provisioning, naming, reservation, scheduling Should we start working on profiles? −E.g. GFS, File replication, database access −Which WG: OGSA, OGSA Data, GFS, …

Remaining work Interface descriptions −Key to integration −Standard format in architecture document −Reference in scenarios Management interface for cache service More details for replication interfaces GFS scenario Metadata catalog scenario

Roadmap External review −Now Presentation of complete documents −GGF18 Public comment −10/2006 (post GGF18) Final submission −02/2007

Architecture document Overview Architectural Context −Requirements on OGSA Security Data Description Data Transfer Data Access Storage Resource Management Cache Services Data Replication Data Federation Metadata Catalogues Appendices: −Specifications referenced −Glossary

Scenarios document Generic example scenarios −To test the OGSA Data Architecture document. −Not a use case document to generate requirements Instead −Illustrates how the OGSA Data Architecture components and interfaces can be put together to enact a typical set of data scenarios.

Services, resources & interfaces Our understanding of the relationships:

Basic Structure Access Sink/ Source Description Access Sink/ Source Description Transfer Registries Lookup Client APIs (non-OGSA) / Other services Data Management Other Data Services Storage Management Storage Managed Storage Stored Data Resources Other Data Resources Transfer Protocols

Composite Entities Replication Federation Cache Access Sink/ Source Description Data Services

Usage Patterns (for access + transfer) Request-response Third-party delivery Factory Bulk load

Scenario 1/5: Data pipelining Completed Animations Visualisation Service Customer 2 1. Submit job.2. Store results. 3. Transfer results. 4. Return results. Customer 1 Data Transfer Service 3. Transfer results. Rendering Service Data Access Service

Scenario 2/5: Bringing data online Storage Devices Customer Data Storage Service Transfer Service 1. Make files online. 2. Read files. Nearline Storage Online Storage 1. Make online. 3. Retire to nearline.

Scenario 3/5: Data Staging Parameter Space Exploration Service Boundary Conditions Simulation Service Parameter Set Boundary Conditions (copy) Results Set 1. Submit job. 6. Delete boundary condition data. 7. Query results set. 3. Set up Results DB. 5. Store results. 9. Delete Results DB. 2c. Bulk load boundary condition data. 4. Generated jobs from parameter set. 8. Return derived data. Customer 1 2b. Query relevant boundary conditions. 2a. Get EPR for storage & CPUs. Data Service Data Service 2

Scenario 4/5: Replication Customer 1 Data Transfer Service Replication Service Data Storage 2 Data Service 2 Data Service 1 Data Storage 1 1b. Publish 2. Transfer copies 6. Update 4. Access data 5. Notify 2. Transfer copies 2. Transfer copies Registry Service 3. Find data 1a. Register data Customer 2

Customer 1 (site 1) Customer 1 (site 2) Scenario 5/5: Personal Data Service Registry Service Data Service 1 Data Service 2 Data Service 3 Local Cache Service 2 Local Cache Service 1 Index 2. Create named space. 3. Name collection. 1. Locate data. 2. Create. 4. Use named space. 6. Use named space. 7. Update. 5. Update. Personal Data Service Global Name Resolver Service

Some suggested policies Throughput Response time Availability Recovery time Data resiliency Access accuracy Data currency

Common properties for data resources? Readable Writable ConcurrentAccess (boolean) ParentDataResource (URI) −For a derived data resource, this is the name of its parent ChildSensitiveToParent (boolean) ParentSensitiveToChild (boolean)

What does OGSA do for us? Profiles Naming (WS-A, WS-Naming, RNS) Management Security Notification Resource Discovery Policies and agreements Provisioning EMS (inc. Scheduling) Reservation Transactions (WS-Coordination, WS-CAF) Sessions

What generic features can we give to OGSA? Generic resource properties? External name mapping? Others?

Some WS-DAI resource properties DataResourceAbstractName (URI) DataResourceDescription (text) DataResourceManagement −Is the resource managed by the service or externally?

WS-DAI CoreResourceList interface GetDataResourceList () − returns a list of abstract names (URIs) and their corresponding EPRs. Resolve () −allows a data resource name to be mapped to an EPR. Allows clients who know the name used by the resource to get an EPR for use in OGSA Would an embedded RNS service provide the same functionality?

OGSA gap analysis Profiles Naming (GAP? –Integrating with existing names) Management (GAP! – Need info model for data services) Security (GAP! – Data has extra requirements) Notification Resource Discovery (GAP! – doesn’t exist) Policies and agreements (GAP! – Which specifications to use?) Provisioning (GAP! – No investigation of data provisioning) EMS (GAP! –Integration with data functionality & scheduling of data) Reservation (GAP! – Reservation of data services) Transactions Sessions (GAP! – doesn’t exist)

Security Chapter 4 discusses additional requirements (beyond basic AAA) −GAP! – Security policies −GAP! – Attaching security policies to data in motion −GAP! – Geographical location of requester and resource −GAP! – Reason for access −GAP! – Authorisation of sequences of access requests −GAP! – Authorisation based on previous requests

OGSA Data Gaps GAP! – File metadata (POSIX style) GAP! – URI Registries −Data formats −Query languages −Access mechanisms GAP! – Integration of access and transfer −3 rd -party delivery GAP! – Policies −Replication coherency, caching coherency, catalogue consistency, etc. GAP! – Replication specification GAP! – Cache specification GAP! – Federation specification Data transfer was a gap but is now being developed

Actions on OGSA WG Generalise OGSA capabilities to cover OGSA Data −Provisioning, Reservation, Scheduling −Information model Integrate OGSA Data with OGSA EMS −Start with common scenarios −Including HPC profile Generalise OGSA Data features to OGSA? Encourage Data Profiles −E.g. GFS, database access, file replication

Actions on OGF Establish URI registry −Extensible list of topics −WGs should define URIs for existing items −Anyone free to propose new ones Encourage new WGs / Profiles as appropriate