We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byGrace Whitaker
Modified over 4 years ago
© 2007Open Grid Forum GGF19, 1'st February 2007 OGSA Data Architecture Services Dave Berry & Allen Luniewski
© 2007 Open Grid Forum 2 OGF IPR Policies Apply I acknowledge that participation in this meeting is subject to the OGF Intellectual Property Policy. Intellectual Property Notices Note Well: All statements related to the activities of the OGF and addressed to the OGF are subject to all provisions of Appendix B of GFD-C.1, which grants to the OGF and its participants certain licenses and rights in such statements. Such statements include verbal statements in OGF meetings, as well as written and electronic communications made at any time or place, which are addressed to: the OGF plenary session, any OGF working group or portion thereof, the OGF Board of Directors, the GFSG, or any member thereof on behalf of the OGF, the ADCOM, or any member thereof on behalf of the ADCOM, any OGF mailing list, including any group list, or any other list functioning under OGF auspices, the OGF Editor or the document authoring and review process Statements made outside of a OGF meeting, mailing list or other function, that are clearly not intended to be input to an OGF activity, group or function, are not subject to these provisions. Excerpt from Appendix B of GFD-C.1: Where the OGF knows of rights, or claimed rights, the OGF secretariat shall attempt to obtain from the claimant of such rights, a written assurance that upon approval by the GFSG of the relevant OGF document(s), any party will be able to obtain the right to implement, use and distribute the technology or works when implementing, using or distributing technology based upon the specific specification(s) under openly specified, reasonable, non- discriminatory terms. The working group or research group proposing the use of the technology with respect to which the proprietary rights are claimed may assist the OGF secretariat in this effort. The results of this procedure shall not affect advancement of document, except that the GFSG may defer approval where a delay may facilitate the obtaining of such assurances. The results will, however, be recorded by the OGF Secretariat, and made available. The GFSG may also direct that a summary of the results be included in any GFD published containing the specification. OGF Intellectual Property Policies are adapted from the IETF Intellectual Property Policies that support the Internet Standards Process.
© 2007 Open Grid Forum 3 Contents Current Status Architecture Document Scenarios Document Storage Management Discussion
© 2007 Open Grid Forum 4 Two Informational Documents OGSA Data Architecture 70+ pages Describes services and their interfaces OGSA Data Scenarios 50+ pages Describes how the services can be composed to address particular scenarios
© 2007 Open Grid Forum 5 Progress since GGF18 Slow progress Focus on adding interfaces to Arch. Mostly complete; review/editing in progress Scenarios Document Scenario refinement Integration w/ Architecture document Preliminary work on EMS Scenario Many thanks to Stephen Davey
© 2007 Open Grid Forum 6 Current State Architecture Document Technical content substantially complete Review of interface work in progress Scenarios Document Some scenarios still need work Adding EMS scenario (next session) Dates for public comment missed
© 2007 Open Grid Forum 7 Architecture Document
© 2007 Open Grid Forum 8 Current Scope Files and databases (& storage) Not streams, sessions, … Services and interfaces Storage, Access, Transfer Replication, Caching, Federation, Metadata catalogues Cross-cutting themes Security, Policies, … Part of the bigger OGSA picture E.g. Naming, Workflow, Transactions, Scheduling, Provisioning, …
© 2007 Open Grid Forum 9 Architecture Document Services Data Transfer Data Access Storage Resource Management Data Cache Data Replication Data Federation Metadata Catalogues Appendices Specifications referenced Mappings to specifications DAIS ByteIO SRM DMI … Glossary
© 2007 Open Grid Forum 10 Data Transfer Create Control EPR e.g. GridFTP User Transfer Factory Transfer Instance Source Sink
© 2007 Open Grid Forum 11 Replication Factory CreateReplica() Replica Management Process + Targets ModifyReplicatedContents() SynchroniseReplica() ValidateReplica() Destroy Replica() Replica Catalogue Maps name to list of (replica, target) pairs
© 2007 Open Grid Forum 12 Gaps: Need people to work on Information model for data resources For management, resource reservation, … File metadata URI Registries Security extensions Integration of access and transfer 3 rd -party delivery Policies Replication coherency, caching coherency, catalogue consistency, etc. Sessions Provisioning, etc. Review!
© 2007 Open Grid Forum 13 Scenarios
© 2007 Open Grid Forum 14 Scenarios document Example scenarios of a generic nature Illustrates how the services and interfaces described in the OGSA Data Architecture document can be put together in a selection of typical data scenarios. Not a use case document generating requirements.
© 2007 Open Grid Forum 15 Scenarios Data replication Data pipelining Data integration Personal data service Peer to peer data discovery Data storage Data provenance Grid file system Data transfer Data staging
© 2007 Open Grid Forum 16 Work to be Done Add EMS Scenario (next session) Review & update scenarios Needs an editor
© 2007 Open Grid Forum 17 Storage Management Discussion
© 2007 Open Grid Forum 18 Storage Management Current section derived from SRM spec. It has concepts of: Storage (e.g., disks, volumes) Files Directories File spaces & systems Is this appropriate for the architecture?
© 2007 Open Grid Forum 19 Storage Management (1) Directory Management Functions Synchronous & asynchronous List Files Release File Locks Remove Files Copy Files Move Files Make Directory Delete Directory
© 2007 Open Grid Forum 20 Storage Management (2) Space Management Functions: Reserve Space() Get Space() Release Space() Set Quota() Sink / Source Covered by Transfer
© 2007 Open Grid Forum 21 Discussion
© 2007Open Grid Forum Questions?
© 2007 Open Grid Forum 23 Full Copyright Notice Copyright (C) Open Grid Forum 2006. All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. The limited permissions granted above are perpetual and will not be revoked by the OGF or its successors or assignees.
© 2007 Open Grid Forum 24 Backup
© 2007 Open Grid Forum 25 Basic Structure Access Sink/ Source Description Access Sink/ Source Description TransferLookup Storage Managed Storage Stored Data Resources Other Data Resources Service interface Resource interface Transfer Protocols Client APIs (non-OGSA) / Other services Data Management Other Data Services Storage Management TransferRegistries
© 2007 Open Grid Forum 26 Unstructured access RandomIO Sessionless Read() Write() Truncate() StreamableIO Stateful session resource Read() Write() Not part of data access layer Create() Destroy()
© 2007 Open Grid Forum 27 Structured access BasicExecute() XUpdate() XQuery() SQLExecute() RDF…
© 2007 Open Grid Forum 28 Federation Factory CreateFederation() Federation AddResourceToFederation() RemoveResourceFromFederation() AddAccessMechanismToFederation() RemoveAccessMechanism() UpdateFederationProperties() Destroy()
© 2007 Open Grid Forum 29 Cache Factory Create() Federation ReConfigure() Synchronise() Destroy()
© 2007 Open Grid Forum 30 Data description Formats, encodings, etc Resources: Generic Readable Writable Resources: Specific Files Relational databases XML databases OO databases RDF File Systems …
© 2007 Open Grid Forum 31 Data description: Files CreateTime ModificationTime LastAccessTime Size Encoding Format Lifetime StorageSpace?? Owner / Provenance?
© 2007 Open Grid Forum 32 Data Description: RDBMS Database Schema Query languages Vendor/Version/Extensions Stored procedures / functions ?? Available Free Space Size Physical Schema See DAI-R specs for more
© 2007 Open Grid Forum 33 Data description: XML DB Query languages Vendor/Version/Extensions Available Free Space Size Physical Schema See DAI-X specs for more?
© 2007 Open Grid Forum 34 Data Pipelining Completed Animations Visualisation Service Customer 2 1. Submit job.2. Store results. 3. Transfer results. 4. Return results. Customer 1 Data Transfer Service 3. Transfer results. Rendering Service Data Access Service
© 2007 Open Grid Forum 35 Data Storage – Bringing data online Storage Devices Customer Data Storage Service Transfer Service 1. Make files online. 2. Transfer files. Nearline Storage Online Storage 1. Make online. 3. Retire to nearline.
© 2007 Open Grid Forum 36 Data Replication – 1 Customer 1 Data Transfer Service Replication Service Data Storage 1 Data Storage 2 Data Access Service 2 Data Access Service 1 1b. Publish 2. Transfer copies 6. Update 4. Access data 5. Notify 2. Transfer copies 2. Transfer copies Registry Service 3. Find data 1a. Register data Customer 2
© 2007 Open Grid Forum 37 Data Replication – 2 1. Register Customer 1 Data Transfer Service Data Storage 1 Data Storage 2 Data Access Service 2 Data Access Service 1 2. Transfer copies 6. Update 3. Find data 4. Access data 5. Notify 2. Transfer copies 2. Transfer copies Repli- cation Service Data Service Replica Catalogue Service Customer 2
© 2007 Open Grid Forum 38 Joint OGSA Data + EMS Scenario The steps of this simple scenario are as follows: 1.Submit job to BES container. (JSDL contains execution & data staging info). 2.Use data transfer service to do the required data staging. 3.Run the executable on the BES container with the input data. 4.Stage result output data back to Data Service 1. 5.Delete staged input data at BES container. 6.Delete staged output data BES container.
© 2007 Open Grid Forum 39 Data Staging Data Transfer Service BES Container Input Data (copy) Output Data 1. Submit JSDL script. 2a. Stage input data. Data Service 1 Data Service 2 2b. Transfer input data. 4a. Stage output data. 2a. 4a. 2a. 4a. Client 4b. Transfer output data. BES Container: 3. Run executable & save resulting output data. Input Data Output Data (copy) 5. Delete input data (copy). 6. Delete output data.
© 2007 Open Grid Forum 40 Personal Data Service Customer 1 (site 1) Registry Service Data Service 1 Data Service 2 Data Service 3 Local Cache Service 2 Local Cache Service 1 Index 2. Create named space. 3. Name collection. 1. Locate data. 2. Create. 4. Use named space. Customer 1 (site 2) 6. Use named space. 7. Update. 5. Update. Personal Data Service Global Name Resolver Service
© 2006 Open Grid Forum GHPN-RG Status update co-chairss:Cees de Laat Dimitra Simeonidou GGF22, Boston, February 2008.
© 2006 Open Grid Forum Discussion of File Catalog Standardization GFS-WG, OGF24 Singapore Osamu Tatebe, co-chair of GFS-WG Univ. of Tsukuba Sep 16, 2008.
© 2006 Open Grid Forum JSDL 1.0: Parameter Sweeps OGF 23, June 2008, Barcelona, Spain.
© 2006 Open Grid Forum Network Services Interface OGF30: Connection Services Guy Roberts, 27 th Oct 2010.
© 2006 Open Grid Forum Ellen Stokes, IBM Michel Drescher, Fujitsu Information Model, JSDL and XQuery: A proposed solution OGF-19 Chapel Hill, NC USA.
© 2006 Open Grid Forum Network Services Interface Introduction to NSI Guy Roberts.
© 2006 Open Grid Forum Federated Identity in the Cloud OGF 32, Salt Lake City.
© 2006 Open Grid Forum JSDL 1.0: Parameter Sweeps: Examples OGF 22, February 2008, Cambridge, MA.
© 2006 Open Grid Forum GridRPC Interoperability Test Response to comments Yusuke Tanimura.
© 2006 Open Grid Forum OGF19 Federated Identity Rule-based data management Wed 11:00 AM Mountain Laurel Thurs 11:00 AM Bellflower.
© 2007 Open Grid Forum JSDL-WG Session OGF27 – General Session 10:30-12:00, 14 October 2009 Banff, Canada.
©2010Open Grid Forum OGF28 OGSA-DMI Status Chairs: Mario Antonioletti, EPCC Stephen Crouch, Southampton Shahbaz Memon, FZJ Ravi Madduri, UoC.
© 2007Open Grid Forum OGF22, 25th February 2008 OGSA Data Architecture Mario Antonioletti.
© 2006 Open Grid Forum JSDL Session (CIM Job) OGF 21 - Seattle, 17 October 2007.
© 2006 Open Grid Forum Joint Session on Information Modeling for Computing Resources OGF 20 - Manchester, 7 May 2007.
© 2007 Open Grid Forum JSDL-WG Session OGF21 – Activity schema session 17 October 2007 Seattle, U.S.
© 2006 Open Grid Forum 2 nd March 09 Enterprise Grid Requirements Research Group OGF25 EGR-RG Session Group.
© 2006 Open Grid Forum OGSA Next Steps Discussion Providing Value Beyond the Specifications.
© 2008 Open Grid Forum Resource Selection Services OGF22 – Boston, Feb
© 2007 Open Grid Forum DMTF-OGF Work Register Current and Proposed Items OGF 22 Boston, MA 26 February 2008.
© 2018 SlidePlayer.com Inc. All rights reserved.