We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byMelanie King
Modified over 4 years ago
© 2007 Open Grid Forum Data Management Challenge - The View from OGF OGF22 – February 28, 2008 Cambridge, MA, USA Erwin Laure David E. Martin Data Area Directors
© 2007 Open Grid Forum 2 Early Grid View of Grids Early Grid systems had a quite simplistic view: 1.Dispatch a job to machine 2.GridFTP files to the machine from Somewhere 3.Run the job 4.GridFTP results to Somewhere Grids defined Computing Elements (CE) Data and storage was considered to be there Storage Elements (SE) concept came much later Barely OK for Initial Data Analysis Physics, Geosciences, etc
© 2007 Open Grid Forum Then Data kicked in … Compute jobs have to deal with input/output data, transient data Data is Heterogeneous (storage, data formats) Distributed Independently managed 3
© 2007 Open Grid Forum 4 The Grid Grows Up Databases Access DAIS Storage/File Management SRM File/Data Transfer gridFTP, RTF, FTS Data Location RLS, LFC Metadata Data Management Systems SRB …
© 2007 Open Grid Forum 5 Client SRM Storage 5 1 2 1.The client asks the SRM for the file providing an SURL (Site URL) 2.The SRM asks the storage system to provide the file 3.The storage system notifies the availability of the file and its location 4.The SRM returns a TURL (Transfer URL), i.e. the location from where the file can be accessed 5.The client interacts with the storage using the protocol specified in the TURL 3 4 SRM Interactions
© 2007 Open Grid Forum 6 MySQL OGSA-DAI service Engine SQLQuery JDBC Data Resources Activities DB2 GZipGridFTPXPath XMLDB XIndice readFile File SWISS PROT XSLT SQL Server Data- bases Application Client Toolkit
© 2007 Open Grid Forum 7 GridFTP and RFT Control Data Control Data Control Data Control Data globus-url-copyRFT Service RFT Client SOAP Messages Notifications (Optional)
© 2007 Open Grid Forum 8 gLite FTS Logical unit of management Represent a directed network pipe between two sites Mono-directional, Dedicated link Independently manageable State Number of streams Number of concurrent transfers Inter-VO scheduling VO share No Routing involved Non-dedicated channels E.g. star channel
© 2007 Open Grid Forum Data Management in Production Grids 9 SRB as a Data Grid SRB MCAT DB SRB Data Grid has arbitrary number of servers Complexity is hidden from users
© 2007 Open Grid Forum 10 Need for Grid Data Architecture and Standards OGF OGSA Data Architecture WG Started in October 2005 Data Architecture document published as GFD.121
© 2007 Open Grid Forum OGSA-Data Architecture 11 Sink/ Source Access Description Access Description Storage Managed Storage Stored Data Resources Other Data Resources Service interface Resource interface Client APIs (non-OGSA) / Other services Data Service Storage Management
© 2007 Open Grid Forum OGSA-Data: Data Replication/Transfer 12 Sink/ Source Transfer Access Sink/ Source Description Access Description ReplicationTransfer Data Resources Service interface Resource interface Transfer Protocols Client APIs (non-OGSA) / Other services Data Service Replication
© 2007 Open Grid Forum OGF Data Area WGs I Data Format Description Language WG (dfdl-wg) Describe the structure of binary and character encoded files and data streams Database Access and Integration Services WG (dais-wg) Provide consistent access to existing, autonomously managed databases from web services Grid File System Working Group (gfs-wg) Service interface(s) and architecture of a logical file system Grid Storage Management WG (gsm-wg) Provide dynamic space allocation and file management of shared storage components on the Grid (Storage Resource Manager – SRM) GridFTP WG (gridftp-wg) Improvements of FTP suitable for grid applications. 13
© 2007 Open Grid Forum OGF Data Area WGs II Info Dissemination WG (infod-wg) Develop a model for Information Dissemination OGSA ByteIO Working Group (byteio-wg) Define a minimal Web Service interface for providing "POSIX-like" file functionality OGSA Data Movement Interface WG (ogsa-dmi-wg) Managed data movement OGSA-Data Working Group (ogsa-d-wg) Data Architecture 14
© 2007 Open Grid Forum Activities related to file system and data movement GFS: Resource Namespace Service Specification (GFD.101) Byte-IO: Byte-IO OGSA WSRF Basic Profile Rendering (GFD.88) GSM The Storage Resource Manager Interface Specification Version 2.2 (in public comment) DMI OGSA-DMI Specification (in public comment) 15
© 2007 Open Grid Forum Data Architecture: Gaps Standardized metadata Identify query languages, data formats, transport protocols, … Needed in DAIS, DMI, ByteIO, … Data catalogs & Registries Discovery an important part of Grids Replication/Caching Data Federation 16
© 2007 Open Grid Forum 17 Standards Gaps Caching and Replication Integrated Data Management Transactions in a Grid Storage Provisioning Virtualization Provenance, Integrity, Policy File Metadata Streaming Versioning
© 2007 Open Grid Forum 18 Standards Gaps Dependencies Security: IETF, OGF Management: DMTF, SNIA WS-*: OASIS and W3C
© 2007 Open Grid Forum Main Focus for Future Work File systems NFSv4, pNFS Interface to Metadata stores Policies (not only Data) Name your favorite 19 Where can we exploit synergies with SNIA?
…to Ontology Repositories Mathieu dAquin Knowledge Media Institute, The Open University From…
Tom Sugden EPCC OGSA-DAI Future Directions OGSA-DAI User's Forum GridWorld 2006, Washington DC 14 September 2006.
1 Reliable File Transfer Service Ravi K Madduri Argonne National Laboratory, University of Chicago.
National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Data Grids for Collection Federation Reagan W. Moore University.
© 2008 Open Grid Forum Data Grid Federation by RNS GFS-WG, OGF23 Balcelona Hideo Matsuda Osaka University / NAREGI.
© 2007Open Grid Forum OGF22, 25th February 2008 OGSA Data Architecture Mario Antonioletti.
© 2007Open Grid Forum GGF19, 1'st February 2007 OGSA Data Architecture Services Dave Berry & Allen Luniewski.
© 2006 Open Grid Forum GGF18, 13th September 2006 OGSA Data Architecture Scenarios Dave Berry & Stephen Davey.
Oct, 26 th, 2010 OGF 30, NSI-WG: Network Service Interface working group Web Services Overview Web Services for NSI protocol implementation
Grid Tech Team Certificates, Monitoring, & Firewall September 15, 2003 Chiang Mai, Thailand Allan Doyle, NASA With the help of the entire Grid Tech Team.
18 Copyright © 2005, Oracle. All rights reserved. Distributing Modular Applications: Introduction to Web Services.
Abstraction Layers Why do we need them? –Protection against change Where in the hourglass do we put them? –Computer Scientist perspective Expose low-level.
Enterprise Java and Data Services Designing for Broadly Available Grid Data Access Services.
Open Grid Service Architecture - Data Access & Integration (OGSA-DAI) Dr Martin Westhead Principal Consultant, EPCC Telephone: Fax:+44.
Eldas 1.0 Enterprise Level Data Access Services Design Issues, Implementation and Future Development Davy Virdee.
Current status of grids: the need for standards Mike Mineter TOE-NeSC, Edinburgh.
OMII-UK Steven Newhouse, Director. © 2 OMII-UK aims to provide software and support to enable a sustained future for the UK e-Science community and its.
Overview Environment for Internet database connectivity
Server Access The REST of the Story David Cleary
Data Management Expert Panel. RLS Globus-EDG Replica Location Service u Joint Design in the form of the Giggle architecture u Reference Implementation.
© 2018 SlidePlayer.com Inc. All rights reserved.