We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byMelanie King
Modified over 2 years ago
© 2007 Open Grid Forum Data Management Challenge - The View from OGF OGF22 – February 28, 2008 Cambridge, MA, USA Erwin Laure David E. Martin Data Area Directors
© 2007 Open Grid Forum 2 Early Grid View of Grids Early Grid systems had a quite simplistic view: 1.Dispatch a job to machine 2.GridFTP files to the machine from Somewhere 3.Run the job 4.GridFTP results to Somewhere Grids defined Computing Elements (CE) Data and storage was considered to be there Storage Elements (SE) concept came much later Barely OK for Initial Data Analysis Physics, Geosciences, etc
© 2007 Open Grid Forum Then Data kicked in … Compute jobs have to deal with input/output data, transient data Data is Heterogeneous (storage, data formats) Distributed Independently managed 3
© 2007 Open Grid Forum 4 The Grid Grows Up Databases Access DAIS Storage/File Management SRM File/Data Transfer gridFTP, RTF, FTS Data Location RLS, LFC Metadata Data Management Systems SRB …
© 2007 Open Grid Forum 5 Client SRM Storage The client asks the SRM for the file providing an SURL (Site URL) 2.The SRM asks the storage system to provide the file 3.The storage system notifies the availability of the file and its location 4.The SRM returns a TURL (Transfer URL), i.e. the location from where the file can be accessed 5.The client interacts with the storage using the protocol specified in the TURL 3 4 SRM Interactions
© 2007 Open Grid Forum 6 MySQL OGSA-DAI service Engine SQLQuery JDBC Data Resources Activities DB2 GZipGridFTPXPath XMLDB XIndice readFile File SWISS PROT XSLT SQL Server Data- bases Application Client Toolkit
© 2007 Open Grid Forum 7 GridFTP and RFT Control Data Control Data Control Data Control Data globus-url-copyRFT Service RFT Client SOAP Messages Notifications (Optional)
© 2007 Open Grid Forum 8 gLite FTS Logical unit of management Represent a directed network pipe between two sites Mono-directional, Dedicated link Independently manageable State Number of streams Number of concurrent transfers Inter-VO scheduling VO share No Routing involved Non-dedicated channels E.g. star channel
© 2007 Open Grid Forum Data Management in Production Grids 9 SRB as a Data Grid SRB MCAT DB SRB Data Grid has arbitrary number of servers Complexity is hidden from users
© 2007 Open Grid Forum 10 Need for Grid Data Architecture and Standards OGF OGSA Data Architecture WG Started in October 2005 Data Architecture document published as GFD.121
© 2007 Open Grid Forum OGSA-Data Architecture 11 Sink/ Source Access Description Access Description Storage Managed Storage Stored Data Resources Other Data Resources Service interface Resource interface Client APIs (non-OGSA) / Other services Data Service Storage Management
© 2007 Open Grid Forum OGSA-Data: Data Replication/Transfer 12 Sink/ Source Transfer Access Sink/ Source Description Access Description ReplicationTransfer Data Resources Service interface Resource interface Transfer Protocols Client APIs (non-OGSA) / Other services Data Service Replication
© 2007 Open Grid Forum OGF Data Area WGs I Data Format Description Language WG (dfdl-wg) Describe the structure of binary and character encoded files and data streams Database Access and Integration Services WG (dais-wg) Provide consistent access to existing, autonomously managed databases from web services Grid File System Working Group (gfs-wg) Service interface(s) and architecture of a logical file system Grid Storage Management WG (gsm-wg) Provide dynamic space allocation and file management of shared storage components on the Grid (Storage Resource Manager – SRM) GridFTP WG (gridftp-wg) Improvements of FTP suitable for grid applications. 13
© 2007 Open Grid Forum OGF Data Area WGs II Info Dissemination WG (infod-wg) Develop a model for Information Dissemination OGSA ByteIO Working Group (byteio-wg) Define a minimal Web Service interface for providing "POSIX-like" file functionality OGSA Data Movement Interface WG (ogsa-dmi-wg) Managed data movement OGSA-Data Working Group (ogsa-d-wg) Data Architecture 14
© 2007 Open Grid Forum Activities related to file system and data movement GFS: Resource Namespace Service Specification (GFD.101) Byte-IO: Byte-IO OGSA WSRF Basic Profile Rendering (GFD.88) GSM The Storage Resource Manager Interface Specification Version 2.2 (in public comment) DMI OGSA-DMI Specification (in public comment) 15
© 2007 Open Grid Forum Data Architecture: Gaps Standardized metadata Identify query languages, data formats, transport protocols, … Needed in DAIS, DMI, ByteIO, … Data catalogs & Registries Discovery an important part of Grids Replication/Caching Data Federation 16
© 2007 Open Grid Forum 17 Standards Gaps Caching and Replication Integrated Data Management Transactions in a Grid Storage Provisioning Virtualization Provenance, Integrity, Policy File Metadata Streaming Versioning
© 2007 Open Grid Forum 18 Standards Gaps Dependencies Security: IETF, OGF Management: DMTF, SNIA WS-*: OASIS and W3C
© 2007 Open Grid Forum Main Focus for Future Work File systems NFSv4, pNFS Interface to Metadata stores Policies (not only Data) Name your favorite 19 Where can we exploit synergies with SNIA?
© 2007Open Grid Forum OGF22, 25th February 2008 OGSA Data Architecture Mario Antonioletti.
© 2007Open Grid Forum GGF19, 1'st February 2007 OGSA Data Architecture Services Dave Berry & Allen Luniewski.
Promoting and Standardizing Grid Computing Defining the Grid: Open Grid Services Architecture Current and Future Generation Grid Technology Summer School.
© 2006 Open Grid Forum GGF18, 13th September 2006 OGSA Data Architecture Scenarios Dave Berry & Stephen Davey.
© 2006 OpenGridForum Standards Orientation OGF-19, Tuesday, January 30, 2007 Lee, Cohen, Subramanian.
Neil Chue Hong Project Manager, EPCC OGSA-DAI data access and integration NERC GridGIS workshop eSI, 1 February.
Grid Interoperability Through Standards Dr. Alistair Dunlop Project Manager.
FP62004Infrastructures6-SSA E-infrastructure shared between Europe and Latin America Architecture of the gLite DMS Claudio Cherubino.
Tecnologia dei Servizi Grid e cloud computing - Lezione 002a 0 Lezione 2a - 14 ottobre 2009 Il materiale didattico usato in questo corso è stato mutuato.
Craig Lee OGF-24 Mark Reichardt Grid-Enabled Geospatial Systems.
INFSO-RI Enabling Grids for E-sciencE EGEE and gLite Slides by: Erwin Laure EGEE Deputy Middleware Manager.
Workflows over Grid-based Web services General framework and a practical case in structural biology gLite 3.0 Data Management David García Aristegui Grid.
Open Grid Service Architecture - Data Access & Integration (OGSA-DAI) Dr Martin Westhead Principal Consultant, EPCC Telephone: Fax:+44.
NeSC Data Projects and Initiatives Dr. Dave Berry Research Manager.
Globus DataGrid Overview Bill Allcock, ANL GridPP Meeting 30 June 2003.
RMS and Scheduling for Future Generation Grids Ramin Yahyapour University Dortmund Leader CoreGRID Institute on Resource Management and Scheduling CoreGRID.
DATABASE SYSTEM CONCEPTS AND ARCHITECTURE CHAPTER 2 1.
Dr. Daniel Sabbah Vice President of Strategy & Technology IBM Software Group Bringing Grid & Web Services Together Globus World San Francisco, CA Tuesday,
Tom Sugden EPCC OGSA-DAI Future Directions OGSA-DAI User's Forum GridWorld 2006, Washington DC 14 September 2006.
© 2008 Open Grid Forum Data Grid Federation by RNS GFS-WG, OGF23 Balcelona Hideo Matsuda Osaka University / NAREGI.
BR 1 SIMDAT HALO meeting – Meteo Activity of the SIMDAT project: Building components of the WIS Baudouin Raoult ECMWF.
Tecnologia dei Servizi Grid e cloud computing - Lezione 003a 0 Lezione 3a - 20 ottobre 2009 Il materiale didattico usato in questo corso è stato mutuato.
Current status of grids: the need for standards Mike Mineter TOE-NeSC, Edinburgh.
December 2009 Data Integration in Grid Environments Alex Poulovassilis, Birkbeck, U. of London.
(2-Tier) (n-Tier) (Component) (Business Components)
Grid Monitoring Futures with Globus Jennifer M. Schopf Argonne National Lab April 2003.
Grids and the Globus Community Dr. Jennifer M. Schopf Argonne National Lab
OGSI Evolution: WS-Resource Framework and WS-Notification Carl Kesselman Globus USC/ISI
Page 1 LAITS Laboratory for Advanced Information Technology and Standards Duh 7/10/03 The GMU Geospatial Grid Technology Development and Application Project.
How Distributed Data Mining Tasks can Thrive as Services on Grids Domenico Talia and Paolo Trunfio Università della Calabria, Italy
© 2016 SlidePlayer.com Inc. All rights reserved.