We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byMelanie King
Modified over 3 years ago
© 2007 Open Grid Forum Data Management Challenge - The View from OGF OGF22 – February 28, 2008 Cambridge, MA, USA Erwin Laure David E. Martin Data Area Directors
© 2007 Open Grid Forum 2 Early Grid View of Grids Early Grid systems had a quite simplistic view: 1.Dispatch a job to machine 2.GridFTP files to the machine from Somewhere 3.Run the job 4.GridFTP results to Somewhere Grids defined Computing Elements (CE) Data and storage was considered to be there Storage Elements (SE) concept came much later Barely OK for Initial Data Analysis Physics, Geosciences, etc
© 2007 Open Grid Forum Then Data kicked in … Compute jobs have to deal with input/output data, transient data Data is Heterogeneous (storage, data formats) Distributed Independently managed 3
© 2007 Open Grid Forum 4 The Grid Grows Up Databases Access DAIS Storage/File Management SRM File/Data Transfer gridFTP, RTF, FTS Data Location RLS, LFC Metadata Data Management Systems SRB …
© 2007 Open Grid Forum 5 Client SRM Storage 5 1 2 1.The client asks the SRM for the file providing an SURL (Site URL) 2.The SRM asks the storage system to provide the file 3.The storage system notifies the availability of the file and its location 4.The SRM returns a TURL (Transfer URL), i.e. the location from where the file can be accessed 5.The client interacts with the storage using the protocol specified in the TURL 3 4 SRM Interactions
© 2007 Open Grid Forum 6 MySQL OGSA-DAI service Engine SQLQuery JDBC Data Resources Activities DB2 GZipGridFTPXPath XMLDB XIndice readFile File SWISS PROT XSLT SQL Server Data- bases Application Client Toolkit
© 2007 Open Grid Forum 7 GridFTP and RFT Control Data Control Data Control Data Control Data globus-url-copyRFT Service RFT Client SOAP Messages Notifications (Optional)
© 2007 Open Grid Forum 8 gLite FTS Logical unit of management Represent a directed network pipe between two sites Mono-directional, Dedicated link Independently manageable State Number of streams Number of concurrent transfers Inter-VO scheduling VO share No Routing involved Non-dedicated channels E.g. star channel
© 2007 Open Grid Forum Data Management in Production Grids 9 SRB as a Data Grid SRB MCAT DB SRB Data Grid has arbitrary number of servers Complexity is hidden from users
© 2007 Open Grid Forum 10 Need for Grid Data Architecture and Standards OGF OGSA Data Architecture WG Started in October 2005 Data Architecture document published as GFD.121
© 2007 Open Grid Forum OGSA-Data Architecture 11 Sink/ Source Access Description Access Description Storage Managed Storage Stored Data Resources Other Data Resources Service interface Resource interface Client APIs (non-OGSA) / Other services Data Service Storage Management
© 2007 Open Grid Forum OGSA-Data: Data Replication/Transfer 12 Sink/ Source Transfer Access Sink/ Source Description Access Description ReplicationTransfer Data Resources Service interface Resource interface Transfer Protocols Client APIs (non-OGSA) / Other services Data Service Replication
© 2007 Open Grid Forum OGF Data Area WGs I Data Format Description Language WG (dfdl-wg) Describe the structure of binary and character encoded files and data streams Database Access and Integration Services WG (dais-wg) Provide consistent access to existing, autonomously managed databases from web services Grid File System Working Group (gfs-wg) Service interface(s) and architecture of a logical file system Grid Storage Management WG (gsm-wg) Provide dynamic space allocation and file management of shared storage components on the Grid (Storage Resource Manager – SRM) GridFTP WG (gridftp-wg) Improvements of FTP suitable for grid applications. 13
© 2007 Open Grid Forum OGF Data Area WGs II Info Dissemination WG (infod-wg) Develop a model for Information Dissemination OGSA ByteIO Working Group (byteio-wg) Define a minimal Web Service interface for providing "POSIX-like" file functionality OGSA Data Movement Interface WG (ogsa-dmi-wg) Managed data movement OGSA-Data Working Group (ogsa-d-wg) Data Architecture 14
© 2007 Open Grid Forum Activities related to file system and data movement GFS: Resource Namespace Service Specification (GFD.101) Byte-IO: Byte-IO OGSA WSRF Basic Profile Rendering (GFD.88) GSM The Storage Resource Manager Interface Specification Version 2.2 (in public comment) DMI OGSA-DMI Specification (in public comment) 15
© 2007 Open Grid Forum Data Architecture: Gaps Standardized metadata Identify query languages, data formats, transport protocols, … Needed in DAIS, DMI, ByteIO, … Data catalogs & Registries Discovery an important part of Grids Replication/Caching Data Federation 16
© 2007 Open Grid Forum 17 Standards Gaps Caching and Replication Integrated Data Management Transactions in a Grid Storage Provisioning Virtualization Provenance, Integrity, Policy File Metadata Streaming Versioning
© 2007 Open Grid Forum 18 Standards Gaps Dependencies Security: IETF, OGF Management: DMTF, SNIA WS-*: OASIS and W3C
© 2007 Open Grid Forum Main Focus for Future Work File systems NFSv4, pNFS Interface to Metadata stores Policies (not only Data) Name your favorite 19 Where can we exploit synergies with SNIA?
© 2007Open Grid Forum OGF22, 25th February 2008 OGSA Data Architecture Mario Antonioletti.
© 2006 Open Grid Forum GGF18, 13th September 2006 OGSA Data Architecture Scenarios Dave Berry & Stephen Davey.
OGF24 15 September 2008 Data Area Overview Erwin Laure David E. Martin Data Area Directors.
Data Management Kelly Clynes Caitlin Minteer. Agenda Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable.
Eldas 1.0 Enterprise Level Data Access Services Design Issues, Implementation and Future Development Davy Virdee.
Tom Sugden EPCC OGSA-DAI Future Directions OGSA-DAI User's Forum GridWorld 2006, Washington DC 14 September 2006.
NextGRID & OGSA Data Architectures: Example Scenarios Stephen Davey, NeSC, UK ISSGC06 Summer School, Ischia, Italy 12 th July 2006.
Preservation Data Services Persistent Archive Research Group Reagan W. Moore October 1, 2003.
CYBERINFRASTRUCTURE FOR THE GEOSCIENCES Data Replication Service Sandeep Chandra GEON Systems Group San Diego Supercomputer Center.
© 2008 Open Grid Forum File Catalog Development in Japan e-Science Project GFS-WG, OGF24 Singapore Hideo Matsuda Osaka University.
DATABASE SYSTEM CONCEPTS AND ARCHITECTURE CHAPTER 2 1.
Amy Krause EPCC OGSA-DAI An Overview OGSA-DAI Technology Update GGF17, Tokyo (Japan)
Enabling Grids for E-sciencE Introduction Data Management Jan Just Keijser Nikhef Grid Tutorial, November 2008.
OGSA-DAI in OMII-Europe Neil Chue Hong EPCC, University of Edinburgh.
Enterprise Java and Data Services Designing for Broadly Available Grid Data Access Services.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE middleware: gLite Data Management EGEE Tutorial 23rd APAN Meeting, Manila Jan.
NAREGI WP4 (Data Grid Environment) Hideo Matsuda Osaka University.
Amy Krause EPCC OGSA-DAI An Overview OGSA-DAI on OMII 2.0 OMII The Open Middleware Infrastructure Institute NeSC,
OPEN GRID SERVICES ARCHITECTURE AND GLOBUS TOOLKIT 4 OGSA AND GT4.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Architecture of LHC File Catalog Valeria Ardizzone INFN Catania – EGEE-II NA3/NA4.
1 School of Computer, National University of Defense Technology A Profile on the Grid Data Engine (GridDaEn) Xiao Nong
INFSO-RI Enabling Grids for E-sciencE Introduction Data Management Ron Trompert SARA Grid Tutorial, September 2007.
Current status of grids: the need for standards Mike Mineter TOE-NeSC, Edinburgh.
© 2008 Open Grid Forum Data Grid Federation by RNS GFS-WG, OGF23 Balcelona Hideo Matsuda Osaka University / NAREGI.
© 2008 Open Grid Forum Data Area Meeting OGF22 Barcelona, Spain Erwin Laure David E. Martin Data Area Directors.
Leading the pervasive adoption of grid computing for research and industry © 2006 Global Grid Forum The information contained herein is subject to change.
18 Copyright © 2005, Oracle. All rights reserved. Distributing Modular Applications: Introduction to Web Services.
Riccardo Zappi INFN-CNAF SRM Breakout session. February 28, 2012 Ingredients 1. Basic ingredients (Fabric & Conn. level) 2. (Grid) Middleware ingredients.
GT Components. Globus Toolkit A “toolkit” of services and packages for creating the basic grid computing infrastructure Higher level tools added to this.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Services Abderrahman El Kharrim
Data Grid Web Services Chip Watson Jie Chen, Ying Chen, Bryan Hess, Walt Akers.
INFSO-RI Enabling Grids for E-sciencE The gLite File Transfer Service: Middleware Lessons Learned form Service Challenges Paolo.
An Overview of OGSA-DAI Kostas Tourlas
17 July 2006ISSGC06, Ischia, Italy1 Agenda Session 26 – 14:30-16:00 An Overview of OGSA-DAI OGSA-DAI today – and future features How to extend OGSA-DAI.
Data Management Expert Panel. RLS Globus-EDG Replica Location Service u Joint Design in the form of the Giggle architecture u Reference Implementation.
Mike Jackson EPCC OGSA-DAI Architecture + Extensibility OGSA-DAI Tutorial GGF17, Tokyo.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
Faculty of Information Technology 31242/32549 Advanced Internet Programming Advanced Java Programming Web services Intro.
Objective What is RFT ? How does it work Architecture of RFT RFT and OGSA Issues Demo Questions.
© 2008 Open Grid Forum Independent Software Vendor (ISV) Remote Computing Primer Steven Newhouse.
Abstraction Layers Why do we need them? –Protection against change Where in the hourglass do we put them? –Computer Scientist perspective Expose low-level.
Open Grid Service Architecture - Data Access & Integration (OGSA-DAI) Dr Martin Westhead Principal Consultant, EPCC Telephone: Fax:+44.
Replica Management Services in the European DataGrid Project Work Package 2 European DataGrid.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Services for Distributed e-Infrastructure Access Tiziana Ferrari on behalf.
4b.1 Grid Computing Software Components of Globus 4.0 ITCS 4010 Grid Computing, 2005, UNC-Charlotte, B. Wilkinson, slides 4b.
Production Data Grids SRB - iRODS Storage Resource Broker Reagan W. Moore
0- 1 Internet Technologies ICEN2202 PART 4 (Grid Computing (OGSA, WSRF) ) Abdullah Said Alkalbani University of Buraimi.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE middleware Data Management in gLite.
FP7-INFRA Enabling Grids for E-sciencE EGEE Induction Grid training for users, Institute of Physics Belgrade, Serbia Sep. 19, 2008.
© 2007Open Grid Forum GGF19, 1'st February 2007 OGSA Data Architecture Services Dave Berry & Allen Luniewski.
© 2017 SlidePlayer.com Inc. All rights reserved.