1. Data Management Kelly Clynes Caitlin Minteer

2. Agenda Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable File Transfer Data Replication Replication Location Service Logical File Data Replication Service OGSA

3. Globus Toolkit Fundamental enabling technology for the "Grid" Does not provide interactive client Globus Toolkit provides: –Server implementation –Scriptable command line client –Set of development libraries

4. Basic Data Management Systems GridFTP - A uniform, secure, high- performance interface to file-based storage systems on the GridGridFTP OGSA-DAI - An OGSA interface for accessing XML and relational data storesOGSA-DAI

5. Overview of Data Management Two Basic Categories of Data Management –Data Movement –Data Replication

6. Data Movement Globus Reliable File Transfer (RFT) Service Globus GridFTP Tools

7. GridFTP What is GridFTP? –Global Grid Forum –GFD.020, RFC 959, RFC 2228, RFC 2389, etc. –based upon the Internet FTP protocol Why use GridFTP? –Secure –Robust –Fast –Efficient

8. GridFTP What does GridFTP Provide? –multiple data channels for parallel transfers –partial file transfers –third-party (direct server-to-server) transfers –reusable data channels –command pipelining.

9. GridFTP Wish to make data available to others –Install server on a host that can access the data –Make sure that there is an appropriate Data Storage Interface (DSI) –To access data that other have made available, you need a GridFTP To add access to files stored behind the GridFTP servers, you need a custom client

10. GridFTP in GT 4.0 Globus Code Backwards Compatible Stripping support IPV6 support included Modular

11. Reliable File Transfer Third-party multi-file transfers Exponential back-off on failure Optional use of parallel streams and TCP buffer size tuning Recursive directory transfer

12. Reliable File Transfer Service Not a web service protocol Requires that the client maintain an open socket connection to the server throughout the transfer Service interfaced based on web services protocols that persist the transfer state in reliable storage

13. Reliable File Transfer Web Service Resource Framework “job – scheduler” You simply: –Provide a list of source and destination URLS The service writes your job destination in DB and the moved the files on your behalf –Service methods are provided for querying the transfer status

14. RFT in Action Registry RFT Factory Grid Service Container

15. Client RFT in Action Registry RFT Factory Grid Service Container

16. Client RFT in Action RFT Factory Grid Service Container

17. Client RFT in Action RFT Factory Grid Service Container RFT Service Instance - Start the Instance - Deserialize XML to Java - Write Request via JDBC - Persist Service State

18. Client RFT in Action RFT Factory Grid Service Container RFT Service Instance - Start the Instance - Deserialize XML to Java - Write Request via JDBC - Persist Service State

19. RFT in Action Service is OGSI compliant Uses existing GridFTP protocols and tools to execute 3 rd Party Transfer for the user Provides extensive state transition notification GridFTP Server RFT Service Instance

20. Data Replication Replica Location Service (RLS) –Provides the ability to keep track of one or more copies or replicas –Helpful for users ore applications that need to find where existing files are located on grid –Services register files in RLS when files are created

21. Replica Location Service Provides a framework for tracking the physical locations of data that has been replicated. RLS maps logical names to physical names. Replication of data can: –reduce access latency – improve data locality –increase robustness, scalability and performance for distributed applications. An RLS typically does not operate in isolation, but functions as one component of a data grid architecture.

22. Replica Location Service RLS may consist of multiple servers at different sites Able to increase over all scale of system and store more mappings then if just on single, centralized catalog

23. Replica Location Service Logical File – unique identifier for the contents of a file Physical File – location of a copy of the file on a storage system User can provide a logical file name to an RLS User can query an RLS server to find name associated with a particular physical file location

24. Logical File Associations between a logical file name and three replicas on different storage sites

25. RLS Example Detect the existence of gravitational waves –Produces millions of data files –Eight other sites need to access files Ten physical locations RLS servers at each site Laser Interferometer Gravitational Wave Observatory

26. RLS Example Requests the file from LIGO’s Data Management System –Lightweight Data Replicator (LDR) LDR queries Replica Location Service for local copy –If no local copy is found, returns where it is in the Grid –Request to copy file to local storage –Registers new copy to local RLS

27. Higher Level Data Services Combines two existing data management components –RFT –RLS

28. Data Replication Service (DRS) Provides a pull – based replication capability Built on top of two GT data management components –Reliable File Transfer –Replica Location Service

29. Data Replication Service Function –To ensure that a specified set of files exist on a storage site Begins by querying RLS to discover where the desired files exist After files are located, creates a transfer request that is executed by RFT DRS then registers the new replicas with RLS

30. Data Replication Service Implemented as a web service and complies with the Web Service Resource Framework When request is received –Creates a WS – resource Used to maintain state about each file being replicated Including which operations on the file have failed

31. OGSA Open Grid Services Architecture (OGSA) Product of the Grid community Service oriented Provides a pure Java data service framework for accessing and integrating data resources

32. OGSA Defines a set of core capabilities and behaviors: –uniform exposed service semantics (the Grid service); –defines standard devices for creating, naming, and discovering temporary Grid service instances; –provides location transparency and multiple protocol bindings for service instances; –supports integration with underlying native platform facilities.

33. OGSA and GT4 Allows multiple data resources to be accessed through a single service. The listResources() operation The data service resource identifiers can then be used by a client to obtain metadata and other information Access to data service resource metadata is provided by an implementation of the WS-ResourceProperties specification. getVersion() operation. A WSRF version of the OGSA-DAI GridDataTransport portType supports asynchronous data delivery between data services.

34. Summary GridFTP RLS RFT DRS OGSA

35. Resources http://www.globus.org/alliance/publications /papers/ogsa.pdfhttp://www.globus.org/alliance/publications /papers/ogsa.pdf http://www.globus.org/toolkit/data/ http://www.globus.org/toolkit/data/gridftp/ http://www.globus.org/toolkit/data/rft/ http://www.globus.org/toolkit/data/rls/