1. OGSA Data Architecture WG Data Transfer Discussion
Allen Luniewski, IBM
Dave Berry, NESC
GGF15 in Boston
October 3-6, 2005

2. GGF Intellectual Property Policy
All statements related to the activities of the GGF and addressed to the GGF are subject to all provisions of Appendix B of GFD-C.1, which grants to the GGF and its participants certain licenses and rights in such statements. Such statements include verbal statements in GGF meetings, as well as written and electronic communications made at any time or place, which are addressed to any GGF working group or portion thereof,
Where the GFSG knows of rights, or claimed rights, the GGF secretariat shall attempt to obtain from the claimant of such rights, a written assurance that upon approval by the GFSG of the relevant GGF document(s), any party will be able to obtain the right to implement, use and distribute the technology or works when implementing, using or distributing technology based upon the specific specification(s) under openly specified, reasonable, non-discriminatory terms. The working group or research group proposing the use of the technology with respect to which the proprietary rights are claimed may assist the GGF secretariat in this effort. The results of this procedure shall not affect advancement of document, except that the GFSG may defer approval where a delay may facilitate the obtaining of such assurances. The results will, however, be recorded by the GGF Secretariat, and made available. The GFSG may also direct that a summary of the results be included in any GFD published containing the specification.

3. Brief report of Data Transfer BOF Discuss fit into wider architecture
Session Agenda
Brief report of Data Transfer BOF
Discuss fit into wider architecture
“Straw man” design
Discussion, work plans

4. Data Transfer BOF Report
Point to point transfer
Bytes only
Assuming large data
Quick win: emphasis on files
Built-in support for multiple files
Extensions considered, e.g. to data in memory
Common API for Globus, SRM, CERN implementations
Protocol independent
But provide ability to tweak the settings
Some QoS configuration (optional)
Maximum bandwidth
Delivery guarantees
Time when data required
Cleanup / failure policies

5. Architecture requirements
Generic, extensible, reusable
Base Layer: point to point, bytes
Further layers:
Preserve semantics
Many sinks (e.g. “at least m of n”)
Integration with replication service
Higher still:
Automatic negotiation of formats, protocols, etc.

6. Known case: file transfer
File hierarchy is semantic information
Filenames (etc.) are metadata
So dealing with file hierarchy should be above the base layer
This is not efficient
session management
scheduling of file access on source
Therefore some “higher layers” must be configurations of / extensions of the base
Rather than calls to a lower level layer

7. Straw man (1) Source URL (e.g. a file folder) Sink URL
Client
Session manager
Source URL (e.g. a file folder)
Sink URL
Type of data (e.g. files)

8. Source and sink must understand the session protocol
Straw man (2)
Client
Prepare for transfer
Session manager
Prepare for transfer
Source
(e.g. SRM)
Sink (e.g. File System)
Source and sink must understand the session protocol

9. Straw man (3) Send (e.g. number of streams)
Client
Session manager
Source
(e.g. SRM)
Sink (e.g. File System)
Schedules blocks to send (e.g. files)

10. Ignoring details of control protocol
Straw man (4)
Client
Blocks of data (e.g. files)
Session manager
+ Metadata (e.g. file names, modtimes, etc.
Source
(e.g. SRM)
Sink (e.g. File System)
Ignoring details of control protocol

11. Straw man (5) Stores blocks (e.g. creates file hierarchy)
Client
Session manager
Stores blocks (e.g. creates file hierarchy)
Source
(e.g. SRM)
Sink (e.g. File System)
Cleans up if transfer fails

12. Can we extend it to other cases?
Questions
Does this preserve capabilities and performance of existing file transfer tools?
Can we extend it to other cases?
Memory to memory
Streaming visualisation data
Database queries / database replication
Resolving replicas on source / sink
Data transformation
Multiple sinks

13. Security Transactions Negotiation Issues
Authentication, authorisation, delegation, encryption, signing
Some uses may need to require encryption / double encryption
May need to associate policy with data being transferred
Transactions
Transaction specs claim to be composable
Negotiation
This can largely be left to higher-level services,