OGSA Data Architecture WG Data Transfer Discussion Allen Luniewski, IBM Dave Berry, NESC GGF15 in Boston October 3-6, 2005

GGF Intellectual Property Policy All statements related to the activities of the GGF and addressed to the GGF are subject to all provisions of Appendix B of GFD-C.1, which grants to the GGF and its participants certain licenses and rights in such statements. Such statements include verbal statements in GGF meetings, as well as written and electronic communications made at any time or place, which are addressed to any GGF working group or portion thereof, Where the GFSG knows of rights, or claimed rights, the GGF secretariat shall attempt to obtain from the claimant of such rights, a written assurance that upon approval by the GFSG of the relevant GGF document(s), any party will be able to obtain the right to implement, use and distribute the technology or works when implementing, using or distributing technology based upon the specific specification(s) under openly specified, reasonable, non-discriminatory terms. The working group or research group proposing the use of the technology with respect to which the proprietary rights are claimed may assist the GGF secretariat in this effort. The results of this procedure shall not affect advancement of document, except that the GFSG may defer approval where a delay may facilitate the obtaining of such assurances. The results will, however, be recorded by the GGF Secretariat, and made available. The GFSG may also direct that a summary of the results be included in any GFD published containing the specification.

Brief report of Data Transfer BOF Discuss fit into wider architecture Session Agenda Brief report of Data Transfer BOF Discuss fit into wider architecture “Straw man” design Discussion, work plans

Data Transfer BOF Report Point to point transfer Bytes only Assuming large data Quick win: emphasis on files Built-in support for multiple files Extensions considered, e.g. to data in memory Common API for Globus, SRM, CERN implementations Protocol independent But provide ability to tweak the settings Some QoS configuration (optional) Maximum bandwidth Delivery guarantees Time when data required Cleanup / failure policies

Architecture requirements Generic, extensible, reusable Base Layer: point to point, bytes Further layers: Preserve semantics Many sinks (e.g. “at least m of n”) Integration with replication service Higher still: Automatic negotiation of formats, protocols, etc.

Known case: file transfer File hierarchy is semantic information Filenames (etc.) are metadata So dealing with file hierarchy should be above the base layer This is not efficient session management scheduling of file access on source Therefore some “higher layers” must be configurations of / extensions of the base Rather than calls to a lower level layer

Straw man (1) Source URL (e.g. a file folder) Sink URL Client Session manager Source URL (e.g. a file folder) Sink URL Type of data (e.g. files)

Source and sink must understand the session protocol Straw man (2) Client Prepare for transfer Session manager Prepare for transfer Source (e.g. SRM) Sink (e.g. File System) Source and sink must understand the session protocol

Straw man (3) Send (e.g. number of streams) Client Session manager Source (e.g. SRM) Sink (e.g. File System) Schedules blocks to send (e.g. files)

Ignoring details of control protocol Straw man (4) Client Blocks of data (e.g. files) Session manager + Metadata (e.g. file names, modtimes, etc. Source (e.g. SRM) Sink (e.g. File System) Ignoring details of control protocol

Straw man (5) Stores blocks (e.g. creates file hierarchy) Client Session manager Stores blocks (e.g. creates file hierarchy) Source (e.g. SRM) Sink (e.g. File System) Cleans up if transfer fails

Can we extend it to other cases? Questions Does this preserve capabilities and performance of existing file transfer tools? Can we extend it to other cases? Memory to memory Streaming visualisation data Database queries / database replication Resolving replicas on source / sink Data transformation Multiple sinks

Security Transactions Negotiation Issues Authentication, authorisation, delegation, encryption, signing Some uses may need to require encryption / double encryption May need to associate policy with data being transferred Transactions Transaction specs claim to be composable Negotiation This can largely be left to higher-level services,