Bulk Data Movement: Components and Architectural Diagram Alex Sim Arie Shoshani LBNL April 2009

Multi-file request coordinator Verify storage at Target Replicate directory structure Generate plan using statistics Monitor and generate statistics Recovery and restart On-demand status Checksum comparison Dynamic progress estimation File transfer client Request submission Initial request estimation Compose request for failed files Initialization Execution Suspend and resume Architectural Diagram (the function of each module are explained on next slides)

Description of modules’ functionality Initialization Verify storage at Target This module extracts the total size from the metadata at the source gateway, and verifies that the target node has sufficient space allocated. Generate plan using statistics This module uses previous statistics to plan the concurrency level to use in moving files (i.e. how many files can be transferred at the same time). Initially, this will be based on bandwidth estimated provided as parameters. Initial request estimation Using total size and concurrency level, an estimate of the total time required for the transfer will be provided. The request can be aborted at this time if the total transfer time is unacceptable. Replicate directory structure Once a “go ahead” is given, this module generates a directory structure that mimics the source directory structure. This is done by finding the structure of the directories and files at the source from the metadata at the source gateway, and creating the exact structure at the target. This module also generates the full list of source-to-target URL pairs for the execution of the transfer. At this time a “request token” is provided to the client calling BDM, and transfer continues as a background task. The token can be used later for tracking the progress of the transfer (see following pages).

Description of modules’ functionality Execution Multi-file request coordinator This module coordinates the entire transfer task. It initiate multiple threads according to the concurrency level. Each thread initiates calls the file transfer client, monitors for successful transfers (checking file sizes), and recovers from transient transfer failures (requesting transfer resumption from the point of failure). For each thread that completes successfully it restarts another thread, keeping the concurrency level at the maximum rate. Note that checksum will be checked later. File transfer client The file transfer client can use any desired transfer protocol. However, initially only GridFTP will be used because all sites agreed to use that. Only a client is used at the mirroring node, and it gets the files by “pulling” them from the source nodes. It is assume that the source nodes will have a GridFTP server available. Note that if the source node is down, retries will be attempted for a pre-specified retry time till it recovers. If the retry time expires, the BDM will suspend its operation. Recovery and restart This module is responsible to recover from failures of the mirroring (target) node. For this purpose, the state of transfers will be recorded on non-volatile storage (disk). The strategy is delete any partially transferred file, and re-transfer this files from the source. Monitor and generate statistics This module will record performance statistics on a per-file-basis, and generate global summaries. Initially, this function will only log begin-end times and file sizes. It will also generate global summary statistics dynamically (i.e transfer rate so far). These statistics will be available for “dynamic progress estimation” module (see next page) Checksum comparison This module will retrieve the checksums from the source metadata, generate checksums to all files that were transferred, and compare. For the files that fail the checksum test, it will generate a (hopefully) small request to re-transfer, and start a new transfer task. The checksums are checked at the end rather than at transfer time, since this is viewed as potentially slowing down the transfers. It was one of the requirements that all checksums are checked at the end of the transfer of all files.

Description of modules’ functionality Interactive functions On-demand status Using the request token provided at initialization, the client can issue a status command to BDM. Summary statistics will be provided, including total files transferred (out of how many), total size transferred so far, average transfer rate. Dynamic progress estimation This information can be provided at the same time as status request. Initially this module will use only the dynamic average transfer rate generated by the “monitor and record statistics” module. At a later time, it may analyze transfer patterns for more accurate estimation. Suspend and resume These two functions will allow the client to suspend the transfer operation for any reason (such as planned power outage, maintenance, etc.), and resume operation at a later time. The suspend module will ensure a consistent state before suspending. It may give an option of continuing for a while longer to complete current transfers. If permission is not granted, it will remove all partially transferred files, and quit.