Jan 12, 2005 Improving CMS data transfers among its distributed Computing Facilities N. Magini CERN IT-ES-VOS, Geneva, Switzerland J. Flix Port d'Informació Científica (PIC), Barcelona, Spain A. Sartirana École Polytechnique, Palaiseau, France On behalf of the CMS experiment CHEP 2010 - International Conference on Computing in High Energy and Nuclear Physics 19 October 2010, Academia Sinica, Taipei, Taiwan

Outline CMS data transfer workflows Jan 12, 2005 Outline CMS data transfer workflows Measuring the performance of CMS data transfers Using data transfer statistics to improve transfer operations

CMS Computing Model 7 Tier-1s Tier-0 ~50 Tier-2s CAF TIER-0 CERN WLCG Computing Grid Infrastructure CMS detector TIER-0 CERN 900 MB/s agg. TIER-1 TIER-1 50-500MB/s 10-20MB/s TIER-2 TIER-2 TIER-2 7 Tier-1s (“online” to the DAQ) High availability centres Custodial mass storage of share of data Data reconstruction and reprocessing Data skimming & selection Distribute analysis data  Tier-2s Tier-0 (the accelerator centre) Data acquisition & initial processing Long-term mass data storage CMS CERN Analysis Facility (latency critical data processing, high priority analysis) Distribution of data  Tier-1 centres ~50 Tier-2s in ~20 countries End-user physics analyses Detector Studies Monte Carlo Simulation  Tier-1

CMS Transfer Workflow CMS transfer management system PhEDEx WLCG middleware FTS SRM gridFTP

CMS Transfer Workflow Transfer request is placed through PhEDEx web interface

CMS Transfer Workflow PhEDEx central agents create and distribute transfer tasks to site agents

CMS Transfer Workflow PhEDEx Download agent submits transfer batch job to FTS server

CMS Transfer Workflow FTS contacts source and destination SRMs to get transfer URLs

CMS Transfer Workflow FTS executes the transfer as third-party with gridFTP FTP extension GSI security parallel streams Other configurations also used: srmCopy started by FTS server srmCopy started directly by PhEDEx Download agent with SRM client

Jan 12, 2005 File Transfer Service Provides scheduling of multiple asynchronous file transfers on CHANNELS Single direction transfer queue between two endpoints Not tied to a physical network path Each endpoint (source and destination) can be: A single site, e.g. CERN-RAL IN2P3-BELGIUMULB A group of sites (“cloud”), e.g. RALLCG2-CLOUDCMSITALY CLOUDCMSFRANCE-RALLCG2 All sites (“star”), e.g. CNAF-STAR STAR-FNAL For FTS, a channel is a transfers management queue; once a job is submitted, according to the source and destination endpoints the VO of the user the job is assigned the most suitable channel according to the channel topology configured on the server. The concept of channel is not related to a physical network path. All file transfer on the same channel are served as part of the same queue; on this queue it is possible to set intra-VO shares (Atlas gets 75%, CMS gets the rest) or priorities within a VO Each channel can be configured to use a particular transfer method (gridftp or srmcopy) and has its own parameters (number of concurrent files running, number streams, TCP buffer, etc). It is also possible to set limits on concurrent transfers on the same storage element at a given time etc. Introduction to FTS

FTS server deployment At Tier-0 At each Tier-1 TIER-0 CERN Jan 12, 2005 FTS server deployment TIER-0 CERN At Tier-0 Dedicated channel to each of the Tier-1s At each Tier-1 Dedicated channel from each of the other Tier-1s Dedicated channels to and from each of the associated Tier-2s CLOUD and/or STAR channels to/from other Tier-2s STAR-T2 channels for each associated Tier-2 TIER-1 TIER-1 TIER-2 TIER-2 TIER-2

FTS channels FTS channel configuration defines: Jan 12, 2005 FTS channels FTS channel configuration defines: Transfer limits Maximum number of concurrent active transfers Protect network, storage Shared among VOs according to policy Transfer priorities Between users in the same VO on a channel Transfer parameters Number of parallel TCP streams, buffer size Timeouts Overall throughput for a link in a channel Link throughput = rate/stream * streams/file * active transfers/link In a dedicated channel Expect ~constant rate/stream up to saturation Fixed number of available active transfer slots/link In cloud or star channel rate/stream can be significantly different for links in same channel Available active transfer slots/link depends on overall channel occupancy Slow links keep transfer slots busy for longer For FTS, a channel is a transfers management queue; once a job is submitted, according to the source and destination endpoints the VO of the user the job is assigned the most suitable channel according to the channel topology configured on the server. The concept of channel is not related to a physical network path. All file transfer on the same channel are served as part of the same queue; on this queue it is possible to set intra-VO shares (Atlas gets 75%, CMS gets the rest) or priorities within a VO Each channel can be configured to use a particular transfer method (gridftp or srmcopy) and has its own parameters (number of concurrent files running, number streams, TCP buffer, etc). It is also possible to set limits on concurrent transfers on the same storage element at a given time etc. Introduction to FTS

Evolution of transfer workflows Jan 12, 2005 Evolution of transfer workflows Scale and complexity of CMS data transfers has been steadily increasing thanks to focused effort on improving transfer quality and throughput 2007 ~300 links T0  T1 T1  T1 T1  “associated” T2s

Evolution of transfer workflows Jan 12, 2005 Evolution of transfer workflows Scale and complexity of CMS data transfers has been steadily increasing thanks to focused effort on improving transfer quality and throughput 2010 ~3000 links T0  T1 T1  T1 T1  all T2s T2  all T2s As more and more data transfer links are commissioned, the sites start competing for the same slots in the FTS channels Making optimal use of bandwidth requires identifying and isolating the problematic links

FTS Monitor FTS server database contains detailed transfer information Jan 12, 2005 FTS Monitor FTS server database contains detailed transfer information Wealth of knowledge that can be used to spot issues Information is exposed through FTS Monitor Transfer summary

FTS Monitor FTS server database contains detailed transfer information Jan 12, 2005 FTS Monitor FTS server database contains detailed transfer information Wealth of knowledge that can be used to spot issues Information is exposed through FTS Monitor Channel configuration details

FTS Monitor FTS server database contains detailed transfer information Jan 12, 2005 FTS Monitor FTS server database contains detailed transfer information Wealth of knowledge that can be used to spot issues Information is exposed through FTS Monitor Individual transfer details

FTS monitor parser Tool to extract data from FTS monitors worldwide Jan 12, 2005 FTS monitor parser Tool to extract data from FTS monitors worldwide Full statistics about transfers are extracted daily, and summary reports are produced Several views available Global: e.g. transfer rate per file and per stream on all T1-T1 channels

FTS monitor parser Tool to extract data from FTS monitors worldwide Jan 12, 2005 FTS monitor parser Tool to extract data from FTS monitors worldwide Full statistics about transfers are extracted daily, and summary reports are produced Several views available Site: e.g. rate per stream on all CNAF-T2 and T2-CNAF channels

FTS monitor parser Tool to extract data from FTS monitors worldwide Jan 12, 2005 FTS monitor parser Tool to extract data from FTS monitors worldwide Full statistics about transfers are extracted daily, and summary reports are produced Several views available Historical: e.g. evolution of rate per stream on IN2P3-PIC channel

FTS channel optimization Jan 12, 2005 FTS channel optimization Using data extracted from FTS Monitoring to improve transfer operations Example: PICT2 exports Massive PICT2s transfers in early October following processing campaign Clogged by slow links on PIC-STAR FTS channel Links with low rate-per-stream identified

FTS channel optimization Jan 12, 2005 FTS channel optimization Using data extracted from FTS Monitoring to improve transfer operations Example: PICT2 exports Created “cloud” FTS channels for “fast” and “slow” links Result: Improved FTS channel occupancy Increased number of transfer attempts Improved overall export throughput

Identifying infrastructure issues Jan 12, 2005 Identifying infrastructure issues The wealth of data available allows to spot potential issues in the site or network infrastructure Example: PIC import/export asymmetry Rate-per-stream lower for exports than for imports on most links Doesn’t seem to depend on distance Potential site issue? Possible explanation: known limitation of kernel used on disk servers Dedicated testing will reveal more

Jan 12, 2005 Expanding the scope First results presented – lots of potential to expand the analysis Gather data for FTS servers not yet included Identify “reference” statistics and publish corresponding plots to monitor regularly For central shifters For site administrators Spot problems in sites and network and assist site administrators with troubleshooting Include more statistics Distributions by file size Transfer preparation times Channel occupancy … Integrating with other VOs

Summary PhEDEx ensures reliable data transfers with FTS The scale and complexity of CMS transfers has constantly increased over the years The FTS Monitor offers detailed information on transfers Extracting and analyzing transfer statistics provides useful insight to improve transfer operations