1. Jan 12, 2005
Improving CMS data transfers among its distributed Computing Facilities
N. Magini
CERN IT-ES-VOS, Geneva, Switzerland
J. Flix
Port d'Informació Científica (PIC), Barcelona, Spain
A. Sartirana
École Polytechnique, Palaiseau, France
On behalf of the CMS experiment
CHEP International Conference on Computing in High Energy and Nuclear Physics
19 October 2010, Academia Sinica, Taipei, Taiwan

2. Outline CMS data transfer workflows
Jan 12, 2005
Outline
CMS data transfer workflows
Measuring the performance of CMS data transfers
Using data transfer statistics to improve transfer operations

3. CMS Computing Model 7 Tier-1s Tier-0 ~50 Tier-2s CAF TIER-0 CERN
WLCG Computing
Grid Infrastructure
CMS
detector
TIER-0
CERN
900 MB/s agg.
TIER-1
TIER-1
50-500MB/s
10-20MB/s
TIER-2
TIER-2
TIER-2
7 Tier-1s
(“online” to the DAQ)
High availability centres
Custodial mass storage of share of data
Data reconstruction and reprocessing
Data skimming & selection
Distribute analysis data  Tier-2s
Tier-0
(the accelerator centre)
Data acquisition & initial processing
Long-term mass data storage
CMS CERN Analysis Facility
(latency critical data processing, high priority analysis)
Distribution of data  Tier-1 centres
~50 Tier-2s
in ~20 countries
End-user physics analyses
Detector Studies
Monte Carlo Simulation  Tier-1

4. CMS Transfer Workflow CMS transfer management system PhEDEx
WLCG middleware
FTS
SRM
gridFTP

5. CMS Transfer Workflow
Transfer request is placed through PhEDEx web interface

6. CMS Transfer Workflow
PhEDEx central agents create and distribute transfer tasks to site agents

7. CMS Transfer Workflow
PhEDEx Download agent submits transfer batch job to FTS server

8. CMS Transfer Workflow
FTS contacts source and destination SRMs to get transfer URLs

9. CMS Transfer Workflow
FTS executes the transfer as third-party with gridFTP
FTP extension
GSI security
parallel streams
Other configurations also used:
srmCopy started by FTS server
srmCopy started directly by PhEDEx Download agent with SRM client

10. Jan 12, 2005
File Transfer Service
Provides scheduling of multiple asynchronous file transfers on CHANNELS
Single direction transfer queue between two endpoints
Not tied to a physical network path
Each endpoint (source and destination) can be:
A single site, e.g.
CERN-RAL
IN2P3-BELGIUMULB
A group of sites (“cloud”), e.g.
RALLCG2-CLOUDCMSITALY
CLOUDCMSFRANCE-RALLCG2
All sites (“star”), e.g.
CNAF-STAR
STAR-FNAL
For FTS, a channel is a transfers management queue; once a job is submitted, according to
the source and destination endpoints
the VO of the user
the job is assigned the most suitable channel according to the channel topology configured on the server. The concept of channel is not related to a physical network path.
All file transfer on the same channel are served as part of the same queue; on this queue it is possible to set intra-VO shares (Atlas gets 75%, CMS gets the rest) or priorities within a VO
Each channel can be configured to use a particular transfer method (gridftp or srmcopy) and has its own parameters (number of concurrent files running, number streams, TCP buffer, etc). It is also possible to set limits on concurrent transfers on the same storage element at a given time etc.
Introduction to FTS

11. FTS server deployment At Tier-0 At each Tier-1 TIER-0 CERN
Jan 12, 2005
FTS server deployment
TIER-0
CERN
At Tier-0
Dedicated channel to each of the Tier-1s
At each Tier-1
Dedicated channel from each of the other Tier-1s
Dedicated channels to and from each of the associated Tier-2s
CLOUD and/or STAR channels to/from other Tier-2s
STAR-T2 channels for each associated Tier-2
TIER-1
TIER-1
TIER-2
TIER-2
TIER-2

12. FTS channels FTS channel configuration defines:
Jan 12, 2005
FTS channels
FTS channel configuration defines:
Transfer limits
Maximum number of concurrent active transfers
Protect network, storage
Shared among VOs according to policy
Transfer priorities
Between users in the same VO on a channel
Transfer parameters
Number of parallel TCP streams, buffer size
Timeouts
Overall throughput for a link in a channel
Link throughput = rate/stream * streams/file * active transfers/link
In a dedicated channel
Expect ~constant rate/stream up to saturation
Fixed number of available active transfer slots/link
In cloud or star channel
rate/stream can be significantly different for links in same channel
Available active transfer slots/link depends on overall channel occupancy
Slow links keep transfer slots busy for longer
For FTS, a channel is a transfers management queue; once a job is submitted, according to
the source and destination endpoints
the VO of the user
the job is assigned the most suitable channel according to the channel topology configured on the server. The concept of channel is not related to a physical network path.
All file transfer on the same channel are served as part of the same queue; on this queue it is possible to set intra-VO shares (Atlas gets 75%, CMS gets the rest) or priorities within a VO
Each channel can be configured to use a particular transfer method (gridftp or srmcopy) and has its own parameters (number of concurrent files running, number streams, TCP buffer, etc). It is also possible to set limits on concurrent transfers on the same storage element at a given time etc.
Introduction to FTS

13. Evolution of transfer workflows
Jan 12, 2005
Evolution of transfer workflows
Scale and complexity of CMS data transfers has been steadily increasing thanks to focused effort on improving transfer quality and throughput
~300 links
T0  T1
T1  T1
T1  “associated” T2s

14. Evolution of transfer workflows
Jan 12, 2005
Evolution of transfer workflows
Scale and complexity of CMS data transfers has been steadily increasing thanks to focused effort on improving transfer quality and throughput
~3000 links
T0  T1
T1  T1
T1  all T2s
T2  all T2s
As more and more data transfer links are commissioned, the sites start competing for the same slots in the FTS channels
Making optimal use of bandwidth requires identifying and isolating the problematic links

15. FTS Monitor FTS server database contains detailed transfer information
Jan 12, 2005
FTS Monitor
FTS server database contains detailed transfer information
Wealth of knowledge that can be used to spot issues
Information is exposed through FTS Monitor
Transfer summary

16. FTS Monitor FTS server database contains detailed transfer information
Jan 12, 2005
FTS Monitor
FTS server database contains detailed transfer information
Wealth of knowledge that can be used to spot issues
Information is exposed through FTS Monitor
Channel configuration details

17. FTS Monitor FTS server database contains detailed transfer information
Jan 12, 2005
FTS Monitor
FTS server database contains detailed transfer information
Wealth of knowledge that can be used to spot issues
Information is exposed through FTS Monitor
Individual transfer details

18. FTS monitor parser Tool to extract data from FTS monitors worldwide
Jan 12, 2005
FTS monitor parser
Tool to extract data from FTS monitors worldwide
Full statistics about transfers are extracted daily, and summary reports are produced
Several views available
Global: e.g. transfer rate per file and per stream on all T1-T1 channels

19. FTS monitor parser Tool to extract data from FTS monitors worldwide
Jan 12, 2005
FTS monitor parser
Tool to extract data from FTS monitors worldwide
Full statistics about transfers are extracted daily, and summary reports are produced
Several views available
Site: e.g. rate per stream on all CNAF-T2 and T2-CNAF channels

20. FTS monitor parser Tool to extract data from FTS monitors worldwide
Jan 12, 2005
FTS monitor parser
Tool to extract data from FTS monitors worldwide
Full statistics about transfers are extracted daily, and summary reports are produced
Several views available
Historical: e.g. evolution of rate per stream on IN2P3-PIC channel

21. FTS channel optimization
Jan 12, 2005
FTS channel optimization
Using data extracted from FTS Monitoring to improve transfer operations
Example: PICT2 exports
Massive PICT2s transfers in early October following processing campaign
Clogged by slow links on PIC-STAR FTS channel
Links with low rate-per-stream identified

22. FTS channel optimization
Jan 12, 2005
FTS channel optimization
Using data extracted from FTS Monitoring to improve transfer operations
Example: PICT2 exports
Created “cloud” FTS channels for “fast” and “slow” links
Result:
Improved FTS channel occupancy
Increased number of transfer attempts
Improved overall export throughput

23. Identifying infrastructure issues
Jan 12, 2005
Identifying infrastructure issues
The wealth of data available allows to spot potential issues in the site or network infrastructure
Example: PIC import/export asymmetry
Rate-per-stream lower for exports than for imports on most links
Doesn’t seem to depend on distance
Potential site issue?
Possible explanation: known limitation of kernel used on disk servers
Dedicated testing will reveal more

24. Jan 12, 2005
Expanding the scope
First results presented – lots of potential to expand the analysis
Gather data for FTS servers not yet included
Identify “reference” statistics and publish corresponding plots to monitor regularly
For central shifters
For site administrators
Spot problems in sites and network and assist site administrators with troubleshooting
Include more statistics
Distributions by file size
Transfer preparation times
Channel occupancy …
Integrating with other VOs

25. Summary PhEDEx ensures reliable data transfers with FTS
The scale and complexity of CMS transfers has constantly increased over the years
The FTS Monitor offers detailed information on transfers
Extracting and analyzing transfer statistics provides useful insight to improve transfer operations