1. FTS Issue in Beijing
Erming PEI
2010/06/18

2. Failures
IN2P3BEIJING experienced many transfer failures since the middle of May.
Almost all large files failed, only small files passed.
Only ~500KB/s per file in average.
FTS Channel configuration:
10files and 10streams
shared by ATLAS and CMS
~1Gb/s bandwidth in total (iperf result: ~90MB)
ATLAS failures is proportional to CMS transfers
CMS had aggressive transfers from all over the world with multiple channels and 2 routes, while ATLAS transfers only from CC-IN2P3 to Beijing in single channel.

3. Network Routes Europe America CSTNET IHEP 2.5Gbps 622Mbps 1Gbps CMS
ATLAS
IHEP

4. Reason/Solution There’s a network bottleneck in CSTNET!!
No matter how we tuned all the parameters, the peak of the total bandwidth can only reach 550Mbps. Even when we tried to get CMS activities stopped!
After negotiation CNIC, the total bandwidth was recovered to ~1Gbps.
Both ATLAS/CMS file transfers had better performance later. All transfers passed.

5. Total Throughput

6. Intermediation ATLAS and CMS still compete the bandwidth
CMS transfers files from all the T1s and many T2s to Beijing, which ate most of the bandwidth.
In order to intermediate the two experiments, we had such
Tuned FTS channel to 40 concurrent files and 16 streams
Tuned TCP maximum window to 32MB
Set the concurrent number of files and VO share specifically for CMS to limit its activities.
CMS channel limit on IN2P3-BEIJING and STAR-BEIJING: 5
Reduce CMS VO share on IN2P3-BEIJING to 20%

7. Now, the files transfer rate is very good.
Big files all got successfully transferred.

8. BEIJINGIN2P3 Since June 4th, BEJINGIN2P3 had very low performance.
It turned out that the fault of bad TCP window default size: only ~80KB

9. Individual Throughput <50KB/s

10. TCP Window parameters:
After tuned the default TCP window size to 4MB, the situation got improved.
TCP Window parameters:
net.ipv4.tcp_rmem =
net.ipv4.tcp_wmem =

11. Conclusion Found a network bottleneck and finally got it eliminated
Tuning of TCP window size has good effect on improving the performance.
Intermediation between ATLAS and CMS is necessary.
Thanks to Eric for keeping eyes on this issue…