1. Network performance issues recently raised at IN2P3-CC
All units are in bit/sec

2. LHCOPN related (1/1)
: Performance problem with US-T1-BNL or CH-CERN (software report 40M)
Paths are 10G (CERN is direct, BNL is reached through CERN)
iperf tests, worst case results achieved both ways:
FR-CCIN2P3 ↔ CH-CERN
0.95G on a 1G interface
1.83G on a 2G trunk
FR-CCIN2P3 ↔ US-T1-BNL
550M on a 1G interface
Not maximal maybe due to some other traffics interfering
Far above throughput experienced by software
No network issue
19 s, case closed for network teams
GCX

3. Generic IP (1/2) IHEP (CN, Beijing), 2010-05-14
Max bandwidth expected: 90M
While iperf achieved:
IHEP → IN2P3-CC = 1.58M
IN2P3-CC → IHEP = 60M
Seems a mix of:
Sporadic router overload or traffic-engineering experienced around Beijing’s peering (TEIN3/GÉANT)
GÉANT’s ticket stillborn (unable to reproduce…)
End hosts’ TCP stacks behaving differently during error recovery
Two different hosts, in the same IN2P3-CC’s subnet, tested at the same time can give very different results (Sun Solaris vs Linux SL4)
IHEP figured out theirs TCP parameters on hosts were mistakenly changed
Reverted back on to those from
Then FTS transfers performance back (~60M both way)
22 s, case closed
GCX

4. Generic IP (2/2) KEK (JP, Tokyo), 2010-02-16
~1G expected, but iperf reported:
IN2P3-CC → KEK = 800M
KEK → IN2P3-CC = 300M
Good results between IN2P3-CC ↔ ICEPP (University of Tokyo) = ~1G
And ICEPP and KEK have a very similar path to IN2P3-CC
Thought to be a TCP stack issue
Bad behaviour of SL3 TCP stack on high latency networks
We figured out no production traffic is exchanged between KEK and IN2P3-CC, so no problem to solve…
17 s, case closed
Future hosts at KEK will be SL4 or SL5
GCX

5. Conclusion Problem often lies on sender’s side
Hard to clearly discriminate or not networks
Sporadic and temporary issues
Iperf tests are good but proof can only be obtained on end-hosts really used
Killing generic measurement points on sites...
Coordinated efforts required on both sides
Too much time spent on administrative issues (get access, open ports etc.)
Upcoming
Clear process to better manage such regular issues at IN2P3-CC
Clear actors, roles, responsibilities and tools
Spanning across several teams: support, storage, system and network
perfSONAR box at IN2P3-CC
Regular bandwidth measurements
GCX