1. The transfer performance of iRODS between CC-IN2P3 and KEK
Yoshimi Iida
Workshop KEK-CC-IN2P3
27-29 November 2007

2. iRODS
iRODS, which stands for i Rule Oriented Data System, is the next generation data management cyberinfrastructure
Rules
Control the operations that are being performed when a rule is invoked by a particular task
The core set are defined in the "core.irb" file
Micro-services
Small, well-defined procedures/functions that perform a certain task
C-functions are called when executing the rule body

3. SRB test system 2 type of SRB system Internet KEK FW SINET KEK LAN SRB
Grid LAN
SRB
test system
KEK SRB system

4. Install at test system At KEK internal network Internet KEK FW
CC-IN2P3
KEK LAN
iRODS client
iRODS
server
iRODS
server
iCAT

5. Transfer parameter
Set the parameter for a data transfer at acSetNumThreads rule in core.irb file on the server at Lyon
acSetNumThreads||msiSetNumThreads(sizePerThrInMb, maxNumThr, windowSize)|nop
sizePerThrInMb is used for computing the number of threads
numThreads=fileSizeInMb/sizePerThrInMb+1
maxNumThr is the maximum number of threads to use (up to 16)
windowSize is the TCP window size in Bytes

6. iput with several window size
The maximum socket buffer size of server at CC-IN2P3 is 4MB

7. iput test
iput 1GB data file from KEK-internal to Lyon with 16 threads during 2days

8. Install at KEK The server at Grid LAN Internet KEK FW SINET KEK LAN
CC-IN2P3
Grid LAN
iRODS
server
iRODS
server
iRODS server
iCAT

9. Specification iRODS server at CC-IN2P3 iRODS server at KEK
OS: Solaris 10
CPU: AMD Opteron 2.6GHz x 4
Memory: 16GB
iRODS server at KEK
OS: RedHat AS 3
CPU: Intel Xeon 3.0GHz x 4
Memory: 4GB

10. iput from KEK to Lyon 1GB data file transfer during 1 days
window size: 4MB, number of threads: 16
Sput from KEK to Lyon was about 2.3MB/sec

11. iput from Lyon to KEK 1GB data file transfer during 12 hours
window size: 4MB, number of threads: 16
Sput from Lyon to KEK was about 3.5MB/sec

12. From KEK to Lyon 1GB data transfer bbcp often fail to connect
window size 4MB
number of parallel streams 16
bbcp often fail to connect

13. From Lyon to KEK 1GB data transfer iput is better than bbcp
window size 4MB
number of parallel streams 16
iput is better than bbcp

14. Plan for iRODS at KEK Data transfer for J-PARC project
Store the data at Tokai storage once, then copy to KEK and delete from Tokai
1PB data in a year in maximum
Bandwidth between 2sites
will become 10Gbps
Tokai
KEK

15. Plan for iRODS at KEK
iRODS provide the means for data registration, replication and purging
Set up with our policy
Investigation for high performance transfer
One of the solutions for data transfer

16. Summary Transfer performance of iRODS is better than SRB
and it’s better than bbcp, too
Further investigation for transfer performance
iRODS test for data transfer

18. Bandwidth Lyon-KEK iperf with some options;
-w 4M : TCP window size [Bytes]
-P 16 : the number of parallel threads
-i 5 : periodic bandwidth reports [sec]

19. Transfer performance with Sput
Sput 1GB data file between CC-IN2P3 and KEK with 16 threads
KEK→CCIN2P3: 2.3MB/sec
CCIN2P3→KEK: 3.5MB/sec

20. Network between EU and JP
From JP to EU, the route passes USA
JP-USA has at least 2.4Gbps
EU-USA has 2.5Gbps or more??
The RTT between KEK and IN2P3 is about 285ms
traceroute to ccbbsn09.in2p3.fr ( ), 30 hops max, 38 byte packets
( ) ms ms ms
2 ns1kb.kek.jp ( ) ms ms ms
3 keksw1-ns.kek.jp ( ) ms ms ms
4 kekgw.kek.jp ( ) ms ms ms
5 kek-S1-2-2.sinet.ad.jp ( ) ms ms ms
6 tokyo-core1-P10-0.sinet.ad.jp ( ) ms ms ms
7 nii-S1-P4-0.sinet.ad.jp ( ) ms ms ms
8 nii-IX1-P2-0.sinet.ad.jp ( ) ms ms ms
9 NYC-gate1-P3-0.sinet.ad.jp ( ) ms ms ms
10 sinet.ny1.ny.geant.net ( ) ms ms ms
11 ny.uk1.uk.geant.net ( ) ms ms ms
12 so rt1.par.fr.geant2.net ( ) ms ms ms
13 renater-gw.rt1.par.fr.geant2.net ( ) ms ms ms
14 lyon-pos6-0.cssi.renater.fr ( ) ms ms ms
15 in2p3-lyon.cssi.renater.fr ( ) ms ms ms
16 lyon-core.in2p3.fr ( ) ms ms ms
17 ccbbsn09.in2p3.fr ( ) ms ms ms