1. Performance measurement of transferring files on the federated SRB
KEK Computing Research Center
Yoshimi Iida

2. Outline The Belle experiment at KEK What is SRB?
HEP Data Grid workshop
Performance measurement
Transfer measurement between sites in the Belle SRB federation
Using WAN emulator
Conclusions
29 April 2005
ISGC 2005

3. The Belle experiment at KEK
The Belle experiment at KEK is one of several ongoing large experimental collaborations
The accumulated data up to now is about 2PB including simulation data for analysis
The large amount of data should be analyzed promptly to get statistically improved data for exploring new physics results
Large numbers of files need to be managed consistently and shared easily among the 400 collaborators
29 April 2005
ISGC 2005

4. What is SRB?
SRB provides a uniform interface for connecting to heterogeneous data resources and accessing data sets
SRB, in conjunction with the Metadata Catalog (MCAT), provides a way to access data sets and resources based on logical attributes
29 April 2005
ISGC 2005

5. SRB federation SRB zone Federated MCAT
consist of one or more SRB servers along with one MCAT
Federated MCAT
allow users to access resources and data sets across zones
Server 1.1
MCAT 1
Server 1.2
MCAT 3
Server 3.1
MCAT 2
Server 2.1
Server 2.2
29 April 2005
ISGC 2005

6. Logical file system Single SRB system / - Zone1/ - container/ - home/
- srbUserA.domain/
- srbUserB.domain/
data.txt
- styles/
- trash/
Federated SRB system /
- Zone1/
- container/
- home/
- styles/
- trash/
- Zone2/
- Zone3/ :
29 April 2005
ISGC 2005

7. SRB data management Logical name space Data replication
Mapping of each logical name to physical attributes
UNIX like API and utilities for collections (directories) and data objects (files)
Data replication
replicate onto different resources
29 April 2005
ISGC 2005

8. User management Single Global User Name Space
Uniquely identify by their usernames combined with their domain
Maintained in MCAT
Single sign-on, access all resources
No need for UNIX account at remote sites
29 April 2005
ISGC 2005

9. HEP Data Grid workshop Pre-workshop Workshop
1 - 3 December, 2004
Participants from 8 institutes at 7 countries
SDSC (US)
Australia National U., U. Melbourne (Australia)
ASCC (Taiwan)
KNU (Korea)
IHEP (China)
Krakow (Poland)
KEK (Japan)
Workshop
6 - 7 December, 2004
29 April 2005
ISGC 2005

10. SRB federated MCAT Zone IHEP (China) Zone ASCC (Taiwan) Zone KNU
(Korea)
10Mbps
Internet
100Mbps
100Mbps
Zone ANU
(Australia)
Zone Krakow
(Poland)
622Mbps
100Mbps
Zone KEK (Japan) FW
FW: Firewall
M: MCAT enabled
SRB server
S: SRB server
C: SRB client C S M C
29 April 2005
ISGC 2005

11. Belle SRB federation
Install SRB server, a storage resource and MCAT at each site
SRB version 3.2.1p
Open the KEK firewall from the fixed IP address of the other sites
After the installation, the functionality of the federation was tested and confirmed to be working
29 April 2005
ISGC 2005

12. Transfer measurement from KEK
Copy files from KEK to the other sites
10 files from 0.5 to 100MB in size
‘Scp’ (SRB copy) use parallel I/O by default
Configure to allocate one thread per 2MB of the file size up to a maximum of 16 threads (i.e. 32MB)
29 April 2005
ISGC 2005

13. Transfer rates from KEK
Measured by ‘Scp’
29 April 2005
ISGC 2005

14. Transfer performance between sites in the Belle SRB federation
Source
Destination
KNU
ASCC
ANU
Krakow
IHEP
KEK
9.65
5.89
3.06
2.69
0.59 -
5.68
4.45
4.47
0.61
7.54
2.84
5.09
0.64
4.79
2.53
3.44
0.58
1.21
1.64
4.19
0.60
0.70
0.39
0.44
0.25
Results of 100MB data file transfer in MB/s
29 April 2005
ISGC 2005

15. WAN emulation NIST Net Network emulation package Machine Specification
NIST Net allows a single Linux PC set up as a router to emulate a wide variety of network conditions
Machine Specification
CPU: Pentium4 3.2GHz
Memory: 1GB
OS: Red Hat Linux 8
NIC: GbE×2
Internet
KEK network
GbE
GbE
SRB server
SRB server
100Mbps
100Mbps
router
NIST Net
29 April 2005
ISGC 2005

16. Transfer measurement with various file size
Copy files on WAN emulation
8 files from 10 to 500MB in size
Measure as a function of the RTT from 0 to 300ms
‘Scp’ use parallel I/O by default
Configure to allocate one thread per 2MB of the file size up to a maximum of 16 threads
Measure the bandwidth with iperf
set the parallel client threads to 16
29 April 2005
ISGC 2005

17. Transfer performance with various file size
iperf -P 16
Emulated by NIST Net
29 April 2005
ISGC 2005

18. Transfer performance on WAN (preliminary)
BW 100Mbps
WAN Emulation
(NIST Net)
KNU
ASCC
ANU
Krakow
IHEP
Measured by Scp
29 April 2005
ISGC 2005

19. Bandwidth
The nominal bandwidth between KEK and some sites are same 100Mbps as WAN emulation
The actual bandwidth measured by iperf between KEK and other site is lower than WAN emulation
e.g. 60Mbps for ASCC with iperf
Further investigation is necessary
29 April 2005
ISGC 2005

20. Conclusions
A federation of SRB servers was demonstrated successfully among ANU, KNU, IHEP, ASCC, Krakow and KEK
The current results are preliminary and were obtained in the limited time available in the pre-workshop session
The functionality of data sharing for analysis was proved to be stable and sufficient to be used by Belle collaboration
29 April 2005
ISGC 2005

21. Acknowledgements SDSC (San Diego Supercomputer Center)
M. Wan
ANU (Australia National University)
S. J. McMahon
University of Melbourne
G. Moloney
ASCC (Academia Sinica Computing Centre)
H. Lin
KNU (Kyungpook National University)
K. Kwon
IHEP (Institute of High Energy Physics)
G. Chen
Krakow (Institute of Nuclear Physics, Krakow)
P. Lason and H. Palka
Fujitsu
S. Honma and T. Nakajima
KEK (High Energy Accelerator Research Organization)
Y. Karita, T. Sasaki, S. Y. Suzuki, S. Yashiro and Y. Watase
29 April 2005
ISGC 2005

22. Backups

23. ‘Sls’ display data objects (file) or collections (directory)
‘SgetD’ display information about SRB data objects
‘SgetR’ display information about SRB resource
This is an example.
In SRB space, we can get the list of the SRB data object using Sls command.
And SgetD command display information about SRB data object.
So this is the information of testdata03.txt.
We can see the two information with different SRB resources for one object.
This is data replication.
And user can get the information about the SRB resource using SgetR command.
29 April 2005
ISGC 2005

24. By the federated MCAT, the SRB zones maintained by each sites can see as a single tree structure directory.
29 April 2005
ISGC 2005