1. SRB at KEK Yoshimi Iida, Kohki Ishikawa KEK – CC-IN2P3 Meeting on Grids at Lyon September 11-13, 2006

2. SRB at KEK 2 The Belle experiment at KEK The Belle experiment at KEK is one of several ongoing large experimental collaborations The accumulated data up to now is about 1PB including simulation data for analysis The large amount of data should be analyzed promptly to get statistically improved data for exploring new physics results The computing facility at KEK is so limited that a large portion of the simulation jobs are shared by the collaboration institutes world wide Some of the collaborators requested us to support a GRID environment for data distribution and efficient analysis

3. SRB at KEK 3 Why SRB? Easy to install and start over Even works with the password authentication Light clients including command line, GUI and the web Server side is also light Only one SQL server is necessary except SRB servers Federation of SRB zones is available Independent operation, but still information is federated

4. SRB at KEK 4 History 2003 Jun. 06 Start operation of the test system 1 with PostgreSQL HPSS interface 2003 Oct. 22 Test system 2 with DB2 PostgreSQL looks better than DB2 2004 Apr. 02 Introduction in BELLE Computer system Interface to SONY Peta Site (HSM) via NFS Test with Melbourne 2004 Dec. 06 HEP Data Grid workshop 2004 Together with Michel Wan, SDSC, SRB federation has been established among AU, KR, TW, CN, PL and JP

5. SRB at KEK 5 The location of the federated zones Krakow, PL SDSC, US KEK, JP KNU, KR IHEP, CN ASCC, TW ANU, AU The latency was measured in December 2004 Destination (from KEK) RTT (msec) Nominal BW (Mbps) IHEP50210 Krakow327100 ANU292622 ASCC33100 KNU23.6100

6. SRB at KEK 6 Transfer rates from KEK Measured by Scp commands that use parallel I/O by default Files ranging from 0.5MB to 100MB were used in the tests as the number of threads that SRB chooses to use is determined by the file size The parallel transfer is effective the improvement of the transfer performance

7. SRB at KEK 7 Transfer performance on WAN KNU ASCC ANU Krakow IHEP WAN Emulation (NIST Net) The results of WAN emulation are typical data transfer performance for the SRB federation There is a difference between WAN emulation and experimental results The packet loss is not generated in WAN Emulation The latency and quality of the network is the key to obtain better performance and that higher network bandwidth alone is not sufficient.

8. SRB at KEK 8 History 2003 Jun. 06 Start operation of the test system 1 with PostgreSQL HPSS interface 2003 Oct. 22 Test system 2 with DB2 PostgreSQL looks better than DB2 2004 Apr. 02 SRB in BELLE Computer system Interface to SONY Peta Site via NFS Test with Melbourne 2004 Dec. 06 HEP Data Grid workshop 2004 Together with Michel Wan, SDSC, SRB federation has been established among AU, KR, TW, CN, PL and JP 2006 Apr. 18 Replace with the new system 2 kinds of interface to HPSS 2006 Jun. 01 Federated with IN2P3

9. SRB at KEK 9 SRB system srbcert SRB Master PostgreSQL UI SRB Client SRB Server srbcert SRB Server srbcert SRB Server HPSS External Disk Sub-sys HPSS APIHPSS VFS rac01rls10rls09rsr01 “rls10-hpss-vfs”“rls09-hpss-hsi” “rsr01-ufs” GRID NW

10. SRB at KEK 10 HPSS Core Disk Mover hpssfsd SRB Server HPSS-enabled NFS v3/v4 Protocol SRB Server HPSS Protocol HSI Lib. SRB Server hpssfsd HSI Lib. Local Access Authentication Authorization Catalog data SRB Integration with HPSS SRB supports HPSS resource type we can compile HPSS- enabled SRB with HPSS before v4.5, relatively easily But at KEK, A little of change of SRB source code was needed Authentication with Kerberos Non-DCE Linux

11. SRB at KEK 11 Challenge When HPSS migrates v4.5 to v6.2 HPSS changes Meta-data management & transaction monitoring component Low-level API Meta-data structure definition So, We doubt the source level compatibility of HPSS driver of SRB with HPSS v6.2

12. SRB at KEK 12 HPSS Core Disk Mover hpssfsd SRB Server HPSS-enabled NFS v3/v4 Protocol SRB Server HPSS Protocol HSI Lib. SRB Server hpssfsd HSI Lib. Local Access Authentication Authorization Catalog data Work Around with HPSS VFS Access Work Around for Integration key is “hpssfsd” provide POSIX I/O function into HPSS name space through VFS layer of Linux wrapper of HIS can be exported by NFS we try 2 pattern 1. SRB UNIX Res. Type 2. SRB NFS Res. Type These method are running in early phase, because they does not require any source code modification

13. SRB at KEK 13 HPSS Core Disk Mover hpssfsd SRB Server HPSS-enabled NFS v3/v4 Protocol SRB Server HPSS Protocol HSI Lib. SRB Server hpssfsd HSI Lib. Local Access Authentication Authorization Catalog data HPSS-enabled SRB with HPSS v6.2 Now running with some restriction with v6.2 Modify Source Code After v6.2, HPSS change its low-level API related to some meta-data structure. So, SRB must change its code related to HPSS Some Restriction necessity to renew Kerberos ticket

14. SRB at KEK 14 Summary of SRB - HPSS HPSS-enabled SRB with HPSS v6.2 is require some modification of source code, but it can be built at first, to use HPSS v6.2 with SRB, POSIX I/O function is useful. performance comparison is a next test subject

15. SRB at KEK 15 Specification HPSS External Disk Sub-sys rls10rls09rsr01rac01 CPURAM sizeOS (kernel)SRB resource rsr01Intel Xeon MP 3.0GHz x 4 4GBRHEL 3 AS (2.4.21-37) Storage Subsystem (1TB) is connected with Fiber-Channel rls09Intel Xeon 3.0GHz x 2 1GBRHEL WS 4, (2.6.9-5) HPSS-enabled SRB Server uses HPSS Client API Library, SRB source code includes HPSS API call. rls10Intel Xeon 3.0GHz x 2 1GBRHEL WS 4, (2.6.9-5) HPSS VFS service, which offers POSIX VFS gateway toward HPSS Hierarchical Storage Area. rac01Intel Xeon 3.0GHz x 2 1GBRHEL ES 3, (2.4.21-37) Client GRID LAN

16. SRB at KEK 16 Transfer performance (preliminary) Data file : 1GB RTT : 0.9ms BW : 941Mbps(iperf) rsr01-ufsrls09-hpss-hsirls10-hpss-vfs rac01rls10rls09rsr01 GRID LAN ImportExport

17. SRB at KEK 17 Transfer performance (preliminary) Data file : 1GB RTT : 0.9ms BW : 941Mbps(iperf) rsr01-ufsrls09-hpss-hsirls10-hpss-vfs rac01rls10rls09rsr01 GRID LAN ImportExport

18. SRB at KEK 18 Transfer performance (preliminary) Data file : 1GB RTT : 0.08ms BW : 941Mbps(iperf) rsr01-ufsrls09-hpss-hsirls10-hpss-vfs GRID LAN rac01rls10rls09rsr01 ImportExport

19. SRB at KEK 19 Data file : 1GB RTT : 262ms BW : 17Mbps(iperf) LyonFS3 rsr01-ufsrls09-hpss-hsirls10-hpss-vfs rac01rls10rls09rsr01 Internet Transfer performance (preliminary) KEKIN2P3 ImportExport

20. SRB at KEK 20 Summary We replaced the new SRB system Need to more test for HPSS We could establish federation with IN2P3 and KEK In order to improve the data transfer performance we should investigate the network between France and Japan Federation with RAL

21. SRB at KEK 21 Back up

22. SRB at KEK 22 Traceroute from KEK to IN2P3 traceroute to ccbbsn09.in2p3.fr (134.158.104.79), 30 hops max, 38 byte packets 1 130.87.208.203 (130.87.208.203) 0.340 ms 0.245 ms 0.224 ms 2 ns1kb.kek.jp (130.87.5.11) 0.273 ms 0.311 ms 0.250 ms 3 keksw1-ns.kek.jp (130.87.4.34) 0.457 ms 0.420 ms 0.397 ms 4 kekgw.kek.jp (130.87.4.1) 0.554 ms 0.564 ms 0.514 ms 5 kek-S1-2-2.sinet.ad.jp (150.99.197.9) 0.735 ms 0.779 ms 0.565 ms 6 tokyo-core1-P10-0.sinet.ad.jp (150.99.197.33) 5.946 ms 6.299 ms 6.247 ms 7 nii-S1-P4-0.sinet.ad.jp (150.99.197.22) 6.243 ms 6.437 ms 6.311 ms 8 nii-IX1-P2-0.sinet.ad.jp (150.99.199.174) 6.148 ms 6.230 ms 6.121 ms 9 NYC-gate1-P3-0.sinet.ad.jp (150.99.198.246) 180.621 ms 180.704 ms 180.666 ms 10 sinet.ny1.ny.geant.net (62.40.103.233) 180.679 ms 180.750 ms 180.729 ms 11 ny.uk1.uk.geant.net (62.40.96.170) 249.457 ms 249.374 ms 249.406 ms 12 so-4-0-0.rt1.par.fr.geant2.net (62.40.112.105) 256.739 ms 256.797 ms 384.238 ms 13 renater-gw.rt1.par.fr.geant2.net (62.40.124.70) 256.639 ms 256.790 ms 256.617 ms 14 lyon-pos6-0.cssi.renater.fr (193.51.179.14) 262.451 ms 262.371 ms 262.214 ms 15 in2p3-lyon.cssi.renater.fr (193.51.181.6) 262.434 ms 262.715 ms 262.433 ms 16 lyon-core.in2p3.fr (134.158.224.3) 263.994 ms 262.435 ms 262.991 ms 17 ccbbsn09.in2p3.fr (134.158.104.79) 262.862 ms 262.470 ms 263.725 ms

23. SRB at KEK 23 Slide by Sakamoto@ICEPP

24. SRB at KEK 24 IN2P3CC-ICEPP iperf Test Kernel 2.6 TCP Congestion Control TCP Reno BIC TCP +PSPacer Precision Software Pacer From AIST GridTCP project Slide by Sakamoto@ICEPP