1. National Institute of Advanced Industrial Science and Technology Belle/Gfarm Grid Experiment at SC04 Osamu Tatebe Grid Technology Research Center, AIST APAN Workshop Jan 27, 2005 Bangkok

2. National Institute of Advanced Industrial Science and Technology Goal and feature of Grid Datafarm Goal Dependable data sharing among multiple organizations High-speed data access, High-performance data computing Grid Datafarm Gfarm File System – Global dependable virtual file system Federates scratch disks in PCs Parallel & distributed data computing Associates Computational Grid with Data GridFeatures Secured based on Grid Security Infrastructure Scalable depending on data size and usage scenarios Data location transparent data access Automatic and transparent replica selection for fault tolerance High-performance data access and computing by accessing multiple dispersed storages in parallel (file affinity scheduling)

3. National Institute of Advanced Industrial Science and Technology Grid Datafarm (1): Gfarm file system - World- wide virtual file system [CCGrid 2002] Transparent access to dispersed file data via global namespace Files can be stored somewhere in a Grid Applications can access Gfarm file system without any modification as if it were mounted at /gfarm Automatic and transparent replica selection for fault tolerance and access-concentration avoidance Gfarm File System /gfarm ggfjp aistgtrc file1file3 file2 file4 file1file2 File replica creation Global namespace mapping

4. National Institute of Advanced Industrial Science and Technology Grid Datafarm (2): High-performance data access and computing support [CCGrid 2002] CPU Gfarm File System Computational Grid User A submits that accesses File A is allocated on a node that has network Job A File A Users viewPhysical execution view User B submits that accesses File B Job A Job B is allocated on a node that has Job B File B Compute and file system nodes Shared network file system Do not separate Storage and CPU Scalable file I/O by exploiting local I/O

5. National Institute of Advanced Industrial Science and Technology Gfarm TM Data Grid middleware Open source development Gfarm TM version 1.0.4-4 released on Jan 11 th, 2005 ( http://datafarm.apgrid.org/ ) http://datafarm.apgrid.org/ Read-write mode support, more support for existing binary applications A shared file system in a cluster or a grid Accessibility from legacy applications without any modification Standard protocol support by scp, GridFTP server, samba server,... application Gfarm client library Metadata server CPU... gfsd gfmdslapd Compute and file system nodes * Existing applications can access Gfarm file system without any modification using LD_PRELOAD

6. National Institute of Advanced Industrial Science and Technology Demonstration File manipulation cd, ls, cp, mv, cat,... grep Gfarm command File replica creation, node & process information Remote (parallel) program execution gfrun prog args... gfrun -N #procs prog args... gfrun -G filename prog args...

7. National Institute of Advanced Industrial Science and Technology Belle/Gfarm Grid experiment at SC2004 1. Online KEKB/Belle distributed data analysis 2. KEKB/Belle large-scale data analysis (terabyte-scale US-Japan file replication) (terabyte-scale US-Japan file replication)

8. National Institute of Advanced Industrial Science and Technology 1. Online KEKB/Belle distributed data analysis (1) Online distributed and parallel data analysis of raw data using AIST and SDSC clusters Realtime histogram and event data display at SC2004 conference hall AIST SDSC KEK SC2004 Gfarm file system - on demand data replication - distributed & parallel data analysis Raw data 10 MB/sec 128 nodes 3.75 TB 64 nodes 50 TB 192 nodes 53.75 TB - realtime histogram display - realtime event data display

9. National Institute of Advanced Industrial Science and Technology 1. Online KEKB/Belle distributed data analysis (2) Construct a shared network file system between Japan and US Store KEKB/Belle raw data to the Gfarm file system Physically, it is divided into N fragments, and stored on N different node Every compute node can access it as if it were mounted at /gfarm AIST SDSC KEK SC2004 Raw data 10 MB/sec 128 nodes 3.75 TB 64 nodes 50 TB 192 nodes 53.75 TB - realtime histogram display - realtime event data display Gfarm File System

10. National Institute of Advanced Industrial Science and Technology 1. Online KEKB/Belle distributed data analysis (3) Basf is installed at /gfarm/~/belle Install once, run everywhere The raw data will be analyzed at AIST or SDSC just after it is stored Analyzed data can be viewed at SC2004 in realtime Histogram snapshot is generated every 5 minutes AIST SDSC KEK SC2004 Raw data 10 MB/sec 128 nodes 3.75 TB 64 nodes 50 TB 192 nodes 53.75 TB - realtime histogram display - realtime event data display Gfarm File System Computational Grid

11. National Institute of Advanced Industrial Science and Technology 2. KEKB/Belle large-scale data analysis in a Grid Gfarm file system using SC conference hall and AIST F cluster Assume data is stored at SC conference hall Terabyte-scale mock data Submit data analysis job at AIST F cluster Required data is automatically transferred from SC to AIST on demand Users just see a shared file system Network transfer rate is measured Conf 1: 8 parallel processes (2GBx8 data) Conf 2: 16 parallel processes (8GBx16 data)

12. National Institute of Advanced Industrial Science and Technology 2. Network & machine configuration SC2004 StorCloud AIST F cluster JGN2 Tsukuba WAN 10Gbps JGN2 Japan-US Tokyo 10Gbps Chicago Pittsburgh PC PCx8 1Gx8FCx16 2TBx16 10G (OC192) AIST F cluster PCx256

13. National Institute of Advanced Industrial Science and Technology SCAIST (Iperf x 8) 7,925,607,155 bps (Wed Nov 10 17:13:22 JST 2004) (5-sec average bandwidth, 991 Mbps / TCP flow)

14. National Institute of Advanced Industrial Science and Technology Iperf measurement Standard TCP (Linux 2.4) Socket buffer size and txqueuelen No kernel patch, no TCP modification No traffic shaping No bottleneck, no problem

15. National Institute of Advanced Industrial Science and Technology Conf 1: 8 processes (2GBx8) 2,084,209,307 bps (Fri Nov 12 03:41:54 JST 2004) (5-sec average, 261 Mbps / TCP flow, disk performance of F cluster)

16. National Institute of Advanced Industrial Science and Technology Conf 2: 16 processes (8GBx16) 738,920,649 bps (Fri Nov 12 05:30:35 JST 2004) (5-sec average, 46 Mbps!! / TCP flow, ?????)

17. National Institute of Advanced Industrial Science and Technology Conf 2: network traffic of JGN2 int l link Heavy traffic when application started Heavy packet loss ssthresh decreases

18. National Institute of Advanced Industrial Science and Technology Summary Belle/Gfarm Grid experiment at SC2004 1. Online KEKB/Belle distributed data analysis 2. KEKB/Belle large-scale data analysis We succeeded distributed & parallel data analysis of KEKB/Belle data and realtime display at SC conference hall

19. National Institute of Advanced Industrial Science and Technology Development Status and Future Plan Gfarm – Grid file system Global virtual file system A dependable network shared file system in a cluster or a grid High performance data computing support Associates Computational Grid with Data Grid Gfarm v1 Data Grid middleware Version 1.0.4-4 released on Jan 11, 2005 (http://datafarm.apgrid.org/)http://datafarm.apgrid.org/ Existing programs can access Gfarm file system as if it were mounted at /gfarm Gfarm v2 – towards *true* global virtual file system POSIX compliant - supports read-write mode, advisory file locking,... Performance and Robustness improved, Security enhanced. Can be substituted for NFS, AFS,... Application area Scientific application (High energy physics, Astronomic data analysis, Bioinformatics, Computational Chemistry, Computational Physics,...) Business application (Dependable data computing in eGovernment and eCommerce,...) Other applications that needs dependable file sharing among several organizations Standardization effort with GGF Grid File System WG (GFS-WG) https://datafarm.apgrid.org/