1. Performance assessment of distributed SAN systems Bartosz Belter Artur Binczewski Wojbor Bogacki Maciej Brzeźniak TERENA Networking Conference, Poznań, 2005 bart@man.poznan.pl artur@man.poznan.pl wojbor@man.poznan.pl maciekb@man.poznan.pl

2. A genda Introduction Storage Networking challenges IP Storage – new approach to build distributed SANs IP Storage – experiments in Polish NREN PIONIER

3. Storage Networking Storage Area Network is a high-speed special- purpose network (or subnetwork) that interconnects different kinds of data storage devices with associated data servers. Usually SANs are based on Fibre Channel or SCSI technology. Storage Networking definition from SNIA The practice of creating, installing, administering, or using networks whose primary purpose is the transfer of data between computer systems and storage elements and among storage elements.

4. Storage Networking – the importance Currently focused on application aspect: Local and remote mirroring, backups and disaster recovery Remote data replication Local and remote storage access Explosion of Storage Data: Data Warehousing statistics, charts, reporting Internet web hosting e-commerce e-bussiness Customer Relationship Management

5. Are separated SANs enough for high performance computing? How to integrate remote, separated HPC centers in single, distributed, scalable high performance system? HPC centers use different technology, not always applicable in backbone network traditional Storage Networking introduces additional limitation: maximum distance to transfer data Traditional Storage Networking technology SCSIFC Maximum cable length25 meters if no more then 2 devices are used, otherwise 12 meters 30 meters device to device (copper), 10 000 meters device to device (optical) Maximum speed2.560 Gbpsup to 2.125 Gbps (10 Gbps in the near future) Maximum number of devices 16126

6. IP Storage IP Storage is a new approach to extend existing Storage Area Networks using IP protocol, usually over Gigabit Ethernet. According to SNIA, IP Storage is: Computer systems and storage elements that are connected via Internet Protocol (IP) The transport of storage traffic over an IP network IP Storage traffic carries the traditional block I/O using SCSI protocols supported by most open systems According to SNIA, IP Storage is not: File-level transfer of data (i.e NAS) Object level access (i.e. http, ftp)

7. IP Storage protocols Internet Small Computer Systems Interface (iSCSI) iSCSI is a protocol which enables transfer of data-block traffic via IP network instead of a direct SCSI compatible bus. It uses a TCP layer and unlike other network storage protocols it requires only Ethernet interface to operate. Internet Fibre Channel Protocol (iFCP) iFCP is a new standard for extending Fibre Channel storage networks across the Internet. It provides a mechanism to deliver storage data to and from Fiber Channel storage devices over SAN infrastructure or even over the Internet using TCP/IP. Fibre Channel Over IP (FCIP) FCIP describes mechanisms that allow the interconnection of islands of Fibre Channel storage area networks over IP-based networks to form a unified storage area network in a single Fibre Channel fabric. FCIP relies on IP-based network services to provide the connectivity between the storage area network islands over local area networks, metropolitan area networks, or wide area networks.

8. The experiment Tests were performed in Polish Optical Internet PIONIER testbed interconnects 9 HPC centers maximum distance length - over 1500 km no QOS provided for FCIP traffic across WAN infrastructure. FCIP was tested based on production network IP Storage vendor solutions used in tests: CNT UltraNet Edge 3000 Cisco MDS 9216 and 8-port IP Storage Services Module The main goals of the experiment: to build the distributed data architecture based on new IP Storage technology to verify IP Storage protocols (iSCSI and FCIP) used in live network environment to evaluate the performance of IP Storage vendor solutions connected via Gigabit Ethernet

9. Testbed description

10. Hardware: PC Processor: Pentium 4 3.0 GHz Memory: 512 MB Hard Disc: Segate Baracuda 7200.7 SATA Western Digital Raptor WD740GD Gigabit Ethernet Controller Fibre Channel interface QLA 2340 IP Storage element Cisco - MDS 9216 CNT - UltraNet Edge 3000 RAID 0 includes two storage arrays PC IP Storage element RAID 0 Gigabit Ethernet switch Gigabit Ethernet

11. Benchmark software: Windows 2000 HD Tach SiSoftware Sandra Linux Suse 9.1 and 9.2 Bonnie IOZone HDParm IOMeter MySQL database benchmark Performance Benchmark from Tivoli SANergy Testing methodology

12. Test results Performance Benchmark from Tivoli SANergy Reading performance MB/s Test Site Poznań 0 km Zielona Góra 161 km Wrocław 390 km Opole 500 km Katowice 650 km Bielsko Biała 740 km Kraków 850 km Radom 1090 km Białystok 1540 km 10 20 30 40 50 60 70 80 90 100 FCIP iSCSI Throughput as it was expected the overall performance decreases, it has linear relationship with the distance interconnection of distant HPC centers is possible even over 1500 km! (but the overall performance decreases twice)

13. Test results Performance Benchmark from Tivoli SANergy Reading/Writing performance (Write Acceleration option) Write Acceleration MB/s Test Site Wrocław 390 km Bielsko Biała 740 km Białystok 1540 km 10 20 30 40 50 60 70 80 90 100 FCIP Reading Throughput MB/s Test Site 10 20 30 40 50 60 70 80 90 100 Writing Throughput some vendors introduce their own improvements to protocols - CISCO implements "Write Acceleration" (WA) feature WA has not affected the reading performance WA introduces interesting results for writing performance – in Białystok (1540 km) writing performance increases twice in comparison to standard FCIP transmission Wrocław 390 km Bielsko Biała 740 km Białystok 1540 km Poznań 0 km Poznań 0 km

14. Test results %CPU Test Site FCIP iSCSI IOMeter: Reading - CPU load 0 100 80 20 40 60 Poznań 0 km Zielona Góra 161 km Wrocław 390 km Opole 500 km Katowice 650 km Bielsko Biała 740 km Kraków 850 km Radom 1090 km Białystok 1540 km iSCSI software driver introduces higher CPU load than FCIP (handled by a hardware)

15. Test results Copying of 700MB raw data sec Test Site 2.5 5 7.5 10 12.5 15 17.5 20 22.5 25 FCIP iSCSI Time Poznań 0 km Zielona Góra 161 km Wrocław 390 km Opole 500 km Katowice 650 km Bielsko Biała 740 km Kraków 850 km Radom 1090 km Białystok 1540 km good linear relationship with the distance

16. Test results – MySQL benchmark MySQL – popular Open Source Relational Database benchw – simple benchmark for relational databases (http://benchw.sourceforge.net) DB Tables: fact01: 1,02 GB - 10mln records,dim1:0,24MB - 10k records, dim0: 0,24 MB - 10k records, dim2 1,40MB - 10k record Query types: Loading data into the database:all tables Q0: select from 2 tables, 2 cond. (dim0 & fact01, =, <>, numbers) Generating indexes for the table:all tables DB & DB filesystem recreated each time

17. Test results sec Test Site FCIP iSCSI Time MySQL database benchmark 0 100 80 20 40 60 Loading data to database server Poznań 0 km Zielona Góra 161 km Wrocław 390 km Opole 500 km Katowice 650 km Bielsko Biała 740 km Kraków 850 km Radom 1090 km Białystok 1540 km load to database performs sequential reading of input file and putting data into the db structure operation performance scales linearly with the distance

18. Test results MySQL database benchmark sec Test Site 0 20 40 60 Query no 0 Time FCIP iSCSI Poznań 0 km Zielona Góra 161 km Wrocław 390 km Opole 500 km Katowice 650 km Bielsko Biała 740 km Kraków 850 km Radom 1090 km Białystok 1540 km operation reads from two database tables only even non-complicated query introduces decrease of performance in comparison between local and remote measurements

19. Test results MySQL database benchmark sec Test Site Index generating Time FCIP iSCSI Poznań 0 km Zielona Góra 161 km Wrocław 390 km Opole 500 km Katowice 650 km Bielsko Biała 740 km Kraków 850 km Radom 1090 km Białystok 1540 km 0 200 400 600 operation reads from all tables stored in database and writes small amount of data (generated indexes) more complicated request introduces significant decrease of performance in comparison between local and remote measurements

20. Test results dd command – FCIP vs. iSCSITime sec Test Site 5 10 15 20 25 30 35 40 45 50 55 4096 16384 32768 131072 Block Size FCIPiSCSI Poznań 0 km Zielona Góra 161 km Wrocław 390 km Opole 500 km Katowice 650 km Bielsko Biała 740 km Kraków 850 km Radom 1090 km Białystok 1540 km

21. Test results dd command – FCIP vs. iSCSITime sec Test Site 5 10 15 20 25 30 35 40 45 50 55 4096 16384 32768 131072 Block Size FCIPiSCSI Poznań 0 km Zielona Góra 161 km Wrocław 390 km Opole 500 km Katowice 650 km Bielsko Biała 740 km Kraków 850 km Radom 1090 km Białystok 1540 km

22. Test results dd command – FCIP vs. iSCSITime using block size 4kB, 16kB or 32kB there are no significant differences between iSCSI and FCIP protocols the greater block size – the better performance, but... too large block size decreases overall performance (block size > raid chunk size) sec Test Site 5 10 15 20 25 30 35 40 45 50 55 4096 16384 32768 131072 Block Size FCIPiSCSI Poznań 0 km Zielona Góra 161 km Wrocław 390 km Opole 500 km Katowice 650 km Bielsko Biała 740 km Kraków 850 km Radom 1090 km Białystok 1540 km

23. IP Storage – tuning up the transmission configured TCP parameters: TCP Maximum Window Size (default: 64 Kbytes, maximum: 32 Mbytes) MWS > B x Dexample: Gigabit Ethernet Network, RTT = 10 ms B – end to end bandwith MWS > 1000 x 10 bit/sec x 10 x 10 sec D – round trip time MWS > ~1,2 Mbytes TCP Selective Acknowledge TCP SACK helps TCP connections that are extended over long distances to recover from any sort of frame loss that may occur MTU set to 2148 bytes on IP Storage devices for iSCSI protocol - hardware TCP Offload Engine was not tested for FCIP protocol – FCIP Compression was not tested 6-3

24. IP Storage – conclusions As it was expected the overall performance decreases, it has linear relationship with the distance (latency) Assuming linear characteristic – its possible and easy to predict how overall performance decreases with the increase of distance (latency): for each 100 km of distance -> performance decreases about 4 MB/s for every 1 ms of latency -> performance decreases about 3 MB/s Interconnection of far HPC centers is possible even over 1500 km! (but the overall performance decreases twice) Write Acceleration feature considerably increases writing performance iSCSI software driver used in tests could really affect the iSCSI performance, especially for short distances Interoperability Even if IP Storage protocols published by IETF – still an important issue!

25. Thank you!