Masaki Hirabaru 情報処理学会四国支部講演会 December 17, 2004 インターネットでの長距離高性能データ転送.

Slides:

Advertisements

Similar presentations

Engineering Meeting Report Aug. 29, 2003 Kazunori Konishi.

Advertisements

e-VLBI Deployments with Research Internet

Pathload A measurement tool for end-to-end available bandwidth Manish Jain, Univ-Delaware Constantinos Dovrolis, Univ-Delaware Sigcomm 02.

The Effect of Router Buffer Size on HighSpeed TCP Performance Dhiman Barman Joint work with Georgios Smaragdakis and Ibrahim Matta.

PFLDNet Argonne Feb 2004 R. Hughes-Jones Manchester 1 UDP Performance and PCI-X Activity of the Intel 10 Gigabit Ethernet Adapter on: HP rx2600 Dual Itanium.

TCP Westwood: Experiments over Large Pipes Cesar Marcondes Anders Persson Prof. M.Y. Sanadidi Prof. Mario Gerla NRL – Network Research Lab UCLA.

IP Performance Measurements using Surveyor Matt Zekauskas Guy Almes, Sunil Kalidindi August, 1998 ISMA 98.

Current plan for e-VLBI demonstrations at iGrid2005 and SC2005 Yasuhiro Koyama *1, Tetsuro Kondo *1, Hiroshi Takeuchi *1, Moritaka Kimura, Masaki Hirabaru.

Ongoing e-VLBI Developments with K5 VLBI System Hiroshi Takeuchi, Tetsuro Kondo, Yasuhiro Koyama, and Moritaka Kimura Kashima Space Research Center/NICT.

Electronic Transmission of Very- Long Baseline Interferometry Data National Internet2 day, March 18, 2004 David LapsleyAlan Whitney MIT Haystack Observatory,

KEK Network Qi Fazhi KEK SW L2/L3 Switch for outside connections Central L2/L3 Switch A Netscreen Firewall Super Sinet Router 10GbE 2 x GbE IDS.

1 A Basic R&D for an Analysis Framework Distributed on Wide Area Network Hiroshi Sakamoto International Center for Elementary Particle Physics (ICEPP),

E-VLBI developments with the K5 VLBI system Koyama Yasuhiro *1, Kondo Tetsuro *1, Kimura Moritaka *1, Takeuchi Hiroshi *1, and Masaki Hirabaru *2 *1 Kashima.

NOC/Engineering Meeting Date & Time: 12: :00 over Lunch 13: :00 at Lotus Hall 1 Yoshinori Kitatsuji: APAN-JP NOC Report & Plan Masaki Hirabaru:

Masaki Hirabaru NICT The 3rd International HEP DataGrid Workshop August 26, 2004 Kyungpook National Univ., Daegu, Korea High Performance.

Developments for real-time software correlation e-VLBI Y. Koyama, T. Kondo, M. Kimura, M. Sekido, M. Hirabaru, and M. Harai Kashima Space Research Center,

Trial on the Efficient Use of Trunk Communcation Lines for VLBI in Japan Noriyuki Kawaguchi NATIONAL Astronomial Observatory,Japan eVLBI Workshop

Large File Transfer on 20,000 km - Between Korea and Switzerland Yusung Kim, Daewon Kim, Joonbok Lee, Kilnam Chon

E-VLBI at ≥ 1 Gbps -- “unlimited” networks? Tasso Tzioumis Australia Telescope National Facility (ATNF) 4 November 2008.

Masaki Hirabaru ISIT and Genkai Genkai / Hyeonhae Workshop in Fukuoka Feb. 27, 2003 Performance of G / H Link.

Masaki Hirabaru Internet Architecture Group GL Meeting March 19, 2004 High Performance Data transfer on High Bandwidth-Delay Product Networks.

Data GRID Activity in Japan Yoshiyuki WATASE KEK (High energy Accelerator Research Organization) Tsukuba, Japan

The Internet Hall of Fame Induction Memorial Party for the late Dr. Masaki Hirabaru Shigeki Goto JPNIC Wednesday 4 March,

1 ASTER DATA Transfer System Earth Remote Sensing Data Analysis Center Kunjuro Omagari ERSDAC 23th APAN Meeting.

1 Masaki Hirabaru and Yasuhiro Koyama APEC-TEL APGrid Workshop September 6, 2005 e-VLBI: Science over High-Performance Networks.

E-VLBI over TransPAC Masaki HirabaruDavid LapsleyYasuhiro KoyamaAlan Whitney Communications Research Laboratory, Japan MIT Haystack Observatory, USA Communications.

Hirabaru, Koyama, and Kimura NICT APAN NOC Meeting January 20, 2005 e-VLBI updates.

Network Tests at CHEP K. Kwon, D. Han, K. Cho, J.S. Suh, D. Son Center for High Energy Physics, KNU, Korea H. Park Supercomputing Center, KISTI, Korea.

Masaki Hirabaru CRL, Japan APAN Engineering Team Meeting APAN 2003 in Busan, Korea August 27, 2003 Common Performance Measurement Platform.

Masaki Hirabaru CRL, Japan ITRC MAI BoF Kochi November 6, 2003 広帯域・高遅延ネットワークでのＴＣＰ性能計測.

Measurement of Routing Switch-over Time with Redundant Link Masaki Hirabaru (NICT), Teruo Nakai (KDDI), Yoshitaka Hattori (KDDI), Motohiro Ishii (QIC),

Data transfer over the wide area network with a large round trip time H. Matsunaga, T. Isobe, T. Mashimo, H. Sakamoto, I. Ueda International Center for.

APAN 10Gbps End-to-End Performance Measurement Masaki Hirabaru (NICT), Takatoshi Ikeda (KDDI/NICT), and Yasuichi Kitamura (NICT) July 19, 2006 Network.

1 Gbps Disk-Based VLBI Observation for Future East Asia VLBI Observations Kiyoaki Wajima 18 March 2009 동아시아 VLBI 워크샵 Yamaguchi University Introduction.

E-VLBI and End-to-End Performance Masaki HirabaruYasuhiro KoyamaTetsuro Kondo NICT KoganeiNICT Kashima

High TCP performance over wide area networks Arlington, VA May 8, 2002 Sylvain Ravot CalTech HENP Working Group.

APII/APAN Measurement Framework: Advanced Network Observatory Masaki Hirabaru (NICT), Takatoshi Ikeda (KDDI Lab), Motohiro Ishii (QIC), and Yasuichi Kitamura.

High-speed TCP  FAST TCP: motivation, architecture, algorithms, performance (by Cheng Jin, David X. Wei and Steven H. Low)  Modifying TCP's Congestion.

Masaki Hirabaru Tsukuba WAN Symposium 2005 March 8, 2005 e-VLBI and End-to-End Performance over Global Research Internet.

E-VLBI Activity in NICT National Institute of Information and Communications Technology (NICT) Kashima Space Research Center M. Sekido, 、 Y. Koyama, M.

Possible Engineering Support for Dual Polarization and Ultra Wide Band Observation with EAVN N. Kawaguchi National Astronomical Observatory East Asia VLBI.

Masaki Hirabaru Network Performance Measurement and Monitoring APAN Conference 2005 in Bangkok January 27, 2005 Advanced TCP Performance.

Data Transport Challenges for e-VLBI Julianne S.O. Sansa* * With Arpad Szomoru, Thijs van der Hulst & Mike Garret.

Masaki Hirabaru NICT Koganei 3rd e-VLBI Workshop October 6, 2004 Makuhari, Japan Performance Measurement on Large Bandwidth-Delay Product.

First of ALL Big appologize for Kei’s absence Hero of this year’s LSR achievement Takeshi in his experiment.

TCP transfers over high latency/bandwidth networks Internet2 Member Meeting HENP working group session April 9-11, 2003, Arlington T. Kelly, University.

1 Analysis of a window-based flow control mechanism based on TCP Vegas in heterogeneous network environment Hiroyuki Ohsaki Cybermedia Center, Osaka University,

Performance Engineering E2EpiPEs and FastTCP Internet2 member meeting - Indianapolis World Telecom Geneva October 15, 2003

1 Masaki Hirabaru Network Architecture Group PL Meeting New Generation Network Research Center July 26, 2006 A Role of Network Architecture in e-VLBI.

Masaki Hirabaru NICT APAN JP NOC Meeting August 31, 2004 e-VLBI for SC2004 bwctl experiment with Internet2.

H. OhsakiITCom A control theoretical analysis of a window-based flow control mechanism for TCP connections with different propagation delays Hiroyuki.

Data Transport Challenges for e-VLBI Julianne S.O. Sansa* * With Arpad Szomoru, Thijs van der Hulst & Mike Garret.

GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 Lessons Learned in Grid Networking or How do we get end-2-end performance to Real Users ? Richard.

TCP transfers over high latency/bandwidth networks & Grid DT Measurements session PFLDnet February 3- 4, 2003 CERN, Geneva, Switzerland Sylvain Ravot

Possible eVLBI connection between Eastern Asian Observatories Noriyuki KAWAGUCHI National Astronomical Observatory, Japan eVLBI Workshop

Final EU Review - 24/03/2004 DataTAG is a project funded by the European Commission under contract IST Richard Hughes-Jones The University of.

INDIANAUNIVERSITYINDIANAUNIVERSITY Status of FAST TCP and other TCP alternatives John Hicks TransPAC HPCC Engineer Indiana University APAN Meeting – Hawaii.

1 Masaki Hirabaru and Yasuhiro Koyama PFLDnet 2006 Febrary 2, 2006 International e-VLBI Experience.

Radio Astronomy Applications Group Kashima Space Research Center IUGG2003, Sapporo (C) ISAS Orbit Determination of The NOZOMI Spacecraft using Differential.

1 eVLBI Developments at Jodrell Bank Observatory Ralph Spencer, Richard Hughes- Jones, Simon Casey, Paul Burgess, The University of Manchester.

Masaki Hirabaru (NICT) and Jin Tanaka (KDDI) Impact of Bottleneck Queue on Long Distant TCP Transfer August 25, 2005 NOC-Network Engineering Session Advanced.

Realization of a stable network flow with high performance communication in high bandwidth-delay product network Y. Kodama, T. Kudoh, O. Tatebe, S. Sekiguchi.

Efficient utilization of 40/100 Gbps long-distance network

Networking between China and Europe

Transport Protocols over Circuits/VCs

Study on Prototype KVN high-speed Recorder Development using FirmOS Jae-Hwan Yeom, Se-Jin Oh, Duk-Gyoo-Roh, Dong-Kyu Jung, Chungsik Oh, Hyo-Ryoung Kim,

e-VLBI Deployments with Research Internet

MB-NG Review High Performance Network Demonstration 21 April 2004

Achieving reliable high performance in LFNs (long-fat networks)

Presentation transcript:

Masaki Hirabaru 情報処理学会四国支部講演会 December 17, 2004 インターネットでの長距離高性能データ転送

2 データ転送昔と今パケット転送遅延＝サイズ速度距離光速＋パケットサイズ = 1500 バイト距離 = 10,000Km 1) 通信速度 = 9600bps 遅延 = 156ms + 33ms 2) 通信速度 = 1Gbps 遅延 = 15us + 33ms ネットワークの帯域が広くなったのに、スループットが出ない！

3 Radio Telescopes NICT Kashima Space Center Onsala Space Observatory 20m 34m 18m Urumqi 25m

4 VLBI* System Transitions K5 Data Acquisition Terminal 1st Generation 2nd Generation 1983~ Open-Reel Tape Hardware Correlator 1990~ Cassette Tape Hardware Correlator e-VLBI over ATM 3rd Generation 2002~ PC-based System Hard-disk Storage Software Correlator e-VLBI over Internet K3 Correlator (Center) K3 Recorder (Right) K4 Terminal K4 Correlator 64Mbps 256Mbps 1 ～ 2Gbps * 超長基線電波干渉計

5 VLBI (Very Long Baseline Interferometry) delay radio signal from a star correlator A/D clock A/D Internet clock e-VLBI geographically distributed observation, interconnecting radio antennas over the world Large Bandwidth-Delay Product Network issue ASTRONOMY GEODESY ~Gbps Gigabit / real-time VLBI multi-gigabit rate sampling A B A B d

6 Motivations MIT Haystack – CRL Kashima e-VLBI Experiment on August 27, 2003 to measure UT1-UTC in 24 hours –41.54 GB CRL => MIT 107 Mbps (~50 mins) GB MIT => CRL 44.6 Mbps (~120 mins) –RTT ~220 ms, UDP throughput Mbps However TCP ~6-8 Mbps (per session, tuned) –BBFTP with 5 x 10 TCP sessions to gain performance HUT – CRL Kashima Gigabit VLBI Experiment -RTT ~325 ms, UDP throughput ~70 Mbps However TCP ~2 Mbps (as is), ~10 Mbps (tuned) -Netants (5 TCP sessions with ftp stream restart extension) They need high-speed / real-time / reliable / long-haul high-performance, huge data transfer. UT1-UTC = / usec

7 TCP Dynamic Behavior time rate slow-start congestion avoidance available bandwidth

8 Example: From Tokyo to Boston TCP on a fast long path with a bottleneck bottleneck overflow queue loss Tokyo sender rate control Boston receiver loss detection feedback It takes 150ms to know the loss (buffer overflow). It keeps overflowing during the period… 150ms is very long for the high-speed network. 150ms at 1Gbps generates ~19MByte on the wire. Los Angeles 50ms 100ms bw 1G bw 0.8G Buffer 25MB

9 Conditions a signle TCP stream (x multi) memory to memory (no disk access) single bottleneck keep end-to-end principle (x relay) packet switched network (scalable) consume all the available bandwidth Target

10 Example How much speed can we get? Receiver Sender a-2) High- Speed Backbone L2/L3 SW GbE 100M RTT 200ms a-1) Receiver Sender High- Speed Backbone SW GbE 100M RTT 200ms GbE SW GbE 100M

11 Average TCP Throughput less than 20Mbps In case we limit the sending rate at 100Mbps This is TCP’s fundamental behavior.

12 Possible Bottlenecks CPU IO NIC Memory PCI, PCI-X Disk driver buffer Interrupt coalesce, delay MTU etc. 1 st Step: Tuning a Host with UDP Iperf theoretical UDP throughput 957 Mbps (IPv4)

13 2 nd Step: Tuning a Host with TCP Maximum socket buffer size (TCP window size) –net.core.wmem_max net.core.rmem_max (64MB) –net.ipv4.tcp_wmem net.tcp4.tcp_rmem (64MB) Driver descriptor length –e1000: TxDescriptors=1024 RxDescriptors=256 (default) Interface queue length –txqueuelen=100 (default) –net.core.netdev_max_backlog=300 (default) Interface queue descriptor –fifo (default) MTU –mtu=1500 (IP MTU) Linux (RedHat 9) with web100 Web100 (incl. High Speed TCP) –net.ipv4.web100_no_metric_save=1 (do not store TCP metrics in the route cache) –net.ipv4.WAD_IFQ=1 (do not send a congestion signal on buffer full) –net.ipv4.web100_rbufmode=0 net.ipv4.web100_sbufmode=0 (disable auto tuning) –net.ipv4.WAD_FloydAIMD=1 (HighSpeed TCP) –net.ipv4.web100_default_wscale=7 (default)

14 Kwangju Busan 2.5G Fukuoka Korea 2.5G SONET KOREN Taegu Daejon 10G 0.6G1Gx2 QGPOP Seoul XP Genkai XP Kitakyushu Tokyo XP Kashima 0.1G Fukuoka Japan 250km 1,000km 2.5G TransPAC 9,000km 4,000km Los Angeles Cicago New York MIT Haystack HUT 10G 1G APII/JGN Abilene 0.1G Helsinki 2.4G Stockholm 0.6G 2.4G GEANT Nordunet funet Koganei 1G 7,000km Indianapolis I2 Venue 1G 10G 100km server (general) server (e-VLBI) Abilene Observatory: servers at each NOC CMM: common measurement machines Network Diagram for TransPAC/I2 Measurement (Oct. 2003) 1G x2 sender receiver Mark5 Linux (RH 7.1) P3 1.3GHz Memory 256MB GbE SK-9843 PE1650 Linux (RH 9) Xeon 1.4GHz Memory 1GB GbE Intel Pro/1000 XT Iperf UDP ~900Mbps (no loss)

15 TransPAC/I2 #1:Reno (Win 64MB)

16 Analyzing Advanced TCP Dynamic Behavior in a Real Network ( Example: From Tokyo to Indianapolis at 1G bps with HighSpeed TCP ) The data was obtained during e-VLBI demonstration at Internet2 Member Meeting in October 2003.

17 Receiver Linux TCP Sender Linux 2.4 ENP2611 Network Processor Emulator GbE RTT 200ms (100ms one-way) GbE Only 800 Mbps available Replaying in a Laboratory -Evaluation of Advanced TCPs- ENP2611 board

18 Test Result #1 Queue size 100 packets TCP NewReno (Linux) HighSpeed TCP (Web100)

19 Example of Advanced TCPs with different bottleneck queue sizes BIC TCP queue size 100 packetsFAST TCP queue size 100 packets BIC TCP queue size 1000 packetsFAST TCP queue size 1000 packets BIC TCP: FAST TCP: TCP Vegas を基にした Delay Base の TCP

20 RouterSwitchRouterSwitch 1Gbps 100Mbps1Gbps a)b) Device Queuing Delay (µs) Capacity (Mbps) Estimated Queue Size (1500B) Switch A *50 Switch B *180 Switch C *169 Switch D Switch E Router F Router G * set to 100M for measurement Measuring Bottleneck Queue Sizes Switch/Router Queue Size Measurement Result Typical Bottleneck Cases ReceiverSender Capacit y C packet train lost packet measured packet Queue Size = C x (Delay max – Delay min )

21 Kwangju Busan 2.5G Fukuoka Korea 2.5G SONET KOREN Taegu Daejon 10G 1G (10G) 1G Seoul XP Genkai XP Kitakyushu Kashima 1G (10G) Fukuoka Japan 250km 1,000km 2.5G JGN II 9,000km 4,000km Los Angeles Chicago Washington DC MIT Haystack 10G 2.4G APII/JGNII Abilene Koganei 1G(10G) Indianapolis 100km bwctl server Experiment for High-Performance Scientific Data Transfer 10G Tokyo XP / JGN II I-NOC *Performance Measurement Point Directory perf server e-vlbi server JGNII 10G GEANT SWITCH 7,000km TransPAC Pittsburgh U of Tokyo A BWCTL account available for CMM including Korean researchers International collaboration to support for science applications

22 VLBI Antenna Locations in North-East Asia Shintotsukawa 3.8m Tomakomai 11m, FTTH (100M) 70km from Sapporo Mizusawa 10m 20m 118km from Sendai Tsukuba 32m, OC48/ATMx2 SuperSINET Kashima 34m, 1Gx2 JGN, OC48/ATM Galaxy Yamaguchi 32m 1G, 75M SINET Gifu 11m 3m, OC48/ATMx2 SuperSINET Usuda 64m, OC48/ATM Galaxy Nobeyama 45m OC48/ATM Galaxy Nanshan (Urumqi) 25m 70km from Urumqi Koganei 34m, 1Gx2 JGN, OC48/ATM Galaxy Miyun (Beijing) 50m 50km from Beijing 2Mbps Yunnan (Kunming) 3m (40m) 10km from Kunming Sheshan (Shanghai) 25m 30km from Shanghai Observatory is on CSTNET at 100M Jeju 20m Tamna U Seoul 20m Yonsei U Ulsan 20m U Ulsan Daejon 14m Taeduk Ishigaki 20m Ogasawara 20m Chichijima 10m Iriki 20m Kagoshima 6m Aira 10m Legend connected not yet connected antenna under construction

23 Speed Races Internet2 Land Speed Record –Single Regular TCP stream –Speed x Distance –6.57 Gbps by Caltech (7.21 Gbps by U Tokyo) SC Bandwidth Challenge –More than 100 Gbps –AIST, U Tokyo, JAXA, …

24 Summary and Future Work High-performance scientific data transfer faces on network issues we need to work out. Big science applications like e-VLBI and High- Energy Physics need cooperation with network researchers. Deployment of performance measurement Infrastructure is on-going on world-wide basis.