1. 1 Masaki Hirabaru and Yasuhiro Koyama {masaki,koyama}@nict.go.jp APEC-TEL APGrid Workshop September 6, 2005 e-VLBI: Science over High-Performance Networks

2. 2 Radio Telescopes NICT Kashima Space Center 34m Onsala Space Observatory 20m (left) Perks 64m (right) Australia Telescope National Facility MIT Haystack 18m Shanghai 25m

3. 3 Geophysics and Plate Tectonics VLBI Applications -63.5  0.5 mm/year -46.1  0.3 mm/year 1.3  0.5 mm/year Kauai Fairbanks Kashima Kashima-Kauai Baseline Length Fairbanks-Kauai Baseline Length Kashima-Fairbanks Baseline Length

4. 4 VLBI Applications (2) Halca （ Muses-B ） NGC4261 Radio Telescope Satellite ‘Halca’ and its images Earth Orientation Parameters Radio Astronomy : High Resolution Imaging, Astro-dynamics Reference Frame : Celestial / Terrestrial Reference Frame Earth Orientation Parameters, Dynamics of Earth’s Inner Core

5. 5 VLBI (Very Long Baseline Interferometry) e-VLBI geographically distributed observation, interconnecting radio antennas over the world ASTRONOMY GEODESY Gigabit / real-time VLBI multi-gigabit rate sampling delay radio signal from a star correlator A/D clock A/D Internet clock ~Gbps A B A B d Large Bandwidth-Delay Product Network issue

6. 6 VLBI System Transitions K5 Data Acquisition Terminal 1st Generation 2nd Generation 1983~ Open-Reel Tape Hardware Correlator 1990~ Cassette Tape Hardware Correlator e-VLBI over ATM 3rd Generation 2002~ PC-based System Hard-disk Storage Software Correlator e-VLBI over Internet K3 Correlator (Center) K3 Recorder (Right) K4 Terminal K4 Correlator 64Mbps 256Mbps 1 ～ 2Gbps

7. 7 Recent e-VLBI System Developments K5 by NICT ADS1000 (1024Msample/sec 1ch 1bit or 2bits) ADS2000 (64Msample/ch·sec, 16ch, 1bit or 2bits) IP-VLBI Board (~16Msample/ch·sec, ~4ch, ~8bits) PC : Data Acquisition Correlation VSI Correlator other DAS Internet PC-VSI Board (Supports VSI-H specifications) VSI VSI-E RTP/RTCP

8. 8 Motivations MIT Haystack – NICT Kashima e-VLBI Experiment on August 27, 2003 to measure UT1-UTC in 24 hours –41.54 GB NICT → MIT 107 Mbps (~50 mins) 41.54 GB MIT → NICT 44.6 Mbps (~120 mins) –RTT ~220 ms, UDP throughput 300-400 Mbps However TCP ~6-8 Mbps (per session, tuned) –BBFTP with 5 x 10 TCP sessions to gain performance HUT – NICT Kashima Gigabit VLBI Experiment -RTT ~325 ms, UDP throughput ~70 Mbps However TCP ~2 Mbps (as is), ~10 Mbps (tuned) -Netants (5 TCP sessions with ftp stream restart extension) There was bandwidth available but we could not utilize.

9. 9 Observing Bandwidth  Data rate  (Precision of Time Delay) -1  (SNR) 1/2 Wave Length / Baseline Length  Angular Resolution Baseline Length  (EOP Precision) -1 VLBI - Characteristics Faster Data Rate = Higher Sensitivity Longer Distance = Better Resolution

10. 10 Long Distant Rover Control (at least) 7 minutes one way delay Image Command Earth Mars When operator saw collision, it was too late.

11. 11 Long-Distance End-to-End Congestion Control Merge (Bottleneck) A+B > C Overflow Sender (JP) Receiver (US) Feedback BWDP: Amount of data sent but not yet acknowledged 64Kbps x 200ms = 1600B ~ 1 Packet 1Gbps x 200ms = 25MB ~ 16700 Packets 200ms round trip delay A B C

12. 12 Average TCP Throughput less than 20Mbps Q=50 Example How much speed can we get? Receiver Sender High- Speed Backbone L2/L3 SW 1G 100M Delay at light speed: 100ms 1G

13. 13 Analyzing Advanced TCP Dynamic Behavior in a Real Network ( Example: From Tokyo to Indianapolis at 1G bps with HighSpeed TCP ) The data was obtained during e-VLBI demonstration at Internet2 Member Meeting in October 2003. Throughput RTT Window Sizes Packet Losses

14. 14 Kwangju Busan 2.5G Fukuoka Korea 2.5G SONET KOREN Taegu Daejon 10G 1G (10G) 1G Seoul XP Genkai XP Kitakyushu Kashima 1G (10G) Fukuoka Japan 250km 1,000km 10G JGN II 9,000km 4,000km Los Angeles Chicago Washington DC MIT Haystack 10G 2.4G APII/JGN II Abilene Koganei 1G(10G) Indianapolis 100km bwctl server Performance Measurement Platform for High-Performance Scientific Data Transfer 10G Tokyo XP / JGN II I-NOC *Performance Measurement Point Directory http://e2epi.internet2.edu/pipes/pmp/pmp-dir.html perf server e-vlbi server JGNII 10G GEANT SWITCH 7,000km TransPAC Pittsburgh U of Tokyo Locate the problem International collaboration to support for science applications U. Hawaii

15. 15 Solutions by Advanced TCPs Loss-Based ► AQM (Advanced Queue Management) Reno, Scalable, High-Speed, BIC, … Delay-Based Vegas, FAST Explicit Router Notification ECN, XCP, Quick Start, SIRENS, MaxNet How can wee foresee collision (queue overflow)?

16. 16 TCP Performance with Different Queue Sizes

17. 17 * set to 100M for measurement Measuring Bottleneck Queue Sizes Switch / Router Queue Size Measurement Result ReceiverSender Capacit y C packet train lost packet measured packet Queue Size = C x (Delay max – Delay min ) Device Queuing Delay (µs) Capacity (Mbps) Estimated Queue Size (1500B) Switch A6161100*50p/75KB Switch B22168100*180p/270KB Switch C20847100*169p/254KB Switch D738100060p/90KB Switch E36621000298p/447KB Router F148463100012081p/18MB Router G188627100015350p/23MB cross traffic injected for measurement

18. 18 RouterSwitch 1Gbps (10G) 100Mbps (1G) b-1) Typical Bottleneck Cases RouterSwitch a) Queue ~100 Queue ~1000 VLANs Switch/ Router 10G LAN-PHY Ethernet Untag b-2) 9.5G WAN-PHY 802.1q Tag

19. 19 e-VLBI Demonstration in JGN II Osaka (Jan. 2005)  e-VLBI data transfer achieved ~700Mbps from Haystack to Osaka ~900Mbps from Kashima to Osaka  Software Cross Correlation ~240Mbps per station Dr. Koyama 4 Apple G5 Server machines (8 CPUs in Total) Osaka #7,#8 1G Raid Disks 1G x4 Raid Disks #5,#6 #1,#2 #3,#4 CPU x8 10G Tokyo NICT Kashima Abilene (10G) MIT Haystack CHI WAS JGN II Int’l (10G) 1G /10G 1G/2.5G *TCP parameters were tuned for the path.

20. 20 VLBI Antenna Locations in North-East Asia Shintotsukawa 3.8m Tomakomai 11m, FTTH (100M) 70km from Sapporo Mizusawa 10m 20m 118km from Sendai Tsukuba 32m, OC48/ATMx2 SuperSINET Kashima 34m, 1Gx2 JGN II, OC48/ATM Galaxy Yamaguchi 32m 1G, 75M SINET Gifu 11m 3m, OC48/ATMx2 SuperSINET Usuda 64m, OC48/ATM Galaxy Nobeyama 45m OC48/ATM Galaxy Nanshan (Urumqi) 25m 70km from Urumqi Koganei 34m, 1Gx2 JGN II, OC48/ATM Galaxy Miyun (Beijing) 50m 50km from Beijing 2Mbps Yunnan (Kunming) 3m (40m) 10km from Kunming Sheshan (Shanghai) 25m 30km from Shanghai Observatory is on CSTNET at 100M Jeju 20m Tamna U Seoul 20m Yonsei U Ulsan 20m U Ulsan Daejon 14m Taeduk Ishigaki 20m Ogasawara 20m Chichijima 10m Iriki 20m Kagoshima 6m Aira 10m Legend connected not yet connected antenna under construction

21. 21 e-VLBI Data Transfer Real-time e-VLBI – flat-rate live data transfer Internet Synchronize Correlation Common e-VLBI – file transfer Carry the disk to the nearest station to put on-line Correlate among many combinations concurrently to get more precise data (like a virtual huge antenna) Future – e-VLBI infrastructure multicast and automated

22. 22 Summary High-performance scientific data transfer faces on network issues we need to work out. Big science applications like e-VLBI and High- Energy Physics need cooperation with network and Grid researchers. Deployment of performance measurement Infrastructure over research networks is on- going on world-wide basis.