1. HPIIS Performance Review
October 25, 2000
Global Scale Tele-Immersion Network Performance Activities Jason Leigh, Oliver Yu, Linda Winkler, Alan Verlo, Tom DeFanti Yong-joo Cho, Ray Fang, Javier Girado, Liujia Hu, Tomoko Imai, Naveen Krishnaprasad, Michael Lewis, Ya Ju Lin, Dave Pape, Kyoung Park, Chris Scharver, Brenda Silva, Liang Wang Josh Eliason, Jinghua Ge, Eric He, Atul Nayak, Shalini Venkatamaran
Global-Scale Tele-Immersion Network Performance Presentation

2. Common Characteristics of Teleimmersive Applications

3. Characterization of Tele-Immersive Streams
Estimated bandwidth
(bits/s)
DiffServ
Types
Burstiness
Latency sensitive
Jitter sensitive
Error sensitive
UDP avatar
6K x n
(15fps)
Interactive Real-time
Constant Y N
UDP audio stream
64K x n
Brief
UDP video stream
10M
(2-way only) YN
UDP stream
With Playback
depends
Non-interactive Real-time
TCP control data
7K x n
Reliable
TCP bulk data
Best Effort or Deadline Delivery
Sustained burst

4. Network Research

5. Maximizing Bandwidth Utilization over Long Fat Networks
Even if QoS via DiffServ or IntServ is available, it still does not solve the Long Fat Network problem
Problem is small TCP window sizes (well known problem but still no widely accepted solution)
On SGI’s change in window size requires kernel rebuild
Size of window should be set to current available BW of the network
CAVERNsoft’s Parallel Socket Striping works well but is considered “irresponsible” use of networks

6. 64K Window Size Amsterdam to Chicago
Bursty as max bw reached
but performance is still good

7. 64K Window Size CERN to EVL
Bursty as max bw reached
but performance still good

8. Window Size: EVL = 1.85M, SARA = 64K
EVL to SARA
When window size is large enough
no real benefit to using parallel sockets
Sending client determines the window size
SARA to EVL

9. Window size: EVL = 1.85M, CERN = 640K
EVL to CERN
Similar story
at CERN
CERN to EVL

10. Anomalies Theoretical BW from EVL to SARA is 100Mbps
Netperf UDP shows reasonable performance:
EVL to SARA 85Mbps
SARA to EVL 65Mbps (5 more hops via Abilene)
Netperf and Parallel sockets TCP shows only:
30Mbps
Perhaps due to asymmetric tcp window size settings?
Argument for UDP-based schemes? E.g. Forward Error Correction

11. Forward Error Correction scheme for low-latency delivery of error sensitive data
Transmit error correction data over high bandwidth networks that can be used for correcting UDP streams to achieve lower latency than TCP but higher reliability.
Transmit error correction data to improve quality of streamed video by correcting for lost packets.
Not intended for bulk data transfer but in light of TCP results this might hold some promise.

12. FEC Experiments EVL to SARA- Amsterdam (45Mb/s 100ms RT latency)
Broader Ques:
Can FEC provide a benefit? How much?
Tradeoff between redundancy and benefit?
Specific Ques:
TCP vs UDP vs FEC/UDP
How much jitter does FEC introduce?
High thru put UDP vs FEC/UDP to observe loss & recovery

13. ` goal FEC greatest benefit is in small packets.
Larger packets impose greater overhead.
As redundancy decreases FEC approaches UDP.

14. G o a l

15. Packet Loss over UDP vs FEC/UDP between Chicago & Amsterdam
HPIIS Performance Review
Packet Loss over UDP vs FEC/UDP between Chicago & Amsterdam
October 25, 2000
UDP
UDP
FEC
Data Rate
(bits/s)
Packet Size
(Bytes)
Packet Loss Rate in UDP (%)
Packet Loss Rate in FEC over UDP (%) 1M
128
0.4
256
0.2
1024
10M 30 4 25 3 21
1.5
Ncftp
Put Parallel ftp in linux
Global-Scale Tele-Immersion Network Performance Presentation

16. Human Factors in Tele-Immersion

17. Collaborative Coordination Experiments between Chicago and Singapore
CAVE to CAVE (STAR TAP)
Audio via Phone call
Scramnet (adjustable latency, 0 jitter)
LAN Ethernet (~ 10ms)
Local ISDN (~ 200ms)
STAR TAP (~ 250ms)
Predict STAR TAP similar to performance over ISDN

18. Collaborative Coordination Experiments between Chicago and Singapore
HPIIS Performance Review
Collaborative Coordination Experiments between Chicago and Singapore
October 25, 2000
200ms RTT is the threshold where performance begins to suffer
Roughly RTT to Asia. Results to Singapore similar to local ISDN
200ms RTT with 0 jitter is same as 10ms RTT with 7ms jitter
1. Determine that 200ms is the threshold where performance begins to suffer.
This is also the approx round trip time to Asia.
2. We noted that user performance to Singapore was about the same as for local ISDN.
3. We noted that 200ms RTT with 0 jitter is as bad as 10ms RTT with 7ms jitter => Jitter is the main culprit.
Now that we know this result we know what to ask for in the networks. The question then is how emerging QoS schemes can help.
Global-Scale Tele-Immersion Network Performance Presentation

19. HPIIS Performance Review
October 25, 2000
DiffServ Experiment 1
+ background
+ DiffServ
Bandwidth recovery good
EVL
fore
back
100Mbps
100Mbps x
80Mbps
25Mbps
Latency recovery good x
42Mbps
42Mbps
ATM link from EVL to ANL has 80Mbps. From ANL to EVL there is 25Mbps.
All routers have priority queuing.
EVL sends 25Mbps foreground to ANL. Then EVL injects additional 25Mbps background traffic.
Effect is shown in degradation in performance.
Then DiffServ activated. Result is bandwidth of foreground flow restored. Latency is also restored.
ANL x x
100Mbps
100Mbps
Small packet loss
Global-Scale Tele-Immersion Network Performance Presentation

20. HPIIS Performance Review
DiffServ Experiment 2
October 25, 2000
+ background
+ DiffServ
Bandwidth recovery good
EVL
fore
back
100Mbps
100Mbps x
80Mbps
25Mbps x
Latency recovery not good
42Mbps
42Mbps
Same experiment except traffic sent in opposite direction.
Note, the link between ANL and EVL is set to be limited at 25Mbps.
Again bandwidth recovery by foreground is good when DiffServ is used.
But latency recovery is not good. Latency is in fact 150ms which is very bad for the short distance from ANL to EVL. Recall that threshold of acceptable latency for coordinated activity is 100ms 1-way.
Also packet loss doubles.
Conclusion: DiffServ is good at bandwidth recovery. It is only good at latency recovery if network is not saturated.
ANL x x
100Mbps
100Mbps
Packet loss double
Global-Scale Tele-Immersion Network Performance Presentation

21. Application of Research Results
CAVERNsoft G2
applications at iGrid 2000 in Yokohama

22. Tele-Immersion Middleware The CAVERNsoft G2 Toolkit
G2 is C++ toolkit for building Tele-Immersive applications with special emphasis on networking
Networking:
UDP, TCP, Multicast, HTTP.
UDP reflector and multicast bridge.
TCP reflector.
Remote procedure calls.
32 and 64bit Remote file I/O.
Parallel 32 & 64 bit TCP socket striping for high throughput data delivery.
FEC library.
Client/Server distributed shared memory persistent database.
Threading, Mutual Exclusion.
Built-in Instrumentation of networking services.
QoS via GARA and MCSP underway.

23. Tele-Immersion Middleware The CAVERNsoft G2 Toolkit
Audio streaming.
Articulated Avatars.
VR navigation.
VR menus.
Speech recognition with IBM ViaVoice.
Collaborative application shell to jumpstart development.

24. TIDE Teleimmersive Data Explorer (TIDE)
In collaboration with National Center for Data Mining
General framework for collaborative visualization of massive data-sets
Current data-set is ozone data from NOAA

25. HPIIS Performance Review
CIBRView
October 25, 2000
Collaborative Image Based Rendering Viewer (CIBRview)
In collaboration with Wes Bethel and Steve Lau at Lawrence Berkeley Lab
Accesses volume data 512x256x256x 256 frames ~ 40Gig data-sets
Generates image slices that are distributed to collaborating clients.
Sent about 500 slices/files from Chicago to Japan
Click on the link to get to more detailed notes on CIBRView.
Global-Scale Tele-Immersion Network Performance Presentation

26. Virtual Harlem University of Missouri
Reconstruction of Harlem during the Harlem Renaissance

27. HPIIS Performance Review
October 25, 2000
Earthquake Hypocenters Space Physics & Aeronomy Research Collaboratory (U of Michigan) A demonstration at Telecom 2000 and SC 2000 between Israel, Dallas, Chicago, Michigan
Earthquake hypocenters from the last 20 years. They are only events between Magnitude 4 and 5.5.
Below 4 and they are too small to locate accurately, above 5.5 and the rupture zone is too large to locate
accurately.
Electron density isosurface of plasma in upper atmosphere
High energy solar photons (in the EUV range) hit neutrals in
the upper atmosphere, producing ion-electron pairs---plasma. these ions
and elestrons recombine into neutrals mostly though three body collisions.
That's why we have higher electron densities during the daytime than
during the nighttime. There is not much solar radiation available in the
nighttime to ionize the neutrals to counter the loss of plasma by
recombination.
In this virtual laboratory, space physicists study many phenomena, such as magnetic storms that originate on the sun.  Such storms send massive numbers of charged particles to Earth, where they interact with the magnetosphere to bring about the aurora borealis and aurora australis.  On a less pyrotechnic level, the arrival of so much energy in Earth's ionosphere can interfere with radio and television reception, disrupt electric power transmission, and threaten orbiting spacecraft and astronauts.
Global-Scale Tele-Immersion Network Performance Presentation

28. Network Visualization

29. QoS Internet Monitoring Tool QoSIMoto

30. STAR TAP Network Visualization

31. Future Work
DiffServ and RSVP from EVL to CERN in collaboration with NWU
Reliable UDP for high throughput bulk data transmission
Integrated Collaboratory for Analysing Networks (iCAN): iCAN-Monitor, iCAN-Visualize, iCAN-Manage, iCAN-Active Test, iCAN-Collaborate