1. N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r Roger Zimmermann Alexander A. Sawchuk, Cyrus Shahabi Ulrich Neumann, Chris Kyriakakis Tom Holman, Christos Papadopoulos Integrated Media Systems Center University of Southern California Los Angeles, CA 90089 http://dmrl.usc.edu and http://imsc.usc.edu RMI: Remote Media Immersion

2. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Outline IMSC Introduction RMI Goals and Challenges System Components Experiments Streaming Media Architecture: Yima Research Challenges Future Possibilities

3. Integrated Media Systems CenterUniversity of Southern California, Los Angeles IMSC ERC Research Structure

4. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Integrated Media Systems Education Entertainment Media Communications Information Management Sensory Interfaces Application Research Communication 3 Vision Areas Charter: Immersipresence User Centered Sciences Media Immersion Environment 6 Research Areas

5. Integrated Media Systems CenterUniversity of Southern California, Los Angeles What is the RMI? The goal of the Remote Media Immersion system is to build a testbed for the creation of immersive applications. Immersive application aspects: 1. Multi-model environment (aural, visual, haptic, …) 2. Shared space with virtual and real elements 3. High fidelity 4. Geographically distributed 5. Interactive

6. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Remote Media Immersion Goals Reproduce the complete audio and video ambience placing people in a virtual space nExperience events occurring at remote site(s) nNatural communication, interaction and collaboration Application scenarios: nUnidirectional off-line acquisition, processing and storage of immersidata and synchronized display (rendering) nReal-time, two-way version nStereoscopic visual display Immersed in a college football game Doctors assisting in a remote procedure Business people negotiating like they are in the same room Students visiting an aquarium a thousand miles away

7. Integrated Media Systems CenterUniversity of Southern California, Los Angeles RMI Challenges n Immersive, high-quality video acquisition and rendering n High Definition video 1080i and 720p (40 Mb/s) n Immersive, high-quality audio acquisition and rendering n 10.2 channels of uncompressed audio (12 Mb/s) n Storage and transmission of media streams across networks n Synchronization between streams (A/V, A/A, V/V)!

8. Integrated Media Systems CenterUniversity of Southern California, Los Angeles RMI Architecture

9. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Remote Media Immersion Client... HD Video 10.2 Ch. Audio HD Video Uncompressed Linear Time Code Word Clock 10.2 Ch. Audio (Digital) HD Video Rendering 10.2 Ch. Audio Rendering Network Yima Client SW

10. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Experimental Setup Synchronized combinations of Immersive audio and HDTV streamed playback from Yima nStreaming of 16 channels of immersive audio, uncompressed at 12 Mb/s nStreaming of 1920x1080i HDTV content, MPEG-2 compressed at 40 Mb/s IMSC ISI East

11. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Client (Receiver) Setup

12. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Internet2 Fall 02 Member Meeting Video: HDTV 1280x720p Audio: 10.2 channel, immersive sound system New World Symphony, Miami, FL

13. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Internet2 Demonstration

14. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Storage, Streaming & Rendering n Server n Storage n Scheduling n Scalability n Clients n Multi-stream Synchro- nized playback n Transmission n Robust VBR Flow control n Requirements: A streaming platform that can scale and handle synchronized, high-bandwidth streams. n Focus: End-to-end streaming architecture.

15. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Yima Architecture n Multi-node, multi- disk architecture n Scalable n Industry-standard network protocols: n RTP, RTSP n Robust media transmission: n Adaptive flow control n Selective retransmission n Clients: n Multi-stream synch.

16. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Research Focus and Unique Approaches Scalability (enable large scale systems) nMulti-node architecture with distributed scheduler and distributed file system on commodity PCs [Computer 02] nIncremental system growth: SCADDAR – an efficient randomized technique to reorganize continuous media blocks [ICDE 2002] Robust stream delivery nMulti-threshold flow control between clients and server: avoids data starvation and overflow, supports variable bit rate media [MTAP 2003?] nSelective packet retransmission protocol [NOSSDAV96, MMCN03]

17. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Challenge: Real-Time Media Bandwidth requirements for different media types: 1 Mb/s 4-6 Mb/s 31 Mb/s 50 Mb/s 20 Mb/s 100 Mb/s

18. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Yima Server S/W Architecture

19. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Challenge: Scalability n As continuous media (CM) repositories increase, the need for larger storage capacity arises 1. Multi-node, multi-disk support 2. Disk scaling (adding new and/or removing old disks) SCADDAR n Quick access to data n Online 24/7 operation (i.e. no downtime) n Fault-tolerance n Load balancing of the data before/after scaling to ensure maximum utilization of disk I/O and capacity

20. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Scalability: Multi-Node, Multi-Disk Yima-1 Yima-2 n Data and control network traffic can be routed with different logical topologies n Yima-1: single data path (high inter-node traffic) n Yima-2: multiple data paths (low inter-node traffic)

21. Integrated Media Systems CenterUniversity of Southern California, Los Angeles SCADDAR n Disk scaling (adding new and/or removing old disks) n Example of adding one disk to an existing 4-disk storage system:

22. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Challenge: Robust Stream Delivery n Variable bit rate (VBR) media encoders allocate more bits to complex scenes and less bits to simple ones n Smoothing of VBR media traffic has the following quality benefits: n Better resource utilization (less bursty) n More streams with the same network capacity n Multi-Threshold Flow Control (MTFC) algorithm objectives: n Online operation n Content independence n Minimizing feedback control signaling n Rate smoothing Motivation & Objectives

23. Integrated Media Systems CenterUniversity of Southern California, Los Angeles MTFC Buffer Management Multiple Thresholds: goal is middle of buffer Send rate adjust command to server whenever threshold is crossed

24. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Multi-Threshold Flow Control Results

25. Integrated Media Systems CenterUniversity of Southern California, Los Angeles –IP networks are based on best effort delivery –Client at USC, LA, server at ISI East, Arlington, VA –One aspect: high-bandwidth video and audio transmissions HDTV @ 40-45 Mb/s 16-channels of uncompressed PCM audio @ 11-22 Mb/s –RTP/UDP is industry standard, but UDP loss creates problems Tests on high-performance network: Internet2 and DARPA NGI SuperNet (WAN) & Gigabit Ethernet (LAN) In the order of 10 packets lost every 1 million (10 -5 ) Such low loss is still visible/audible! Loss may result in synchronization problems Challenge: Packet Loss!

26. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Solution Space –No silver bullet (one size fits all) We chose selective retransmissions FEC Fast Sacrifices BW for reliability Vulnerable to burst loss FEC Fast Sacrifices BW for reliability Vulnerable to burst loss Retransmission One RTT to recover Optimal use of BW Retransmission One RTT to recover Optimal use of BW Concealment Fast Optimal use of BW Quality may suffer Media dependent Concealment Fast Optimal use of BW Quality may suffer Media dependent Tradeoffs Reliability vs. BW Reliability vs. Latency Tradeoffs Reliability vs. BW Reliability vs. Latency

27. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Selective Retransmissions –Originally proposed in [NOSSDAV96] by Papadopolous and Parulkar –Need: sender retransmission buffer and receiver playout buffer Receiver-driven operation: Ask for retransmissions only missing data will be consumed after estimated RTT

28. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Fast Error Recovery Sender read frame every T secs 111101 Play-out buffer send new frame every T secs Receiver discard kjikji ijkijk Retransmit buffer 110111 101011 New frame NAK

29. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Multi-node Server Centralized Design Bipartite Design –Originally Data and control network traffic can be routed with different logical topologies Centralized: single data path (high inter-node traffic) Bipartite: multiple data paths (low inter-node traffic)

30. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Multi-node Server: RBEC –To improve scalability and online data reorganization data blocks are randomly assigned to server nodes. Challenge: If a packet does not arrive at the client side, how does the client know which node attempted to send it? Possible solutions: 1.Broadcast retransmission requests 2.Compute which node should have the data 3.Introduce node-specific local sequence numbers (LSN) in addition to a global sequence number (GSN)

31. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Yima Approach 33 GSNLSN Payload 22 GSNLSN Payload 11 GSNLSN Payload 36 GSNLSN Payload 25 GSNLSN Payload 14 GSNLSN Payload 69 GSNLSN Payload 58 GSNLSN Payload 47 GSNLSN Payload Server Node 1: Server Node 2: Server Node 1: Assumption: 3 packets per storage block

32. Integrated Media Systems CenterUniversity of Southern California, Los Angeles LSN Retransmission Operation

33. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Experiments –Multi-node server with 1, 2 and 4 nodes –LAN and WAN: DARPA NGI SuperNet, cross- continental link (4000km) –Gilbert loss model:and with p = 0.0192 and q = 0.8454, therefore P loss is approx. 2.2% –Media file Twister (MPEG-2) avg. BW of 698 kB/s length 25 minutes throughput std. dev. 308283

34. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Results for LAN 4% 2% 0% 3% 1% 4% 2% 0% 3% 1% 4% 2% 0% 3% 1% 4% 2% 0% 3% 1% Raw Loss1 Node 2 Nodes4 Nodes 25 Minutes

35. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Results for WAN 4% 2% 0% 3% 1% 4% 2% 0% 3% 1% 4% 2% 0% 3% 1% 4% 2% 0% 3% 1% 25 Minutes Raw Loss1 Node 2 Nodes 4 Nodes Natural Losses

36. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Result Summary –LAN loss reduction: 1 Node: from 2.2% to 0.0582% 2 Nodes: from 2.2% to 0.0623% 4 Nodes: from 2.2% to 0.162% –WAN loss reduction: 1 Node: from 2.2% to 0.0556% 2 Nodes: from 2.2% to 0.0859% 4 Nodes: from 2.2% to 0.0429% –> One order of magnitude improvement

37. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Yima Client Features Synchronization between multiple clients nCoarse-grained via flow & rate control nFine-grained via hardware support (30 fps & 48,000 s/sec) nMedia streams can come from different physical locations

38. Integrated Media Systems CenterUniversity of Southern California, Los Angeles RMI & Yima Accomplishments Publications: nSCADDAR ICDE, March 2002 nIEEE Computer, June 2002 nGMeN IEEE TPDS, June 2002 RMI Transcontinential Tests: nServer at ISI East, Arlington, VA nInternet2 Fall02 Meeting RMI Press Coverage: nNew York Times, May 9, 2002 nNBC-4, May 9, 2002 nKTLA-5, May 9, 2002 COMPUTER

39. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Future Possibilities Distributed virtual social eventsImmersive gaming Large screen displays Multiple cameras and microphones; 3-D scene description Speech and gesture extraction Face and body tracking Wireless glasses or head-mounted displays Stereo display without glasses (autostereoscopic)

40. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Distributed Immersive Performance Outgrowth of Remote Media Immersion (RMI) –Create seamless immersive environment for distributed musicians, conductor (active) and audience (passive) –Compelling relevance for any human interaction scenario: education, journalism, communications Scenario: –Orchestra not available in town –Famous soloist cannot fit travel into schedule –Multiple soloists in different places

41. Integrated Media Systems CenterUniversity of Southern California, Los Angeles

45. 30 ms 20 ms 30 ms 10 ms 40 ms 60 ms Challenge: network latency

46. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Technical Challenges processing power latency (delay) compression data rates, error characteristics multi-stream synchronization

47. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Yima Ongoing Work Real-time recording of multiple streams –For example, from a panoramic camera with 5 individual camera heads: –Streams need to recorded in sync and played back in sync Statistical admission control algorithm for better utilization of the storage system Interactive, live streaming

48. Integrated Media Systems CenterUniversity of Southern California, Los Angeles Thank You! Questions? More info at: –Data Management Research Lab http://dmrl.usc.edu –Integrated Media Systems Center http://imsc.usc.edu Acknowledgments: –Kun Fu, Didi Shu-Yuen Yao, Beomjoo Seo, Shihua Liu, Mehrdad Jahangiri, Farnoush Banaei-Kashani, Nitin Nahata, Sahitya Gupta, Vasan N. Sundar, Rishi Sinha, Hong Zhu