1. ANET Workshop History of Internet Audio Experiments at McGill (and elsewhere) Jeremy R. Cooperstock Centre for Intelligent Machines Centre for Interdisciplinary Research in Music Media and Technology McGill University

2. ANET Workshop Software available from: http://ultravideo.mcgill.edu Research supported by:

3. ANET Workshop Some Early (Pre-)History Eve Schooler: Distributed Music: A Foray into Net Performance (Sept. 1993)  synchronized three real-time streams from different hosts; delays in the order of 200 ms  difficult for performers to be listeners Paul Hoffert: CyberSoiree (Feb. 1996)  ATM-based technology for audio and video streaming of a four-way jazz performance  delay >0.5s delay but musicians earned to compensate through extensive practice Dimitri Konstantas: Distributed Musical Rehearsal Studio (May 1996)  ATM based distributed rehearsal with conductor at different location from musicians  80 ms one-way delay for audio-video synch; echo resulted in "extreme confusion" Seiji Ozawa: Opening Ceremony Nagano Winter Olympics (1998)  conduct choruses on 5 continents: singers in Sydney, New York, Beijing, Berlin, False Bay  time lag adjustor used to eliminate satellite delay

4. ANET Workshop World's first Internet AC-3 stream Montreal-New York for the AES 107th Convention, Sept. 26, 1999

5. ANET Workshop System Diagram 1.5 Mbps SGI Indy PC Client Cisco IP/TV PC ServerSGI Indy Network Dolby encoder AC-3 (640 Kbps) (5.1 channels,16 bits @ 48kHz) coded in AES/EBU McGill University Montreal, Canada NYU Cantor Hall New York, USA

6. ANET Workshop In action Receiver Transmit Receive,check, playback Send Audio Queue Sender Receive Audio Queue Read

7. ANET Workshop Recording Studio that Spanned a Continent Montreal-Los Angeles for the AES 109th Convention, Sept. 23, 2000 McGill Redpath Hall, Montreal USC Norris Theatre, LA 12 channels, 96kHz, 24 bits/channel

8. ANET Workshop Microphone Configuration

9. ANET Workshop 3 Mbps

10. ANET Workshop 27.6 Mbps

11. ANET Workshop Some Modern History Internet2 Initiative: World's First Remote Barbershop Quartet (Nov. 2000)  multi-location quartet; each of the 4 singers in different cities, conductor in 5th  network delay variances prevented singers from hearing each other or conductor Internet2 Initiative: Music Video Recording via Internet2 (Nov. 2000)  multi-location music video recording session using real-time streaming video  musicians simultaneously connected via timing tracks to a mixing board Chris Chafe: QoS Enabled Audio Teleportation (Nov. 2000)  CD quality sound (750 kbps) of 2 separated musicians in Dallas streamed to Stanford  musicians played "together" in same space (Stanford) but delay was severe John Wawrzynek: Network Musical Performance (May 2001)  gestural coding (e.g. MIDI) used to manage data for distributed musical performance  musicians at Berkeley and CalTech, playing on MIDI keyboards; local feedback only

12. ANET Workshop "Frères Jacques" experiment Montreal, RISQ 2000 Conference "low-latency" MPEG-2 videoconferencing

13. ANET Workshop "Playing Together" experiment New York-Ottawa, Remote Masters class, Dec. 8, 2000 used Litton MPEG-2 codec @ 10 Mbps Pinchas Zuckerman, Columbia U, NYC Wu Ji, Canarie ARDNOC, Ottawa

14. ANET Workshop Quality vs. Latency  near-equivalent quality of encoded video at greatly reduced bandwidth  but... compression takes time

15. ANET Workshop Tolerated Synchronicity compress transmit 10 0 20 40 30 50 70 60 80ms speed of light: Mtl-LA net latency Mtl-LA networks small ensemble large ensemble lip synch musical activity * This figure is a theoretical “best-case” scenario. Actual performance may vary based on local hardware. The manufacturer makes no warranties, either express or implied, and will not be held liable for damage resulting from use of the product. Certain restrictions may apply. Offer void where prohibited by law. Batteries not included. MPEG* acquire

16. Latency: The Interaction-Killer or put another way...

17. ANET Workshop Interactive Success! Chris Chafe: SoundWIRE (August 2001)  streaming pro audio from remote locations for musical collaboration  UDP/IP streaming: good results between Armonk, NY and Stanford  round trip delay 125ms, musicians able to "catch-up" during pauses

18. ANET Workshop The future according to Sympatico?

19. ANET Workshop For those with DSL at home... yeah, right! (tu n'es pas sérieux!)

20. ANET Workshop So what does it take?  much bigger network pipes ("abusive bandwidth")  need Fast Ether (100Mbps) for audio  GigE (1000Mbps) for video  more efficient network topologies – hops add delay

21. ANET Workshop So what does it take?  much bigger network pipes ("abusive bandwidth")  need Fast Ether (100Mbps) for audio  GigE (1000Mbps) for video  more efficient network topologies – hops add delay  more powerful hardware  CPU speed (fast process switching, especially for mixed media)  pro-audio interfaces with small buffers

22. ANET Workshop So what does it take?  much bigger network pipes ("abusive bandwidth")  need Fast Ether (100Mbps) for audio  GigE (1000Mbps) for video  more efficient network topologies – hops add delay  more powerful hardware  CPU speed (fast process switching, especially for mixed media)  pro-audio interfaces with small buffers  more masochistic researchers

23. ANET Workshop Sometimes things work...

24. ANET Workshop Sometimes things work... and sometimes they don't note: look of extreme anguish

25. ANET Workshop Distributed Violin Duet McGill-Université de Montréal for RISQnet Conference, Nov. 21, 2001  connected two Montreal universities via IP over 1 Gbps fiber link  uncompressed audio and video  < 30ms latency

26. ANET Workshop "The medium is the message" Marshall McLuhan Cooperstock's Corollary: "The medium must be faster than the message."

27. ANET Workshop Size does matter Montreal-Ottawa (NRC), Distance Teaching Trial, Feb. 20, 2002  connected McGill to National Research Council (Ottawa) with SDI video and multichannel 96kHz/24bit audio  display on 50" plasma screen (near life-size)

28. ANET Workshop Distributed Jazz Jam Montreal-Stanford (CCRMA), Research Demo, Oct. 24, 2002  uncompressed audio and video @ < 50ms latency

29. ANET Workshop Remote Video Interpretation (Sign Language) Montreal-Fredericton (UNB), research trials, 2003  DV (patient, physician) exchanged with remote sign-language interpreter physician's setup patient's setup

30. ANET Workshop Vibrosensory Transmission VRQ demo, March 8, 2004

31. ANET Workshop Coming next: DSD Transmission AES demo, October 31, 2004

32. ANET Workshop Milestones Ahead  multicast  traffic shaping  region-of-interest client selection  multichannel echo-supression/cancellation

33. Self-Signal Isolation: Echo-suppression  the big problem in videoconferencing  same issue for video and vibrosensory data

34. achieving high-fidelity distributed interaction, at levels of presence and latency that support the most demanding applications and doing so in spite of sensor and bandwidth limitations

36. http://ultravideo.mcgill.edu