A Systems Architecture for Ubiquitous Video Neil J. McCurdy William G. Griswold Department of Computer Science and Engineering University of California, San Diego

2 What is ubiquitous video? Cameras fading into the woodwork Networked cameras in “the wild” Can these cameras support remote exploration? RealityFlythrough makes it possible Virtual mobility for disabled Any-angle stadium views Pre-drive driving directions Virtual shopping My-day diaries

3 The need for an abstraction To make remote exploration feel like local exploration, we need a camera at every position and orientation Move the Camera “Telepresence” does this by moving the camera Mimics walking through space –Think Mars Explorer Challenge: Mobility is limited by the physicality of the robot Camera Field of View Telepresence: [Kuzuoka, Tachi]

4 To make remote exploration feel like local exploration, we need a camera at every position and orientation The need for an abstraction Interpolate (Tele-reality) If there are enough cameras, construct “novel views” Reconstructs scene geometry using vision techniques –Think “Matrix Revolutions” Challenge: Requires precise knowledge of camera locations and optics properties. It’s slow. Tele-reality: [Szeliski, Kanade]

5 To make remote exploration feel like local exploration, we need a camera at every position and orientation The need for an abstraction Interpolate (Tele-reality) If there are enough cameras, construct “novel views” Reconstructs scene geometry using vision techniques –Think “Matrix Revolutions” Challenge: Requires precise knowledge of camera locations and optics properties. It’s slow. Novel Views Tele-reality: [Szeliski, Kanade]

6 To make remote exploration feel like local exploration, we need a camera at every position and orientation The need for an abstraction Combine VR and Reality Pre-acquire a model Project live video onto the model Challenge: What happens when the model is no longer accurate? What can realistically be modeled? ? Augmented Virtual Environments: [Neumann] User’s view

7 The need for an abstraction Challenges of ubiquitous video Camera density is low Environment is dynamic –People and objects are moving –Sensors are moving Environment is uncalibrated –Geometry of the environment is not known –Sensors are inaccurate Need data live and in real-time we need a camera at every position and orientation San Diego MMST Drill, May 12, 2005

8 Roadmap The need for an abstraction Need a camera at every position and orientation Challenges of ubiquitous video Building the RealityFlythrough abstraction Motion as a substitute for ∞ cameras Choosing what (not) to show Handling dynamic environments Archiving live imagery Evaluating the abstraction Usability Robustness to change Scalability

9 Simplifying 3d space We know the location and orientation of each camera From a corresponding location in virtual space we project the camera’s image onto a virtual wall When the user’s virtual position is the same as the cameras, the entire screen is filled with the image Results in a 2d simplification of 3d space Motion as a substitute for ∞ cameras

10 The transition A transition between cameras is achieved by moving the user’s location from the point of view of the source camera to the point of view of the destination camera The virtual walls are shown in perspective Overlapping portions of images are alpha-blended Motion as a substitute for ∞ cameras

11 Why transitions are effective Humans commit closure [McCloud] –Visual cortex automatically makes sense of incomplete information –Eg. Blind spots Transitions reveal rather than conceal inaccuracies –Overlaps help us make sense of imagery –Position accuracy is important Transitions provide the following cues –Motion, speed, filler images, grid-lines Key assumption: User is largely content to directly view a camera’s image, or is in transition to another camera Motion as a substitute for ∞ cameras

12 Non-intersecting camera views Pacing and gridlines help Intervening space can be filled with other camera views –Either other live cameras or archived imagery (discussion in a moment) Motion as a substitute for ∞ cameras

13 Choosing what (not) to show How do we decide which cameras to choose There are no obvious choices along the path What if we just show all of them? Choosing what (not) to show

14 We project where we will be in the future We choose the best camera at that location Fitness functions: Proximity, Screen fill Liveness, Recency The trick is to hide information Heuristics for choosing cameras Current image should stay in view for as long as possible Once the destination image is visible, choose it There should be a minimum duration for subtransitions Choosing what (not) to show

15 Roadmap The need for an abstraction Need a camera at every position and orientation Challenges of ubiquitous video Building the RealityFlythrough abstraction Motion as a substitute for ∞ cameras Choosing what (not) to show Handling dynamic environments Archiving live imagery Evaluating the abstraction Usability Robustness to change Scalability

16 The destination moved! Computing the path and the cameras to display at the start of the transition does not work Problem 1: Destination may be a moving target Problem 2: Intervening cameras may not be optimal Handling Dynamic Environments

17 Handling dynamic environments Step 1: Paths need to be dynamic Step 2: Cameras need to be selected just-in-time Handling Dynamic Environments

18 There are still some problems Problem 1: Course correction is too disorienting Problem 2: Too many dimensions of movement –User’s movement (x,y,z) –Camera’s movement –Scene movement What we tried: Paths need to be dynamic Cameras need to be selected just-in-time Handling Dynamic Environments

19 The current state of affairs Problem 1: Course correction is too disorienting Problem 2: Too many dimensions of movement Solutions: First move to where the camera was. Then quickly capture the moving target Pause the live video whenever it’s visible and play at increased speed until we’re back to live action Handling Dynamic Environments

20 Roadmap The need for an abstraction Need a camera at every position and orientation Challenges of ubiquitous video Building the RealityFlythrough abstraction Motion as a substitute for ∞ cameras Choosing what (not) to show Handling dynamic environments Archiving live imagery Evaluating the abstraction Usability Robustness to change Scalability

21 Archiving live imagery Why do it? Still-images generated from live video feeds increase camera density Help us create the illusion of infinite camera coverage Competing desires Maximal camera density Quality images –Still image act as the anchors in a sea of confusing movement

22 Archiving live imagery How do we do it? Each frame from every camera is considered Sensor data (location, orientation) is validated for accuracy Images are assigned a quality based on possible blurriness (eg. high position delta) What is stored The most recent highest quality image, for a particular location (eg. 1 meter² with a 15 degree arc) The image is treated as if it was a non-moving camera

23 Roadmap The need for an abstraction Need a camera at every position and orientation Challenges of ubiquitous video Building the RealityFlythrough abstraction Motion as a substitute for ∞ cameras Choosing what (not) to show Handling dynamic environments Archiving live imagery Evaluating the abstraction Usability Robustness to change Scalability

24 Scalability H323 Video Conferencin g Stream Bottleneck 1: 10 stream max (Fewer with higher FPS) Bottleneck 2: 112 stream max decode MCU (Multipoint Control Unit) RFT Engine Cameras ImageCaptureSensorCapture StreamCombine (352x288 video resolution) X

25 Scalability Bottleneck 2: 112 stream max decode MCU (Multipoint Control Unit) RFT Engine Bottleneck 3: 15 streams max (no image archive) MCU (Multipoint Control Unit) RFT Engine (w/ image archive) Conclusion: It is the number of live cameras, and not the total number of cameras that is the immediate bottleneck 550 archive “cameras”

26 Future work Implement “virtual camera metaphor” –Contrasts with the hitch-hiking metaphor described so far –Abstraction stretched to support “best” views from any point in space –Essentially, novel views, but with the view changing based on which cameras cover the desired view Integrate high-level information that is present in the birdseye view into the first-person view Support sound Scale to multiple viewers with multiple servers User’s Desired View Archived imagery

27 Conclusion RealityFlythrough creates the illusion of infinite cameras Possible despite the challenges of ubiquitous video –Camera density is low –Environment is dynamic –Environment is uncalibrated –Need data live and in real-time We do this by –Using motion as a substitute for ∞ cameras –Choosing the best imagery to show –Selecting imagery and path just-in-time –Using archived live imagery to increase camera density Questions? Ask to see a demo

28 Other slides

29 The illusion of infinite camera coverage Camera PhysicalCameraVirtualCamera CameraWithStatePositionSource ImageSource EnvironmentState 1 * 1 1 Model in MVC

30 RealityFlythrough Engine EnvironmentState (Model) Views Controller CameraRepository StillImageGen TransitionPlanner TransitionExecuter H323Connection Manager << uses 1 st Topic 2 nd Topic 3 rd Topic

31 Transitions TransitionPlannerPlannerSimplePlannerBestFitTransitionExecuterPathPathStraightFitnessFunctorOverallFitness ProxFitness Kinds of Fitness Generates>> * ProximityFitness, LivenessFitness,...

32 Related Work Use model so can approximate photorealism –Pre-acquired Model Neumann, et al. [1] w/ Augmented Virtual Environments Only works w/ static structures –Acquire model from image data Preprocess still imagery –Szeliski [2] –Chen [3] w/ Quicktime VR Know exact camera locations –Kanade [4] w/ Virtualized Reality

33 How are these images related?

34 How are these images related?

35 Like this!