Perceptive Context for Pervasive Computing Trevor Darrell Vision Interface Group MIT AI Lab

MIT Project Oxygen A multi-laboratory effort at MIT to develop pervasive, human-centric computing Enabling people “to do more by doing less,” that is, to accomplish more with less work Bringing abundant computation and communication as pervasive as free air, naturally into people’s lives

Human-centered Interfaces Free users from desktop and wired interfaces Allow natural gesture and speech commands Give computers awareness of users Work in open and noisy environments -Outdoors -- PDA next to construction site! -Indoors -- crowded meeting room Vision’s role: provide perceptive context

Perceptive Context Who is there? (presence, identity) What is going on? (activity) Where are they? (individual location) Which person said that? (audiovisual grouping) What are they looking / pointing at? (pose, gaze)

Vision Interface Group Projects Person Identification at a distance from multiple cameras and multiple cues (face, gait) Tracking multiple people in indoor environments with large illumination variation and sparse stereo cues Vision guided microphone array Joint statistical models for audiovisual fusion Face pose estimation: rigid motion estimation with long- term drift reduction

Vision Interface Group Projects Person Identification at a distance from multiple cameras and multiple cues (face, gait) Tracking multiple people in indoor environments with large illumination variation and sparse stereo cues Vision guided microphone array Joint statistical models for audiovisual fusion Face pose estimation: rigid motion estimation with long- term drift reduction

Person Identification at a distance Multiple cameras Face and gait cues Approach: canonical frame for each modality by placing the virtual camera at a desired viewpoint Face: frontal view, fixed scale Gait: profile silhouette Need to place virtual camera -explicit model estimation -search -motion-based heuristic  trajectory We combine trajectory estimate and limited search

Virtual views Frontal Profile silhouette: Face: Input

Examples: VH-generated views Faces: Gait:

Effects of view-normalization

Vision Interface Group Projects Person Identification at a distance from multiple cameras and multiple cues (face, gait) Tracking multiple people in indoor environments with large illumination variation and sparse stereo cues Vision guided microphone array Joint statistical models for audiovisual fusion Face pose estimation: rigid motion estimation with long- term drift reduction

Range-based stereo person tracking Range can be insensitive to fast illumination change Compare range values to known background Project into 2D overhead view IntensityRangeForeground Plan view Merge data from multiple stereo cameras.. Group into trajectories… Examine height for sitting/standing…

Visibility Constraints for Virtual Backgrounds virtual background for C 1

Virtual Background Segmentation Sparse Background New Image Detected Foreground! Second View Virtual Background for first view Detected Foreground!

Points -> trajectories -> active sensing Active Camera motion Microphone array Activity classification trajectories Spatio- temporal points

Vision Interface Group Projects Person Identification at a distance from multiple cameras and multiple cues (face, gait) Tracking multiple people in indoor environments with large illumination variation and sparse stereo cues Vision guided microphone array Joint statistical models for audiovisual fusion Face pose estimation: rigid motion estimation with long- term drift reduction

Audio input in noisy environments Acquire high-quality audio from untethered, moving speakers “Virtual” headset microphones for all users

Vision guided microphone array Cameras Microphones

System flow (single target) Vision-based tracker Gradient ascent search in array output power Delay-and-sum beamformer Video Streams Audio Streams

Vision Interface Group Projects Person Identification at a distance from multiple cameras and multiple cues (face, gait) Tracking multiple people in indoor environments with large illumination variation and sparse stereo cues Vision guided microphone array Joint statistical models for audiovisual fusion Face pose estimation: rigid motion estimation with long- term drift reduction

Audio-visual Analysis Multi-modal approach to source separation Exploit joint statistics of image and audio signal Use non-parametric density estimation Audio-based image localization Image-based audio localization A/V Verification: is this audio and video from the same person?

Audio-visual synchrony detection

Audio weighting from video (detected face) + AVMI Applications Image localization from audio Audio associated with left face Audio associated with right face New: Synchronization Detection! image variance AVMI

Audio-visual synchrony detection MI: Compute confusion matrix for 8 subjects: No errors! No training! Also can use for audio/visual temporal alignment….

Vision Interface Group Projects Person Identification at a distance from multiple cameras and multiple cues (face, gait) Tracking multiple people in indoor environments with large illumination variation and sparse stereo cues Vision guided microphone array Joint statistical models for audiovisual fusion Face pose estimation: rigid motion estimation with long- term drift reduction

Face pose estimation rigid motion estimation with long-term drift reduction

Brightness and depth motion constraints I t I t + 1 II ZZ Z t Z t + 1 y t =  y t-1 Parameter space

New bounded error tracking algorithm Influence region open loop 2D tracker closed loop 2D tracker Track relative to all previous frames which are close in pose space

Closed-loop 3D tracker Track users head gaze for hands-free pointing…

Head-driven cursor Related Projects: Schiele Kjeldsen Toyama Current application for second pointer or scrolling / focus of attention…

Head-driven cursor MethodAvg. error. (pixels) Cylindrical head tracker 25 2D Optical Flow head tracker22.9 Hybrid 30 3D head tracker (ours)7.5 Eye gaze 27 Trackball3.7 Mouse1.9

Gaze aware interface Drowsy driver detection: head nod and eye-blink… Interface Agent responds to gaze of user -agent should know when it’s being attended to -turn-taking pragmatics -anaphora / object reference First prototype -E21 interface “sam” -current experiments with face tracker on meeting room table Integrating with wall cameras and hand gesture interfaces…

“Look-to-talk” Subject not looking at SAM ASR turned off Subject looking at SAM ASR turned on

Vision Interface Group Projects Person Identification at a distance from multiple cameras and multiple cues (face, gait) Tracking multiple people in indoor environments with large illumination variation and sparse stereo cues Vision guided microphone array Joint statistical models for audiovisual fusion Face pose estimation: rigid motion estimation with long- term drift reduction Conclusion and contact info.

Conclusion: Perceptual Context Take-home message: vision provides Perceptual Context to make applications aware of users.. activity -- adapting outdoor activity classification [ Grimson and Stauffer ] to indoor domain… So far: detection, ID, head pose, audio enhancement and synchrony verification… Soon: gaze -- add eye tracking on pose stabilized face pointing -- arm gestures for selection and navigation.

Contact Prof. Trevor Darrell Person Identification at a distance from multiple cameras and multiple cues (face, gait) -Greg Shakhnarovich Tracking multiple people in indoor environments with large illumination variation and sparse stereo cues -Neal Checka, Leonid Taycher, David Demirdjian Vision guided microphone array -Kevin Wilson Joint statistical models for audiovisual fusion -John Fisher Face pose estimation: rigid motion estimation with long-term drift reduction -Louis Morency, Alice Oh, Kristen Grauman