Computer Vision Technologies for Remote Collaboration Using Physical Whiteboards, Projectors and Cameras Zhengyou Zhang Microsoft Research mailto: R

Motivations  WB provides a large shared space for the participants to focus their attention and express their ideas spontaneously  Many meetings use whiteboard heavily brainstorming sessions, lectures, project planning meetings, patent disclosures, etc  WB is ubiquitous in knowledge workers’ offices → Whiteboard will still exist for long time

Motivations (cont’d)  Difficulties: –Content is hard to archive or share –Busy in note taking, instead of spending time sharing and absorbing ideas  Need tools to increase productivities –Don’t lose any sparks –Share with others –Keep for later review –Integrate with your office document

Motivations (cont’d)  Projector –A great tool for presentation –Not so convenient for discussions  Whiteboard-Whiteboard-Camera System –Whiteboard: Projecting surface (Output) and Writing surface (Input) –Seamless integration of computer presentation with whiteboard discussions –Enable remote collaboration on a shared workspace

Whiteboard Scenarios  Note taking with a high-res digital camera –Whiteboard It!  Note taking with a low-res PC camera –Whiteboard Scanning  Real-time collaboration –Live Whiteboard  Meeting (WB + Audio / Video) Archiving –Whiteboard Archiving

♥ Whiteboard It! ♥  Convert WB Content into an Electronic Document

Overview of the System

♥ Whiteboard Scanning ♥  Get a High-res Scan With an Inexpensive PC Camera  How to do it? –Take multiple pictures with overlap –Stitch them automatically into a high-res image –Image enhancement (Whiteboard It!)

System Overview

Example + +

Document Scanning w/ Low-Res camera

♥ Live Whiteboard ♥  Real-time distributed collaboration Whiteboard Display Mic (array) Table Projector Speaker PC Meeting Room/Office Windows Communications Desktop TabletPC Wire or wireless Camera Offices

A typical image sequence Segmentation of the person and WB background is needed

Demo  Integration with Windows Messenger Whiteboard update Annotations Windows Messenger T120 Local WB clientRemote WB client USB 2 Video Real-Time WB Processing Video Whiteboard changes

♥ Whiteboard Archiving ♥  Capture whiteboard content + audio/Video –high-resolution digital still camera  Produce key frames –A KF usually corresponds to a major topic –Print as notes, or cut & paste into documents  Record time stamps of each stroke  Efficient meeting browsing –Key frames to navigate between sections –Strokes to bring up the audio at the moment when they were written

A Typical System Installation

Example: Sample Images

Key Frame Extraction  Number of strokes Chapter 1Chapter 2 Key Frame 1 Key Frame 2

Browsing Interface: DemoDemo Key Frame Thumbnails Future Strokes Current Strokes Raw Image VCR & Timeline Control

♥ Projector-Whiteboard-Camera System ♥ Communi- cation Network Video Mixer Presentation Video Mixer P Visual Echo Cancellation W r i t i n g s A A Annotation A PP P W W Local Meeting Room Remote Room Whiteboard Camera Projector Display Remote Room P+A P+A+W P+W

Visual Echo Cancellation  Prerequisites –Geometric calibration –Color calibration recover writings W albedo estimation & color clustering Color Lookup Table (given by color calibration) Color Mapping& Geometric Warping estimated visual echo E captured image I Homography H (given by geometric calibration)

VEC Results

Conclusions  Tools to use whiteboard and projector  Whiteboard It! for anyone who has a digital camera and it is easy to use  Whiteboard Scanning for anyone who has a PC camera and it is easy to use  Whiteboard Archiving and Streaming for meeting rooms and collaboration  Integration of Projector and Whiteboard for seamless remote collaboration

Acknowledgements  Thanks to Li-wei He, Zicheng Liu & Hanning Zhou for contributions