stanford hci group / cs376 u Scott Klemmer · 28 November 2006 Vision- Based Interacti on

2 cs547: Blake Ross and Asa Dotzler Mozilla: Creating simple software in a geek-driven culture

3 The first vision-based interface  Myron Krueger used computer vision to create Responsive Environments (1970s)  “Reaction is the Medium”  timeline/videoplace_video.html

4 How it works  Video and background are separated in analog using chroma key techniques (think broadcast news)  The first and last points of each raster are stored in the computer, and represent the person’s outline

5 Vision-based UIs: “Verbs”  Detecting and Tracking elements of a certain type in a scene  Capturing contents of detected objects  Recognizing individual members in an object class

6 Vision-based UIs: “Verbs”  Detecting and Tracking elements of a certain type in a scene

7 Vision-based UIs: “Verbs”  Capturing contents of detected objects

8 Vision-based UIs: “Verbs”  Recognizing individual members in a class

9 Vision-based UIs: “Nouns”  People (one or multiple)  Bodies  Faces  Hands  Documents  Objects

10 Vision-based UIs: “Nouns”  People (one or multiple)  Bodies  Faces  Hands  Documents  Objects

11 Vision-based UIs: “Nouns”  People (one or multiple)  Bodies  Faces  Hands  Documents  Objects

12 Background Subtraction I N F R A S T R U C T U R E

13 Image Moments (of Inertia)  0 th moment is mass (total number of pixels)

14 Image Moments (of Inertia)  1 st moment is center

15 Image Moments (of Inertia)  2 nd moment is orientation

16 Tools for Vision apps  Intel’s OpenCV  C API to highly optimized image processing functions (threshold, dilate, optical flow, …)  ncv  Fast to run! Slow to develop  Great for vision folks; too low-level for app folks  Papier-Mâché  Java API (and to some extent visual UI) for vision (and other physical input)   Fast to develop! Slow to run  Great for app folks; ~5 fps can sometimes be too slow

17 Good Vision Books  Computer Vision: A Modern Approach  David Forsyth and Jean Ponce (2003)  Fantastic book; but goal is more theoretical understanding than practical application  Robot Vision  Berthold Horn (1987)  More focused on apps and interactive algorithms  Somewhat out of date

18 Next Time… Software Tools Past, Present, and Future of User Interface Software Tools, Brad Myers, Scott E. Hudson, Randy Pausch Natural Programming Languages and Environments, Brad A. Myers, John F. Pane, Andy Ko