National Research Council Canada Conseil national de recherches Canada National Research Council Canada Conseil national de recherches Canada Canada Dmitry Gorodnichy, Gerhard Roth, Shahzad Mallik Institute for Information Technology Institut de technologie de l'information Institute for Information Technology Institut de technologie de l'information Computational Video Group Groupe Vidéo Informatique Computational Video Group Groupe Vidéo Informatique Nouse “Use your N ose as M ouse ” – New Technology for Hands-free Games and Interfaces Nouse “Use your N ose as M ouse ” – New Technology for Hands-free Games and Interfaces

(2) Face-Tracking Based User Interfaces 1.Replacing cumbersome track-ball (track-stick) on laptops. 2.Extra degree of control (e.g. to switch the focus of attention). 3.Hands free control (e.g. for handicap users). 4.Interactive games: more physical, entertaining, 3D control, multiple-user Fig.1. A user plays an aim-n-shoot Bubble-Frenzy game aiming the turret by pointing with her nose. (slight rotation of head allows to aim precisely in 180 o range)

(3) Fig. 2. Two users play a virtual ping-pong game, bouncing the ball with their heads. Image-based tracking allows one to track heads, however it doesn’t allow one to pin-point with head. Key Issues and Approaches 1.Speed (in real time). 2. Affordability (with cheap easy-to-install, but low-quality USB cams) 3. Robustness Precision 3. Robustness (to normal head motion). 4. Precision (with pixel precision) Image-based Face Tracking: - Uses global facial cues: skin colour, head shape, head motion not precise - Doesn’t require high-quility images, robust, but not precise

(4) Should be used for precise tracking. However, it’s not robust. [ Bradsky, Toyama, Gee, Cipolla, Zelinsky, Matsumoto, Yang, Baluja, Newman, …] …”still not ready for practical implementation” - Feature f is associated with vector V f (obtained by centering a mask on the feature) - Features are tracked by template matching with V f in the local area of interest (calculated with image- based cues) - The pixel u=(i,j) which has the largest score s(V u, V f ) is returned Feature-based Face Tracking Fig. 3. Tracking eyes (from [Gorodnichy97]). Question: What features to use? Proposition 1: Robust and precise tracking can be achieved by designing an invariant to head motion feature template.

(5) Features are conventionally thought of as visually distinctive (ie with large  I(f) ). Hence, the commonly used features are edge-based, such as corners of brows, eyes, lips, nostrils etc. They however are not robust not always visible Desired feature properties: 1. Uniqueness: s(V f, V u )  min 2. Robustness: s(V f t=0, V f t )  max 3. Continuity (for sub-pixel accuracy): the closer a pixel u in an image is to the pixel corresponding to f, the larger the score between V u and V f (Then evidence-based convolution can be applied to refine feature position u ) Edge-based Features Edge-based Features – not good

(6) Convex-shape features Convex-shape features – much better Definition 1: Convex-shape feature is defined as an extremum of a convex-shape surface Shape-from-Shading theory shows, that these features exhibit the desired properties (for the fixed camera-user-light configuration) Nose feature Definition 2: Nose feature is the extremum of the 3D nose surface curvature defined as z=f(x,y) in camera centered coordinate system. Thus defined, Nose feature is Very robust Can be detected with sub-pixel precision PLUS, It is always visible!

(7) Nouse TM Face Tracking Technology Based on tracking the convex-shape nose feature. Enables precise hand-free 2D control in a) joystick or b) mouse modes. Allows aiming and drawing with the nose. Just think of your nose as a chalk or a joystick handle! NB: Left/Right head motion is very natural and can be easily applied for control, provided it can tracked precisely. Affordable and downloadable. Uses a generic USB camera! Zero intialization of NouseUsing Nouse for Painting

(8) Performance: Robustness & Precision The range of head motion tracked ‘No’ motion ‘Yes’ motion Robustness to rotation Robustness to scale Test: The user rotates his head only! (the shoulders do not move)

(9) Demo: Range of Tracked Motion

(10) On Importance of Two Cameras For humans: it is much easier to track with two eyes than with one eye. Not only extends tracking from 2D to 3D, but also makes tracking more precise and robust! For computers however: … 1. The relationship between “eyes” is not known. 2. Tracking of features is not robust (to rotation and scale) StereoTracker from CVG NRC: Tracks face in 3D with two USB cams to control a virtual man, by using 1) Projective Vision Theory and 2) robust Nose Feature Tracking

(11) StereoTracking with USB webcams Stage 1: Self-calibration The relationship between the cameras is represented using the Fundamental Matrix F : (u left, F u right )=0 F can be found automatically for any two cameras by observing the same scene using : find corners  matching  filtering  robust solution with 7 selected corners ( RANSAC )  F Stage 2: Feature selection and calibration verification Select features in one image Verify that the epipolar line passes thru each feature in the second image Use nose tip feature and two other common features (eg brow corners) More at

(12) Using Nose for StereoTracking Proposition 2 : With F known, the tracked 3D feature is the one that minimizes the epipolar error defined by Proposition 3 : First detect convex-shape nose feature. Then use rigidity constraint to find other features.

(13) Demo: Stereotracking at Work Robustness to Scale and Rotation around all (!) axis of rotation

(14)Conclusions Acknowledgements Nouse TM is trademark of Computational Video Group IIT NRC BubbleFrenzy game is provided by Nose is a very unique feature. Humans are lucky to have it! Nose allows us to track a face very robustly and precisely. Pointing with Nose is natural. This makes 2D perceptual user interfaces a reality! Nose helps recovering other facial features. Two cameras (even bad webcams) make tracking more robust. This makes 3D face tracking affordable, precise and robust. Use your Nose as MouseUse NouseUse your Nose as Mouse! – Use Nouse! Nouse TM is open for public evaluation at