Supporting Beyond-Surface Interaction for Tabletop Display Systems by Integrating IR Projections Hui-Shan Kao Advisor : Dr. Yi-Ping Hung

Outline 2  Introduction  Related Work  System Design  Interaction Techniques  Applications  Conclusion

Outline 3  Introduction  Related Work  System Design  Interaction Techniques  Applications  Conclusion

Introduction 4  Interaction with surfaces mainly support  Multi-touch  Tangible input  Interaction beyond surfaces  Combine with mobile displays iPadTablet PCPico Projector iPhone

Introduction 5  Two scenarios  Type 1: Using pico projector Multi-resolution presentation Augmenting personal information  Type 2: Using tablet PC Interactive 3D viewer Type2Type1

Challenge 6 How to know the 3D spatial relationship with surface in real-time ? ? Type2Type1

Outline 7  Introduction  Related Work  System Design  Interaction Techniques  Applications  Conclusion

Localization of Device 8  Enable 3D interaction on tabletop display  Need to recognize the 6DOF of device  The way to know 6DOF  Magnetic tracker Penlight H.Song, T.Grossman, G. Fitzmaurice, F. Guimbretiere, A. Khan, R. Attar, and G. Kurtenbach. Penlight: combining a mobile projector and a digital pen for dynamic visual overlay. In Proc. CHI ’09, 2009

Localization of Device 9  Vision based tracker Markers with known structure  Interactive handheld projector  ARToolkit Visible Marker ! Xiang Cao, Clifton Forlines, and Ravin Balakrishnan. Multi-user interaction using handheld projectors. In Proc. UIST ’07, D. Wagner and D. Schmalstieg. ARToolKitPlus for Pose Tracking on Mobile Devices. In Proc CVWW’07: Proceedings of 12th Computer Vision Winter Workshop, 2007.

Outline 10  Introduction  Related Work  System Design  Hardware Configuration  Interaction Techniques  Applications  Conclusion

Hardware Configuration 11 Color Projection  visible content IR Camera Color Projector IR Camera IR Projector IR Projection  invisible markers Mirror

Hardware Configuration 12  Multi-touch Surface Glass layer  support touch force Diffuser layer  display image  Placement of glass layer and diffuser layer Glass on top of diffuser Diffuser on top of glass

Hardware Configuration 13  Option 1: Glass on top of diffuser pico-projection reflection degrade illumination damage user’s eye IR & color projector IR camera Pico- projector  IR camera diffuser touch-glass Hard to Solve!

Hardware Configuration 14  Option 2: Diffuser on top of Glass IR-projection reflection spot effect in IR images IR & color projector IR camera Pico- projector spot touch-glass diffuser

Hardware Configuration 15  Removing IR spot by using two cameras Weighting Mask (2) IR Camera(2) Weighting Mask (1) IR Camera(1) Stitched View IR Camera 1 IR Camera 2

Outline 16  Introduction  Related Work  System Design  Interaction Techniques  Adaptive Markers for Camera Estimation  Dynamic Markers for Multi-Touch  Applications  Conclusion

3D Position Estimation 17 3D position ? Type2Type1 3D position ?

 ARToolKitPlus for 3D estimation Fiducial marker Self-identify by ID  Projecting a grid of markers on tabletop  Each marker with  Unique ID  Associated position  Compute the camera’s position based on the marker perceived 3D Position Estimation 18 (0,0) (1,1) ID = 0

Multi-Level Markers 19  Uni-level marker  Camera might observe the markers too small or too big  Multi-level marker  System resizes the IR markers according to camera position

Marker Split and Merge 20  Adapting the maker size  Markers merge, when camera moves far enough  Markers split, when camera moves too close  Ensure camera to see at least 4 markers

Cooperating with multi-touch 21  Marker on for camera positioning  Marker off for finger detection  Foreground-ROI detection for marker on/off

22

Cooperating with multi-touch 23 ?

Cooperating with multi-touch 24

Background Simulation 25  Offline  Save each marker as a patch image and record the position of marker _ = patch collection (offline) :: for each marker base view

Background Simulation 26  Online  As the layout re-arrange, the simulated background can be built by the saving patch in real time _ = simulated background (online) :: for each marker in layout ( ) patch of marker base view

Background Simulation 27

ROI Generation 28 1 st frame after finger touch2 nd and the other frames

29 IR Projector IR Cameras IR Camera of Mobile Device Smoothing Simulated Background Observed Image Foregrounds Tangible Objects Finger Touches Layout Manager Prediction Color Projector ROI Applications Kalman Filtering Real scene Summary of Foreground Detection

Outline 30  Introduction  Related Work  System Design  Interaction Techniques  Applications  Conclusion

Three Applications 31  Three applications provide intuitive and natural manipulation. iLampiFlashlightiView

iLamp 32

iLamp  Combine a pico projector and an IR camera  Project seamless high-resolution content, bringing more detailed information  Act as a desk lamp for personal use

iLamp 34 ? Real-time R c T c R pc T pc R p T p [R p |T p ] = [R pc |T pc ] x [R c |T c ] R pc T pc

Pico projector and camera calibration 35  Off-line : Find transformation between camera and pico projector  The projector can be viewed as a dual of camera  R pc T pc are the Rotation and translation of the attached camera R pc T pc

36  On-line : Find transformation between pico projector and tabletop  Estimate R c T c in real-time  Compute R p T p  [R p |T p ] = [R pc |T pc ] x [R c |T c ] Pico projector and camera calibration R pc T pc

iFlashlight 37

iFlashlight  A mobile version of iLamp, can be moved easily.  Multi-user cooperation

iView 39

 An intuitive tool to see 3D content or augmented information of the 2D map from different perspectives. iView

Outline 41  Introduction  Related Work  System Design  Interaction Techniques  Applications  Conclusion

Conclusion 42  A new interactive surface based on the programmable invisible markers.  Supporting both on-surface and above-surface interaction for any device outfitted with an IR camera.  Bring another level of information on interactive surface.

43 Thank you