1. Supporting Beyond-Surface Interaction for Tabletop Display Systems by Integrating IR Projections Hui-Shan Kao Advisor : Dr. Yi-Ping Hung

2. Outline 2  Introduction  Related Work  System Design  Interaction Techniques  Applications  Conclusion

3. Outline 3  Introduction  Related Work  System Design  Interaction Techniques  Applications  Conclusion

4. Introduction 4  Interaction with surfaces mainly support  Multi-touch  Tangible input  Interaction beyond surfaces  Combine with mobile displays iPadTablet PCPico Projector iPhone

5. Introduction 5  Two scenarios  Type 1: Using pico projector Multi-resolution presentation Augmenting personal information  Type 2: Using tablet PC Interactive 3D viewer Type2Type1

6. Challenge 6 How to know the 3D spatial relationship with surface in real-time ? ? Type2Type1

7. Outline 7  Introduction  Related Work  System Design  Interaction Techniques  Applications  Conclusion

8. Localization of Device 8  Enable 3D interaction on tabletop display  Need to recognize the 6DOF of device  The way to know 6DOF  Magnetic tracker Penlight H.Song, T.Grossman, G. Fitzmaurice, F. Guimbretiere, A. Khan, R. Attar, and G. Kurtenbach. Penlight: combining a mobile projector and a digital pen for dynamic visual overlay. In Proc. CHI ’09, 2009

9. Localization of Device 9  Vision based tracker Markers with known structure  Interactive handheld projector  ARToolkit Visible Marker ! Xiang Cao, Clifton Forlines, and Ravin Balakrishnan. Multi-user interaction using handheld projectors. In Proc. UIST ’07, 2007. D. Wagner and D. Schmalstieg. ARToolKitPlus for Pose Tracking on Mobile Devices. In Proc CVWW’07: Proceedings of 12th Computer Vision Winter Workshop, 2007.

10. Outline 10  Introduction  Related Work  System Design  Hardware Configuration  Interaction Techniques  Applications  Conclusion

11. Hardware Configuration 11 Color Projection  visible content IR Camera Color Projector IR Camera IR Projector IR Projection  invisible markers Mirror

12. Hardware Configuration 12  Multi-touch Surface Glass layer  support touch force Diffuser layer  display image  Placement of glass layer and diffuser layer Glass on top of diffuser Diffuser on top of glass

13. Hardware Configuration 13  Option 1: Glass on top of diffuser pico-projection reflection degrade illumination damage user’s eye IR & color projector IR camera Pico- projector  IR camera diffuser touch-glass Hard to Solve!

14. Hardware Configuration 14  Option 2: Diffuser on top of Glass IR-projection reflection spot effect in IR images IR & color projector IR camera Pico- projector spot touch-glass diffuser

15. Hardware Configuration 15  Removing IR spot by using two cameras Weighting Mask (2) IR Camera(2) Weighting Mask (1) IR Camera(1) Stitched View IR Camera 1 IR Camera 2

16. Outline 16  Introduction  Related Work  System Design  Interaction Techniques  Adaptive Markers for Camera Estimation  Dynamic Markers for Multi-Touch  Applications  Conclusion

17. 3D Position Estimation 17 3D position ? Type2Type1 3D position ?

18.  ARToolKitPlus for 3D estimation Fiducial marker Self-identify by ID  Projecting a grid of markers on tabletop  Each marker with  Unique ID  Associated position  Compute the camera’s position based on the marker perceived 3D Position Estimation 18 (0,0) (1,1) ID = 0

19. Multi-Level Markers 19  Uni-level marker  Camera might observe the markers too small or too big  Multi-level marker  System resizes the IR markers according to camera position

20. Marker Split and Merge 20  Adapting the maker size  Markers merge, when camera moves far enough  Markers split, when camera moves too close  Ensure camera to see at least 4 markers

21. Cooperating with multi-touch 21  Marker on for camera positioning  Marker off for finger detection  Foreground-ROI detection for marker on/off

22. 22

23. Cooperating with multi-touch 23 ?

24. Cooperating with multi-touch 24

25. Background Simulation 25  Offline  Save each marker as a patch image and record the position of marker _ = patch collection (offline) :: for each marker base view

26. Background Simulation 26  Online  As the layout re-arrange, the simulated background can be built by the saving patch in real time _ = simulated background (online) :: for each marker in layout ( ) patch of marker base view

27. Background Simulation 27

28. ROI Generation 28 1 st frame after finger touch2 nd and the other frames

29. 29 IR Projector IR Cameras IR Camera of Mobile Device Smoothing Simulated Background Observed Image Foregrounds Tangible Objects Finger Touches Layout Manager Prediction Color Projector ROI Applications Kalman Filtering Real scene Summary of Foreground Detection

30. Outline 30  Introduction  Related Work  System Design  Interaction Techniques  Applications  Conclusion

31. Three Applications 31  Three applications provide intuitive and natural manipulation. iLampiFlashlightiView

32. iLamp 32

33. iLamp  Combine a pico projector and an IR camera  Project seamless high-resolution content, bringing more detailed information  Act as a desk lamp for personal use

34. iLamp 34 ? Real-time R c T c R pc T pc R p T p [R p |T p ] = [R pc |T pc ] x [R c |T c ] R pc T pc

35. Pico projector and camera calibration 35  Off-line : Find transformation between camera and pico projector  The projector can be viewed as a dual of camera  R pc T pc are the Rotation and translation of the attached camera R pc T pc

36. 36  On-line : Find transformation between pico projector and tabletop  Estimate R c T c in real-time  Compute R p T p  [R p |T p ] = [R pc |T pc ] x [R c |T c ] Pico projector and camera calibration R pc T pc

37. iFlashlight 37

38. iFlashlight  A mobile version of iLamp, can be moved easily.  Multi-user cooperation

39. iView 39

40.  An intuitive tool to see 3D content or augmented information of the 2D map from different perspectives. iView

41. Outline 41  Introduction  Related Work  System Design  Interaction Techniques  Applications  Conclusion

42. Conclusion 42  A new interactive surface based on the programmable invisible markers.  Supporting both on-surface and above-surface interaction for any device outfitted with an IR camera.  Bring another level of information on interactive surface.

43. 43 Thank you