1. Vision-Based Finger Detection and Its Applications 基於電腦視覺之手指偵測及其應用 Yi-Fan Chuang Advisor: Prof. Yi-Ping Hung Prof. Ming-Sui Lee

2. Outline  Introduction  Related Work  Fingertip Detection and Tracking  Applications i-m-Top Magic Crystal Ball (MaC Ball)  Conclusion & Future Work 2

3. Outline  Introduction  Related Work  Fingertip Detection and Tracking  Applications i-m-Top Magic Crystal Ball (MaC Ball)  Conclusion & Future Work 3

4. Introduction  Interactive system Bare-handed operations are more intuitive to manipulate digital objects directly Interactions Users: hover/touch gestures Objects: rotation, scaling, selection, special effects…. Use computer vision techniques Fingertip detection Fingertip tracking 4

5. Outline  Introduction  Related Work  Fingertip Detection and Tracking  Applications i-m-Top Magic Crystal Ball (MaC Ball)  Conclusion & Future Work 5

6. Related Work  Enhance Desk Fingertip finding Template matching Palm finding Morphological erosion H. KOIKE and Y. KOBAYASHI, “Integrating paper and digital information on enhanceddesk: a method for realtime finger tracking on an augmented desk system,“ ACM Transation Computer-Human Interaction, vol. 8, no. 4, pp. 307-322, 2001.

7. Related Work  FTIR (Frustrated Total Internal Reflection) J. Y. Han, “Low-cost multi-touch sensing through frustrated total internal reflection," in Proceedings of the 18th annual ACM symposium on User interface software and technology (UIST '05). New York, NY, USA: ACM Press, 2005, pp. 115-118.

8. Related Work  Visual Touchpad Fingertip detection Curvature of contour Finger Orientation Touch detection Two warp images S. Malik and J. Laszlo, “Visual touchpad: a two-handed gestural input device," in Proceedings of the 6th international conference on Multimodal interfaces (ICMI '04). New York, NY, USA: ACM Press, 2004, pp. 289-296.

9. Related Work  PlayAnyWhere Touch and Hover Use appearance of shadows A. D. Wilson, “Playanywhere: a compact interactive tabletop projection-vision system," in Proceedings of the 18th annual ACM symposium on User interface software and technology (UIST '05). New York, NY, USA: ACM Press, 2005, pp. 83-92.

10. Related Work  Comparison 10 Enhance Desk (CHI 01’) FTIR (UIST 05’) Visual Touchpad (ICMI 04’) PlayAnyWhere (UIST 05’) Method Template matching Connected component analysis Curvature of contour Shadow DetectionTouch Touch/Hover (2 cameras) Touch/Hover (1 camera) Disadvantage One user Only touch Only touchOne user One finger for one palm Our Work: Touch/Hover (1 camera) Multi-user with multi-finger

11. Outline  Introduction  Related Work  Fingertip Detection and Tracking  Applications i-m-Top Magic Crystal Ball (MaC Ball)  Conclusion & Future Work 11

12. Overview  System setup An IR camera with IR illuminators to observe hands J. Rekimoto and N. Matsushita, “Perceptual surfaces: Towards a human and object sensitive interactive display," Workshop on Perceptural User Interfaces (PUI'97), 1997. Direct observation Clear shapes Indirect observation Unclear shapes

13.  System setup Direct observation Clear shapes Indirect observation Unclear shapes Overview Clear shapes (MaC Ball) Gesture Captured image

14.  System setup Direct observation Clear shapes Indirect observation Unclear shapes Overview 14 Unclear shapes (i-m-Top) Gesture Captured image

15. Overview  Fingertip detection  Fingertip tracking Detection results Kalman filtering Background subtraction Morphological opening Difference of previous two Principal component analysis Fingertip detection 15

16. Fingertip Detection  Background subtraction  Observation Contact area leaves strong reflection (Enhanced image) Background subtraction Morphological opening Difference of previous two PCA Fingertip detection 16

17. Fingertip Detection  Extract finger part Use a morphological opening operation The size of structuring element is larger than that of a normal finger and smaller than a palm. Original Morphological opening Finger (Enhanced image) Background subtraction Morphological opening Difference of previous two PCA Fingertip detection 17

18. Fingertip Detection  Difference and binarization  Principal component analysis Finger direction Possible fingertips’ positions (Enhanced image) Background subtraction Morphological opening Difference of previous two PCA Fingertip detection Background subtraction Morphological opening Difference of previous two PCA Fingertip detection 18

19. Fingertip Detection  Template matching (pattern matching) Template Remove false matchings Distance check Diagonal check Background subtraction Morphological opening Difference of previous two PCA Fingertip detection Finger patch

20. Detection Results on i-m-Top Fingertip detection Difference of previous two & PCA Morphological opening Background subtraction (Enhanced image) Separate finger touching and palm hovering

21. Detection Results on MaC Ball Fingertip detection Difference of previous two & PCA Morphological opening Background subtraction 21

22. Fingertip Tracking  Kalman filter Smooth the path Predict the new state and its uncertainty Correct the tracker with its new measurement Assume white noise and uniform velocity Original After Kalman filter 22

23. Performance Evaluation  Real-time system is possible 23

24. Outline  Introduction  Related Work  Fingertips Detection and Tracking  Applications i-m-Top Magic Crystal Ball (MaC Ball)  Conclusion & Future Work 24

25. Hardware Configuration  i-m-Top Interactive Multi-resolution Tabletop Cooperate with Yi-Wei Chia 25

26. Software Implementation  Fingertip detection and tracking  Palm detection and tracking  Association between fingertips and palms  Pen detection  Priority of pens and palms  Events definitions 26

27. Palm Detection and Tracking  Palm detection The average position is picked as palm position  Palm tracking Kalman filtering Palm detection Morphological opening (Palm part) Background subtraction (Enhanced Image)

28. Association between Fingertips and Palms  Allocate fingertips to palms Use the angle and the distance between fingers and palms (Enhanced Image) 28

29. Pen Detection and Priority  Pen Detection Use a higher threshold  Priority of pens and palms Higher priority for the pen tip If there is a palm around the pen tip, this palm and the fingertips associated to the palm will be ignored. 29

30. Events Definitions  Hover Palm position  Touch Palm position Finger position and its correspondent palm Pen position 30

31. Application I: Browsing and Editing  Operations Drag photo/document Zoom in and zoom out Stroke Move the foveal area 31

32. Application II: Video Retrieval System  Operations Query Browse Feedback 32 Video Top button Video plane Video wall Scroll bar

33. Outline  Introduction  Related Work  Fingertip Detection and Tracking  Applications i-m-Top Magic Crystal Ball (MaC Ball)  Conclusion & Future Work 33

34. Magic Crystal Ball (MaC Ball)  An interactive 3D display Slide fingers (like a wizard) on the ball surface to control the content 34

35. Hardware Configuration IR Camera IR Illuminator Pressure Sensors User Transparent Glass Ball Reflection Mirror Fresnel Lens LCD Display Module Detection Module

36. Software Implementation  Fingertip detection and tracking Detect fingertips’ positions Pointing gesture  Motion detection Optical flow algorithm Waving gesture  Contact Detection Pressure Sensor (Flexi Force) Detect whether users touch MaC Ball

37. Software Implementation  Contact detection – Pressure Sensors Observation p1 p2 p3 p1 p2 p1 p2 37

38. Events Definitions  Gesture switch Pointing Gesture Fingertips Detection Motion Detection Fingertips Detection Waving Gesture Pressure sensors Touch / Hover 38

39. Application: Virtual Exhibition  Operations Rotate Select Magnifier Change relics 39 Video

40. Outline  Introduction  Related Work  Fingertip Detection and Tracking  Applications i-m-Top Magic Crystal Ball (MaC Ball)  Conclusion & Future Work 40

41. Conclusion & Future Work  Multi-finger detection techniques Simple and real-time Cost-effective and flexible  Applications i-m-Top: interactive interface MaC Ball: interactive 3D display  Future work Use FTIR to enhance the accuracy of fingertip detection More gestures for interactive systems 41