1. A Framework for Multi-resolution and Multi-touch Systems Presenter: 胡婷婷 多重解析度及多重觸控互動系統之開發 Advisor: 洪一平 教授

2. A Framework for Multi-Resolution & Multi-Touch Systems Page 2 of 41 Outline  Introduction  Related Work  i-m-Top System  i-m-Top SDK  Application  Conclusion & Future Work

3. A Framework for Multi-Resolution & Multi-Touch Systems Page 3 of 41 Outline IIntroduction RRelated Work ii-m-Top System ii-m-Top SDK AApplication CConclusion & Future Work

4. A Framework for Multi-Resolution & Multi-Touch Systems Page 4 of 41 Introduction Motivation  Multi-resolution Variable-resolution nature of human vision Cost-effective large high-res display  Multi-touch Intuitive interaction Computer-vision based approach  Software Toolkit Hide engineering challenges

5. A Framework for Multi-Resolution & Multi-Touch Systems Page 5 of 41 Introduction i-m-Top  interactive multi-resolution tableTop interactive: multi-touch multi-resolution: fovea + peripheral projectors i-m-Top SDK  Toolkit for rapid prototyping applications Featured on multi-resolution/multi-touch

6. A Framework for Multi-Resolution & Multi-Touch Systems Page 6 of 41 Overview Introduction Projectors Touch Detection Other sensors Cams User Interface i-m-Top SDK Image Processing Applications Pan-tilt Unit Multimedia

7. A Framework for Multi-Resolution & Multi-Touch Systems Page 7 of 41 Outline IIntroduction RRelated Work ii-m-Top System ii-m-Top SDK AApplication CConclusion & Future Work

8. A Framework for Multi-Resolution & Multi-Touch Systems Page 8 of 41 Related Work Multi-Resolution Display  Fovea Plus Context LCD screen (fixed) + projector Baudisch, P., Good, N., and Stewart, P. "Focus Plus Context Screens: Combining Display Technology with Visualization Techniques.“ In proceedings of ACM UIST '01

9. A Framework for Multi-Resolution & Multi-Touch Systems Page 9 of 41 Related Work Multi-Resolution Display  Fovea-Tablett Tablets as high-res displays Tracked with visual marker Multiple high-res is allowed Cons: physical boundary J. Geisler, R. Eck, N. Rehfeld, E. Peinsipp-Byma, C. Schutz, and S. Geggus, “Fovea- tablette: A new paradigm for the interaction with large screens," in HCI (8), 2007

10. A Framework for Multi-Resolution & Multi-Touch Systems Page 10 of 41 Related Work Software Development Toolkit  DiamondTouch SDK (C++) Multi-touch, multi-user Esenther, A.; Forlines, C.; Ryall, K.; Shipman, S., "DiamondTouch SDK: Support for Multi-User, Multi-Touch Applications", ACM Conference on CSCW, 2002

11. A Framework for Multi-Resolution & Multi-Touch Systems Page 11 of 41 Related Work Software Development Toolkit  DiamondSpin (Java): Around-the-table interaction Real-time polar to Cartesian transformation public + personal work areas rotatable circular tabletop Shen, C.; Vernier, F.D.; Forlines, C.; Ringel, M., "DiamondSpin: An Extensible Toolkit for Around-the-Table Interaction", CHI 2004

12. A Framework for Multi-Resolution & Multi-Touch Systems Page 12 of 41 Related Work Software Development Toolkit  T3 (Java) Multiple projectors → single high-res display Mixed-presence collaboration Philip Tuddenham and Peter Robinson. “T3: Rapid Prototyping of High- Resolution and Mixed-Presence Tabletop Applications”, TABLETOP 2007.

13. A Framework for Multi-Resolution & Multi-Touch Systems Page 13 of 41 Outline  Introduction  Related Work  i-m-Top System Hardware Configuration Touch Detection  i-m-Top SDK  Application  Conclusion & Future Work

14. A Framework for Multi-Resolution & Multi-Touch Systems Page 14 of 41 i-m-Top System Peripheral Projector Fovea Projector IR Camera IR Illuminator Mirror Pan-tilt Unit 56” 96 ppi 24 ppi

15. A Framework for Multi-Resolution & Multi-Touch Systems Page 15 of 41 i-m-Top System

16. A Framework for Multi-Resolution & Multi-Touch Systems Page 16 of 41 Touch Detection i-m-Top System Background Subtraction Opening Down- scale Opening Connected Component Palm Tracking Connected Component PCA Analysis Palm-finger Association Finger Tracking Connected Component PCA Analysis Palm Association Finger Fusion Background Subtraction - - Opening - fingerpalm

17. A Framework for Multi-Resolution & Multi-Touch Systems Page 17 of 41 Outline  Introduction  Related Work  i-m-Top System  i-m-Top SDK Design Principles Architecture Implementation  Application  Conclusion & Future Work

18. A Framework for Multi-Resolution & Multi-Touch Systems Page 18 of 41 i-m-Top SDK Design Principles  Efficient prototyping Hide engineering details Use metaphors Provide reusable components  Touch-detection module independent Vision-based & electronic-based approaches  Real-time requirement Touch event handling + UI rendering  Extensibility Integrate with other components

19. A Framework for Multi-Resolution & Multi-Touch Systems Page 19 of 41 i-m-Top SDK Architecture Virtual Director Scene ViewNode Scene ViewNode Scene Node Viewport / Virtual Camera Position / Textures / Event Handlers / Animation Effects Global Coordinate Local Coordinate Node Virtual Camera (Fovea View) Virtual Camera (Peripheral View)

20. A Framework for Multi-Resolution & Multi-Touch Systems Page 20 of 41 i-m-Top SDK Architecture Scene Touch Detection View Application Node Other Sensors Projector PTU Speaker Socket Window Message COM Message Parser OpenGL c# Device Control Virtual Director Event queue LED Light CamNode

21. A Framework for Multi-Resolution & Multi-Touch Systems Page 21 of 41 i-m-Top SDK Loop of virtual director Check Inputs Update States Render Scenes Multimedia Output Affiliated Timer Functions Sleep 1/fps Time spent 1/fps-T T 0

22. A Framework for Multi-Resolution & Multi-Touch Systems Page 22 of 41 i-m-Top SDK Sample Code  A scene with a photo InitScenes( ){ oScene scene = new oScene();//construct a scene director.AddScene(scene);//add scene to director oView view = new View(0, 0, 1280, 720);//construct a view scene.AddView(view);//add view to scene oPhoto photo = new oPhoto(0,0,-100,640,480,”sun.png”);//construct a photo scene.AddNode(photo);//add to scene }

23. A Framework for Multi-Resolution & Multi-Touch Systems Page 23 of 41 i-m-Top SDK Implementation  Multi-resolution Handling  Touch-event Processing  Reusable Components

24. A Framework for Multi-Resolution & Multi-Touch Systems Page 24 of 41 Implementation Multi-resolution Handling  Calibration Homography of fovea & peripheral projections Run-time warping  Masking  Steerable fovea projection Region/Object oriented  Multi-presented Information

25. A Framework for Multi-Resolution & Multi-Touch Systems Page 25 of 41 Calibration  Homography of fovea & peripheral projections Provide tools for manual calibration Save homography/ center point /pan-tilt of PTU Implementation F1F1 F2F2 F3F3 F4F4 V1V1 V2V2 V3V3 V4V4 Y X V2V2 V1V1 V3V3 V4V4 P1P1 P4P4 P2P2 P3P3 Y X F1F1 F2F2 F3F3 F4F4 V1V1 V2V2 V3V3 V4V4

26. A Framework for Multi-Resolution & Multi-Touch Systems Page 26 of 41 Calibration  Run-time warping Adapt OpenGL rendering pipeline Implementation Projection Matrix Perspective Division Viewport Transformation Model-View Matrix Local coordinate Global coordinate Eye coordinate Clipping coordinate Normalized device coordinate Window coordinate Warping Homography Matrix

27. A Framework for Multi-Resolution & Multi-Touch Systems Page 27 of 41 Masking Implementation fovea peripheralfovea

28. A Framework for Multi-Resolution & Multi-Touch Systems Page 28 of 41 Masking  Find maximum region in fovea Utilize Stencil buffer in OpenGL Implementation Apply arbitrary masks Set ‘0’ in mask regions Set ‘1’ in the fovea region Invert pixels in fovea region

29. A Framework for Multi-Resolution & Multi-Touch Systems Page 29 of 41 Implementation Steerable Fovea Projection  Region-oriented Spotlight on a specific region Steered by finger gestures  Object-oriented Spotlight on a specific object Move with object  Real-time update Steer the mirror Retrieve homography Apply mask

30. A Framework for Multi-Resolution & Multi-Touch Systems Page 30 of 41 Multi-presented Information  Different content in fovea and peripheral regions Eg. Google Map Automatically change Implementation oRectangle multi_presentation = new oRectangle(0, 0,-100, 200, 200); multi_presentation.LoadTexture(“context.png");//load low-resolution image multi_presentation.LoadHighTexture(“detail.png");//load high-resolution image

31. A Framework for Multi-Resolution & Multi-Touch Systems Page 31 of 41 Touch-event Processing Virtual Director Implementation Touch Detection Pass Message Parser Node i-m-Top SDK Touch events ReceiveCheckDispatchHandle (by event handler)

32. A Framework for Multi-Resolution & Multi-Touch Systems Page 32 of 41 Implementation Touch-event Processing  Receive & parse Run on background threads  Pass to director Gather all events in one detection frame before pass if rendering rate < detection rate » Keep vital events: ‘Up’, ‘Down’ » Discard other events: ‘Move’ Virtual Director Pass Message Parser Node i-m-Top SDK Receive

33. A Framework for Multi-Resolution & Multi-Touch Systems Page 33 of 41 Implementation Touch-event Processing  Check ‘Down’: Test if hit some node By SELECTION mode in OpenGL If yes: add to TargetTable (event id → node) ‘Move’, ‘Up’: Check if there is a target Hit TargetTable by event id ‘Up’ : Remove from TargetTable  Dispatch Dispatch events to each node Virtual Director Dispatch Message Parser Node i-m-Top SDK Check

34. A Framework for Multi-Resolution & Multi-Touch Systems Page 34 of 41 Implementation Touch-event Processing  Define event handler Node decides how to act Default handler Move/Scale/Rotate under ‘Move’ events Redefine handler InitScenes( ){ //…… //add new event handlers to the photo object photo.MultiFingerEvent += new MultiFingerHandler(MyMultiFingerHandler); photo.FingerClick += new FingerClickHandler(MyFingerClickHandler); } Virtual Director Handle Message Parser Node i-m-Top SDK

35. A Framework for Multi-Resolution & Multi-Touch Systems Page 35 of 41 P4P4 Rotation: R Implementation P1P1 P2P2 P3P3 P4P4 Translation: T Scaling: S P 12 P3P3 P4P4 P3P3 P 34 P3P3 P4P4 P1P1 P2P2

36. A Framework for Multi-Resolution & Multi-Touch Systems Page 36 of 41 Implementation Reusable components  Multimedia Video/audio Camera capture  UI components Virtual keyboard Text-pad Button, scrollbar…

37. A Framework for Multi-Resolution & Multi-Touch Systems Page 37 of 41 Outline  Introduction  Related Work  i-m-Top System  i-m-Top SDK  Application  Conclusion & Future Work

38. A Framework for Multi-Resolution & Multi-Touch Systems Page 38 of 41 Application Map Browser  Different content in the fovea & peripheral Photo Browser  Object/Region oriented fovea projection Emotion Player  Interactive art c

39. A Framework for Multi-Resolution & Multi-Touch Systems Page 39 of 41 Outline  Introduction  Related Work  i-m-Top System  i-m-Top SDK  Application  Conclusion & Future Work

40. A Framework for Multi-Resolution & Multi-Touch Systems Page 40 of 41 Conclusion i-m-Top  Multi-resolution Accommodating to human vision Cost-effective large high-res display  Multi-touch Intuitive interaction i-m-Top SDK  Hide engineering challenges Rapid prototyping applications Investigating new interfaces / interactions

41. A Framework for Multi-Resolution & Multi-Touch Systems Page 41 of 41 Future Work  Attentive computing Combine with gaze-tracking  3D interaction Surface to space  Auto calibration Vision-based Continuous pan-tilt / homography map

42. Thanks for your listening