Presentation is loading. Please wait.

Presentation is loading. Please wait.

A Projector Based Hand-held Display System

Similar presentations

Presentation on theme: "A Projector Based Hand-held Display System"— Presentation transcript:

1 A Projector Based Hand-held Display System
LEUNG Man Chuen WONG Kin Hong

2 Outline Introduction Previous work System overview System detail
Implementation and Experimental Results Limitations and discussions View dependent projection and application Conclusion Question and Answer Session

3 Introduction

4 Introduction Aim of this project: Desired result:
Build a movable hand-held display system Uses an ordinary cardboard as the display Uses a projector to project display content Desired result:

5 Introduction Motivation:
Popular forms of graphical interfaces are usually in fixed positions Monitor Projection onto a wall / screen Hand-held display device gives users greater freedom of control Viewing angle, viewing distance

6 Introduction Motivation Screen of hand-held display devices are small
Mobile phone, portable DVD player, iPod… Increasing the size of the screen => heavy ! Electronic paper Technology not very mature yet Not available at low cost Therefore, an alternative solution is needed!

7 Introduction Our approach
Use a projector-camera pair to serve as the input and output devices Use a white cardboard as the movable display surface Can be light weight Flexible screen size

8 Introduction Contributions
Proposed a computer vision approach to enable the projector to project display content precisely onto a movable ordinary cardboard in 3D space Projector-camera calibration method to find the projection matrix from a 3D point in camera coordinate to 2D projector image coordinate Real-time quadrangle detection and tracking algorithm to detect and track the display in real time Experimental results show that projection content can be registered onto the hand-held display precisely This work has been published in the Proc. Of CVPR2009

9 Contributions (con’t)
Proposed an application with view dependent projection “Hand-held 3D Model Viewer” Use a head-mounted camera to track the viewing position Project views of a 3D model according to user view Give user an perception of a real object sitting on the display

10 Previous work

11 Previous work Sensor based methods
Dynamic Shader Lamp [Bandyopadhyay et al], magnetic sensor Free Form Projection Display [Kondo et al], magnetic sensor PaperWindows [Holman et al], infrared reflective marker Foldable Interactive Display [Lee et al], IR emitter Projector Based Tracking [Lee at al], light sensors Disadvantages: Need to attached sensors or special markers onto the projection surface Accurate tracking systems are very expensive Magnetic location tracker Vicon Motion Capturing System, for infrared reflective marker

12 Previous work Vision based methods
Portable Display Screen [Borkowski et al] Put the rectangular screen to a particular position for detection Use Kalman filter to track the motion Register the image to the portable screen using homographies: A homography is a 3x3 matrix that match a projective plane to a projective plane that maps straight lines to straight lines Disadvantages: Hpc is calibrated on an initial plane, not updated at each time step Mismatch when the screen move out of the calibration plane Hps: projector-screen homography Hcs: camera-screen homography Hpc: projector-camera homography

13 Previous work Vision based methods (con’t)
Active Pursuit Tracking [Gupta et al] Calibrate Hpc on an initial plane Put at least four color markers onto the projection screen Detect these four color markers in camera image Project four virtual markers onto the projection screen according to the detected real markers and previous Hpc Update Hpc using the image points of the virtual markers Disadvantages: Color markers are needed to be put on the projection screen Virtual markers are needed to be projected onto the screen constantly Make the projection screen unnecessarily large Once lose tracking, need to start from the calibration plane again

14 System overview

15 System overview Real-time quadrangle detection and tracking
Cardboard size, Camera parameters Image frame Real-time quadrangle detection and tracking Pre-warp display content Projection Offline calibration of projector-camera pair Gp Relative pose (R, T) Pre-warped display content

16 System detail

17 Part 1: New projector-camera pair calibration
What to calibrate? Gp (3x4)

18 Part 1: Projector-camera pair calibration
1. Create image points in known positions 2. Project onto a cardboard with known size 3. Capture image using camera 4. Calculate 3D positions of the corners 7. Collect a set of corresponding points 6. Calculate 3D positions of projected points 8. Solve Gp

19 Part 2: Quadrangle detection and tracking
Line feature extraction Line detector based on Hough transform in OpenCV Open source Computer Vision library We detect line segments Hough Transform in OpenCV Input: Image Edge map Line feature map

20 Part 2: Quadrangle detection and tracking
Select long line segments Check for every 4 segments Find the formed quadrangle Check criteria Based on property of a quadrangle, angles, ratio of side lines…

21 Part 2: Quadrangle detection and tracking
Quadrangle tracking Using particle filter An state estimation algorithm Two main components State dynamic model Observation model To avoid degeneracy Re-sampling Output The weighted sum of samples Initial samples set State update Updated samples Evaluation function Weighted samples Re-sampling New sample set

22 Part 2: Quadrangle detection and tracking
Quadrangle tracking (cont’) Tracking target: relative pose(R, T) of the camera and the cardboard Calculated from detection result State at time k: State dynamic model Random walk [Pupilli] based on uniform density Observation Line segment feature map of current frame rx : rotation along x-axis ry : rotation along y-axis rz : rotation along z-axis tx : translation along x-axis ty : translation along y-axis tz : translation along z-axis p : density function qk-1 : state at time k-1 qk : state at time k U : uniform density function e : uncertainty about movement

23 Part 2: Quadrangle detection and tracking
Evaluation method Re-projection

24 Part 2: Quadrangle detection and tracking
Evaluation method closest line matching and criteria checking

25 Part 2: Quadrangle detection and tracking
Evaluation method Particle content replacement

26 Part 2: Quadrangle detection and tracking
Evaluation method Weight update According to quadrangle detection criteria Re-sampling Handling tracking failure Determination of tracking failure Use distribution particles If variance is greater than a certain threshold Lost tracking ! Perform detection again

27 Part 3: Pre-warping and projection
From tracking result Get 3D corner positions From calibration result Get projector image points Warp display content Project onto the plane T T T Image to be displayed Projector image

28 Implementation and experimental results

29 System setup Projector camera pair Testing platform White cardboard
Projector: 1280x1024 Camera: 320x240 Testing platform Dual core processor at 2.16GHz 1GB RAM White cardboard 351mm x 300mm

30 Calibration of projector-camera pair
Project one point each time 32 images are collected Mark the points manually Sample images

31 Calibration of projector-camera pair
Calibration error: About 4.6pixels pixel O : projection points + : re-projection points

32 Detection and tracking experiments
High tracking precision Robust to partial occlusions Robust to dense clutter Works well when both camera and cardboard are moving

33 Projection results Display content can be projected precisely onto the cardboard Match to the cardboard in different poses

34 Processing time Processing time for line feature extraction
~16ms Processing time of tracking algorithm Depends on number of line segments used Depends on number particles

35 Processing time Processing time of tracking algorithm
About 30ms/frame for 80 particles and 20 line segments About 30fps Processing time for image warping Negligible Total processing time ~ 46 ms ~20 fps

36 Projection latency Estimation method:
Move the cardboard from left to right Stop sharply, wait until the projection image match the edge Repeat for several times Use an external video camera to capture the process Count the number of frames between stopping the motion and the projection image match the edge of the cardboard Average about 5 frames, 167 ms

37 View Dependent Projection and Application

38 View dependent projection
Project according to view of the user Head pose tracking Mount a magnetic location tracker on the head of the user [Cruz-Neira et al], [Kondo et al] Expensive Tracked pose is relative to the receiver, not to the cardboard directly Mount a fixed camera in front of the user Track the eye positions of the user Track point model head pieces Not very accurate and easily be occluded

39 View dependent projection
Our approach Use a head mounted camera Apply the proposed quadrangle detection and tracking algorithm Advantage: accurate and get the relative pose directly

40 View dependent projection
Application: Hand-held 3D Model Viewer Project different views of a 3D model according to head poses User can change the view by moving the display or his/her head Projection is distortion compensated Give the user an illusion of a real object sitting on the display Motivation Traditional way of interacting with a virtual model: Mouse and keyboard Natural way of interacting with a real object: Hold the object in the hands and interact with it directly To simulate such natural way of interacting with objects

41 Design detail Create virtual scene Create user view image
Implemented using OpenGL An virtual object sitting on an virtual cardboard Create user view image Set a virtual camera Apply tracked head pose Generate virtual camera image Equivalent to desired user view

42 Design detail Generate projector image

43 Experimental results Use another webcam to be the head-mounted camera
Move the cardboard and the camera to different relative poses Capture the view of this webcam Result video:

44 Discussion 44

45 Discussion Limitation on projection resolution
Down sampled in the image warping process Possible solutions : Warp physically , need a 6dof optical redirect system Use a projector with higher resolution Limitation on depth of field About 30 cm in our prototype Possible solution: Use multiple projectors 45

46 Discussion Handling projected light Hand-held 3D Model Viewer
Projected light may affect tracking stability Solved using property of camera: Set the contrast and gain of light to high value Projected light on cardboard is brighter - > saturated Hand-held 3D Model Viewer Limitation on speed of motion Fast motion will break the interaction experience because of projection latency Acceptable error: ~4cm in translation, ~5 degrees in rotation Limited motion: ~24cm / second, ~30 degrees / second 46

47 Discussions Limitation on viewing angle

48 Discussions Possible extensions Use multiple cardboards
Project onto a cube instead of a cardboard

49 Conclusion

50 Conclusion Proposed a projector-based hand-held display system
Proposed a novel computer vision approach to guide the projector to project onto a hand-held display in the 3D space Calibration method to find the projection matrix from 3D camera coordinate system to 2D projector image coordinate system Robust quadrangle detection and tracking algorithm Precise and can run in real time Implementation of the whole system Project onto the cardboard correctly in real-time An interactive application with view dependent projection is proposed “Hand-held 3D Model Viewer” Use head-mounted camera to obtain user view to the display Give 3D perception to user

51 Question and Answer Session

Download ppt "A Projector Based Hand-held Display System"

Similar presentations

Ads by Google