A Keystone-free Hand-held Mobile Projection System Li Zhaorong And KH Wong Reference: Zhaorong Li, Kin-Hong Wong, Yibo Gong, and Ming-Yuen Chang, “An Effective Method for Movable Projector Keystone Correction”, IEEE Transactions on Multimedia, VOL. 13, NO. 1, Feb keystone correction 1

Outline Introduction Methodology – Pro-cam pair calibration – Projection region detection and tracking – Automatic keystone correction Experimental results Conclusion keystone correction 2

Introduction keystone correction 3

Aim of this work A mobile projector keystone correction method – project keystone free content on a general flat surface – without adding markings or boundaries on the surface keystone correction 4

Aim of this work Configuration keystone correction 5

Aim of this work Desired Results Keystoned projectionCorrected projection keystone correction 6

Motivation Mobile projector becomes popular – Flexible projector-screen position – More freedom of display control such as viewing angle, distance etc – General flat surface can be used as screen – Physical size and cost shrinks quickly – Emergence of mobile devices with embedded projector – Others …… keystone correction 7

Motivation Limitation: keystone distortion ! – When projecting onto a screen at oblique position, the projection region becomes a trapezoid instead of a rectangle – Traditional static projection system also face this problem – Stringent for mobile projection system keystone correction 8

Motivation Existing keystone correction method – All for static projector system, not suitable for mobile projection, eg. Su [1]: requires bounded screen Li et al[2]: requires bounded screen Raskar [3]: high computation, not real-time One-time correction, not continuous keystone correction 9

Motivation Special need for mobile keystone correction – Screen independence no specially designed or position-fixed screen should be required – Continuous processing in real time Continuous correction instead of one-time correction is expected keystone correction 10

Our approach Propose an effective method that can continuously correct the keystone distortion on a general markless screen Only additional device used is a webcam attached with the projector keystone correction 11

Our approach We add a green frame to the projector screen and project it to the display screen Track the green frame use the camera and then automatically correct the keystone correction keystone correction 12

Contributions An artfully designed particle filter tracker for tracking projection region An efficient and accurate recovery algorithm for recovering 3d projection region A continuous and markless mobile projector keystone correction system keystone correction 13

Methodology keystone correction 14

Overview Three major modules – Projector-camera pair calibration – Projection region detection and tracking – Automatic keystone correction Flow chart keystone correction 15

Part 1: Projector-camera pair calibration Projector-camera relationship – Projection matrix from 3D camera coordinate system to projector image plane Benefits – Independent from projector movement – No need to estimate explicit parameters – Easy to estimate keystone correction 16

Projector-camera pair calibration Projective model : 2D projector image pixel : 3D point in camera frame : 3x4 projection matrix : scale factor keystone correction 17

Projector-camera pair calibration Estimation method – Project cross to an ordinary cardboard with known size – Collect a number of projector-camera corresponding crosses – Calculate the 3D coordinate of the cross – Estimate G matrix using SVD keystone correction 18

Part 2: Projection region detection and tracking Add a green frame to the projection region Detect the frame in the initial frame Track the frame in the subsequent frames keystone correction 19

Detection Based on the Canny edge map and Hough line segments Find a quadrangle satisfying following criteria – each side of the quadrangle longer than a threshold – opposite sides should have similar lengths – each angle within the range from 30 to 150 degree – the overlapping ratio of the line segments to the four sides of the formed quadrangle bigger than a threshold – the quadrangle approximately located around the center of the camera image keystone correction 20

Tracking The relationship between projector screen and the projection region in camera : the homography matrix : the projector image pixel : the camera image pixel keystone correction 21

Tracking The homography matrix : intrinsic parameter matrix of projector : rotation matrix of the projector w.r.t camera : translation vector of the projector w.r.t camera : normal of the screen : distance of the screen from the camera : intrinsic parameter matrix of camera H is dominated by n and d since J,R,t,K are all fixed keystone correction 22

Tracking Tracking state vector Particle filter tracking – State dynamic model – Observation model – Initialization keystone correction 23

Tracking State dynamic model : the state of k-1 and k frame : the uncertainty of the movement of the projector keystone correction 24

Tracking Observation model – Re-project each particle to the camera keystone correction 25

Tracking Observation model – Compute likelihood by comparing the re-projected quadrangle with the edge map Specifically, check how many edge points on each of four sides The likelihood of each side is the percentage of the edge points among all side points The likelihood of the particle is the sum of likelihoods of four sides keystone correction 26

Tracking Initialization – The detected quadrangle in the detection stage is used to initialize the particle filter – First, recover the 3D position of the quadrangle [using the method in Part 3] – Second, compute its normal and distance from the camera keystone correction 27

Part 3: Automatic keystone correction Three steps – Recover 3D projection region – Look for inscribed rectangle – Pre-warp projection image keystone correction 28

Recover 3D projection region Solve following equation set for each corner of the projection region using SVD : the corner of the projection region in camera : the 3D corner of the projection region : the corner of the projector screen keystone correction 29

Recover 3D projection region Incorporate co-planarity constraint : three rows of : weight Use the SVD solution as the initial value keystone correction 30

Look for inscribed rectangle Simple geometric intersection keystone correction 31

Pre-warp projection image Re-project inscribed rectangle into the camera image Find homography between the projection and projector screen Pre-warp the display image using the homography keystone correction 32

Pre-warp projection image Pre-warped projection imageDisplay result keystone correction 33

Experimental results keystone correction 34

Devices PC with 2.16GHz CPU, 1 GB memory Optoma mobile projector 1280x1024 Logitech Quickcam Pro 4000 webcam 320x240 keystone correction 35

Projector camera pair calibration Error distribution Back-projection error corresponding to 80% inliers is 4.2 pixels keystone correction 36

Projection region tracking Mean and std tracking error w.r.t different noise levels on simulation data keystone correction 37

Projection region tracking Mean and std tracking error on real data are 3.4 and 3.6 pixels Trajectory of the real projector movement keystone correction 38

Keystone correction Some real correction results keystone correction 39

Keystone correction The error of keystone correction against different poses of the projector keystone correction 40

Keystone correction Comparison of our keystone correction module with the static projector keystone correction keystone correction 41

Speed 16 fps on our platform The per-frame processing time is about 60 ms The pre-warping step occupies most of the time (about 36 ms) keystone correction 42

Conclusion Proposed an effective mobile keystone correction method Mobility and Markinglessness are the most distinguishing features Limitation: need to project a green frame Future work: Use invisible IR LED to eliminate the green frame keystone correction 43

Reference [1] R. Sukthankar, R. Stockton, and M. Mullin, “Smarter presentations: Exploiting homography in camera-projector systems,” in Intl. Conf. on Computer Vision, [2] B. Li and I. Sezan, “Automatic keystone correction for smart projectors with embedded camera,” in Intl. Conf. on Image Processing, 2004 [3] R. Raskar and P. Beardsley, “A self-correcting projector,” in Intl. Conf. on Computer Vision and Pattern Recognition, 2001 [4] Zhaorong Li, Kin-Hong Wong, Yibo Gong, and Ming-Yuen Chang, “An Effective Method for Movable Projector Keystone Correction”, IEEE Transactions on Multimedia, VOL. 13, NO. 1, Feb keystone correction 44

Thank you keystone correction 45