1. IITB-Monash Research Academy An Indian-Australian Research Partnership IIT Bombay Projection Defocus Correction using Adaptive Kernel Sampling and Geometric Correction in Dual-planar Environments Shamsuddin Ladha, Sharat Chandran, Kate Smith-Miles

2. IITB-Monash Research Academy www.IITBMonash.org Outline Goal Features and Contributions Solution Framework Experiments and Results

3. IITB-Monash Research Academy www.IITBMonash.org Goal Create visually appealing projection on dual-planar surfaces Artifacts to correct Defocus blur Geometric distortion

4. IITB-Monash Research Academy www.IITBMonash.org Why? R. Raskar, G. Welch, M. Cutts, A. Lake, L. Stesin, and H. Fuchs. The office of the future: A unified approach to image-based modeling and spatially immersive displays. ACM SIGGRAPH:179-188, 1998. Humans crave for large displays Real estate is seldom available The ubiquitous office cubicle is dual-planar! Can we replace the desktop monitor(s) by a projector?

5. IITB-Monash Research Academy www.IITBMonash.org Features and Contributions Non-parametric blur estimation No assumption of exemplar (in-focus) region No explicit depth computation Sparse and adaptive sampling for defocus kernel computation Correction for both luminance and chrominance channels Geometry correction

6. IITB-Monash Research Academy www.IITBMonash.org Solution Framework P1 Q1 Projector P(x) Lambertian Dual-planar Display Surface Q2 Camera I op z In-focus point Not in-focus points f(x; z) – Defocus kernel Γ – Ambient light radiance I op = α f(x;z)P(x) + Γ L. Zhang and S. Nayar. Projection defocus analysis for scene capture and image display. ACM Transactions on Graphics, 25(3):907-915, 2006.

7. IITB-Monash Research Academy www.IITBMonash.org Solution Framework P1 Q1 Projector P(x) Q2 Camera (I op ) z In-focus point Not in-focus points

8. IITB-Monash Research Academy www.IITBMonash.org Solution Framework P’ = arg min{d( α fP + Γ; I ip ) P(x) є [0, 255]} Project P(x) = I ip then I op ≠ I ip Project P’ = (αf ) -1 * (I op - Γ) then I op = I ip Unknowns: α f and Γ

9. IITB-Monash Research Academy www.IITBMonash.org Kernel Estimation Incorrect kernel estimates near the edge Can we do sparse and geometry aware sampling? Project point patterns and measure Point Spread Function (PSF) L. Zhang and S. Nayar. Projection defocus analysis for scene capture and image display. ACM Transactions on Graphics, 25(3):907-915, 2006. PSFs as columns of F Defocus kernel

10. IITB-Monash Research Academy www.IITBMonash.org Sparse Sampling With increasing distance blur radius can be linearly approximated Sample few points and interpolate S. Chaudhuri and A. Rajagopalan. Depth from defocus: a real aperture imaging approach. Springer, 1999.

11. IITB-Monash Research Academy www.IITBMonash.org Adaptive Sparse Sampling Measure PSF at few points on each plane Measure PSF at the edge (adaptive) Compute geometry in image and projector spaces!

12. IITB-Monash Research Academy www.IITBMonash.org Geometry in Image Space K. Paidimarri and S. Chandran. Computer Vision, Graphics and Image Processing, pages 289-298. Lecture Notes in Computer Science, Springer 2006. Project structured light Detect bend points Fit a line Detect the edge between two planar surfaces in image space

13. IITB-Monash Research Academy www.IITBMonash.org Geometry in Projector Space Find corresponding points in image and projector space for each planar surface Compute forward and inverse homography matrices Find the edge in projector space Sparse adaptive sampling can be done!

14. IITB-Monash Research Academy www.IITBMonash.org Geometric Correction Display scene surface Corrected input Corrected image (camera) Inv(H L ) Inv(H R ) HLHL HRHR Original input

15. IITB-Monash Research Academy www.IITBMonash.org Summary We have seen Compute defocus kernels and measure Γ Perform geometry correction Next Improve color fidelity of the corrected image

16. IITB-Monash Research Academy www.IITBMonash.org Better Distance Metric L. Zhang and S. Nayar. Projection defocus analysis for scene capture and image display. ACM Transactions on Graphics, 25(3):907-915, 2006. P’ = arg min{d( α fP + Γ; I ip ), P(x) є [0, 255]} Sum of squared pixel difference R, G, B channels treated independently Humans differently sensitive to luminance and chrominance Do not treat a pixel in isolation – consider its neighborhood Neighborhood based distance metric in YCbCr space!

17. IITB-Monash Research Academy www.IITBMonash.org (absolute luminance error) (absolute chrominance error) (spatial luminance discontinuities error) (spatial chrominance discontinuities error) non-negative weighting parameters Better Distance Metric Y. Sheng, T. Yapo, and B. Cutler. Global illumination compensation for spatially augmented reality. In Computer Graphics Forum:387-396,2010 ‒ ‒ ‒

18. IITB-Monash Research Academy www.IITBMonash.org Experiments One time offline processing Run-time correction Homography matrices Defocus kernel matrix Measurement of Γ Pre-warp and stitch image Optimal image using steepest descent

19. IITB-Monash Research Academy www.IITBMonash.org Results

20. IITB-Monash Research Academy www.IITBMonash.org Results

21. IITB-Monash Research Academy www.IITBMonash.org Results

22. IITB-Monash Research Academy www.IITBMonash.org Results (Quantitative) Edge Pixels Input Image Corrected Input Image Output Image Corrected Output Image Cat181081240577167383218688 Plant10959518240092347118577 Dog681191153296751278905 Leaf176973212434180990322382

23. IITB-Monash Research Academy www.IITBMonash.org Summary A better method Projection defocus correction Geometric distortion correction Future work Indirect illumination compensation Composite framework for all corrections

24. IITB-Monash Research Academy www.IITBMonash.org References 1.K. Paidimarri and S. Chandran. Computer Vision, Graphics and Image Processing, pages 289-298. Lecture Notes in Computer Science, Springer 2006. 2.L. Zhang and S. Nayar. Projection defocus analysis for scene capture and image display. ACM Transactions on Graphics, 25(3):907-915, 2006. 3.Y. Sheng, T. Yapo, and B. Cutler. Global illumination compensation for spatially augmented reality. In Computer Graphics Forum:387-396,2010. 4.R. Raskar, G. Welch, M. Cutts, A. Lake, L. Stesin, and H. Fuchs. The office of the future: A unified approach to image-based modeling and spatially immersive displays. ACM SIGGRAPH, 32(Annual Conference Series):179- 188, 1998. 5.S. Chaudhuri and A. Rajagopalan. Depth from defocus: a real aperture imaging approach. Springer, 1999.

25. IITB-Monash Research Academy www.IITBMonash.org Thank You