1. An Interactive Background Blurring Mechanism and Its Applications NTU CSIE Yan Chih-Yu Advisor: Wu Ja-Ling, Ph.D. 1

2. 2

3. Outline Introduction Related Work Method ◦ Object Segmentation ◦ Depth Map Generation ◦ Image Defocus Experimental Result Applications Conclusion 3

4. Introduction Real world Photo plane Depth of field ◦ Circle of confusion 4

5. Photo plane Introduction Real light Circle of Confusion Out of focus Depth of field ◦ Circle of confusion 5

6. Introduction Depth of field 6 Real light Circle of Confusion - Readability range Focus plane Photo plane Out of focus – blur range Depth of field

7. Introduction Real world Photo plane Shallow focus 7

8. Introduction ◦ Highlight the subject by softening background diffusion Deep focus by DC NIKON E4300 (2003) Shallow focus by DSLR NIKON D90(2008) 8

9. Method Lazy Snapping Alpha Matting Face Detection Perspective Box Pop-up Card Camera Settings Defocus Blur Privacy Preserving Photo Browser Object Segmentation Depth Map Generation Applications Image Defocus 9

10. Method Segment ◦ Lazy snapping ◦ Mean shift ◦ Alpha matting ◦ Face detection Resize to 20% Do hard graph cut by lazy snapping User Stroke Mean shift 10 Do hard graph cut by lazy snapping User Stroke Resize to 20% Mean shift Segmentation tri-Map Segmentation alpha Map Lazy Snapping. ACM Trans. On Graphics 2004. Mean Shift: A Robust Approach Toward Feature Space Analysis. PAMI 2002. A Bayesian Approach to Digital Matting. CVPR 2001.

11. Method Depth map ◦ Perspective Box  Vanish point  Rear wall Depth Map Perspective Box Normal vector Pop-up card 11 Depth Map Perspective Box Normal vector Pop-up card Tour Into the Picture: Using a spidery mesh user interface to make animation from a single image. SIGGRAPH 1997

12. Method Depth map ◦ Pop-up card Depth Map Perspective Box Normal vector Pop-up card 12 Depth Map Normal vector Pop-up card

13. Method Depth map ◦ Normal vector Depth Map Perspective Box Normal vector Pop-up card 13 Depth Map Normal vector

14. Method Image defocus ◦ Blur circle diameter Blur circle diameter Segment Map Depth Map Camera setting Aperture size Focal length of the lens Distance of focus Aperture size Focal length of the lens Distance of focus uv b d vbvb ubub uv b d vbvb ubub (a)(b) 14

15. Method Defocus blur ◦ bokeh 15

16. Method 16 Defocus blur

17. Method ◦ Near-by object case 17 Focus on the flower

18. ◦ Near-by object case Method Blur circle diameter Objects in front of the focus Objects behind the focus Objects behind the focus Defocus Blur Text synthesis inpainting Blur shape as alpha map Defocus Blur Interpolation by alpha map Shallow focus image 18

19. Experimental Result 1 Defocus blur method proposed in the system comparing with other blur filter results 19 Deep focus Shallow focus

20. Experimental Result 1 Defocus blur method proposed in the system comparing with other blur filter results Gaussian blur Defocus blur Defocus blur + Bokeh 20

21. Experimental Result 1 21

22. Defocus blur Experimental Result 1 22 Gaussian blur Defocus blur + Bokeh Original photograph Defocus blur method proposed in the system comparing with other blur filter results

23. Experimental Result 2 with / without depth variation in the background Real photograph taken by DSLR 23

24. Experimental Result 2 with / without depth variation in the background Original photograph Result after post-processing 24 Depth variation Depth fixed

25. Experimental Result 3 ◦ Near-by object case 25 Focus on the flower

26. ◦ Near-by object case Without inpainting Interpolation result Experimental Result 3 26

27. Experimental Result 4 Comparing with Photoshop 27 Photoshop (1 Hour) Our system (3~5min) Original image

28. Applications - Privacy Preserving 28

29. Applications - Partial Viewing 29

30. Applications - Partial Viewing 30

31. Applications - Image Transition at Photo Browser 31

32. 32 Applications - Image Transition at Photo Browser

33. Movie with defocus blur effect 33 Applications - Image Transition at Photo Browser

34. 34 Concatenate two unrelated images

35. Conclusion We proposed an interactive refocusing tool for background blurring ◦ Simple user hint ◦ Defocus blur kernel ◦ Concatenate two related picture Future work ◦ concatenate two or more unrelated pictures  Color based image retrieval technique 35

36. Q&A 36

37. Experimental Result 1 Defocus blur method proposed in the system comparing with other blur filter results 37 Deep focus Shallow focus Difference

38. Experimental Result 1 Defocus blur method proposed in the system comparing with other blur filter results 38 Shallow focus Difference Defocus blur PSNR:22.79 Defocus blur PSNR:22.79 Gaussian blur PSNR:20.34 Gaussian blur PSNR:20.34

39. END THANK YOU 39