Yu-Wing Tai, Hao Du, Michael S. Brown, Stephen Lin CVPR’08 (Longer Version in Revision at IEEE Trans PAMI) Google Search: Video Deblurring Spatially Varying Deblur Image/Video Deblurring using a Hybrid Camera Project Page (visit): 1 / 25

Image Deblurring: The Problem Given a motion blurred image, we want to recover a sharp image: Input Desired Output Point Spread Function (PSF) Motion blur Kernel 2 / 25

Blur kernel is known: Why this is a difficult problem ? This is an ill-posed under constrained problem: Different inputs can produce the same (very similar) output after convolution 3 / 25

Blind deconvolution problem Blur kernel is unknown: 4 / 25

Two causes for motion blur Hand shaking (Camera “ego motion”) Object motion Blur is the same Blur is different 5 / 25

Properties of motion blur Hand shaking PSF is globally the same for the whole image Observations are the whole image Deconvolution is a global process Relatively Easy – ``Well studied, some current works produce very good results’’ Object Motion PSF is varying across the whole image Observations are only valid for local regions Deconvolution is a local process Have problem of mixing colors Might have problem of occlusions and disocclussions Very Difficult – ``Nothing closed, there is still have no good solution’’ 6 / 25

Related works (Hand shaking) Traditional approaches: Wiener filter [Wiener, 1949] Richardson and Lucy [Richardson 1972; Lucy 1974] Recent approaches: Regularization based: Total variation regularization [Dey et al. 2004] Natural image statistics [Fergus et al. Siggraph 2006] Alpha matte [Jia CVPR 2007] Multiscale regularization [Yuan et al. Siggraph 2008] High-order derivatives of gaussian model [Shan et al. Siggraph 2008] Auxiliary information: Different exposure [Ben-Ezra and Nayar, CVPR 2003] Flutter shutter [Raskar et al. Siggraph 2006] Coded aperture and sparsity prior [Levin et al. Siggraph 2007] Blurred and noisy pairs [Yuan et al. Siggraph 2007] Two blurred Images [Rav-Acha and Peleg2005; Chen and Tang CVPR2008] arg min I,K f(I◦K – B) arg min I,K f(I◦K – B) + Regularization Terms arg min I,K f(I◦K’ – B’) + Regularization Terms 7 / 25

Related works (Object motion) Translational motion Natural image statistics [Levin, NIPS 2006] Two blurred Images [Cho et al. ICCV 2007] Motion Invariant Photography [Levin et al., Siggraph 2008] In-plane rotational motion Shan et al. ICCV 2007 Our approach [CVPR 2008] Handle motion blur from both hand shaking and object moving Handle translational, in-plane/out-of-plane rotational, zoom-in motion blur in a unified framework 8 / 25

Basic idea [Ben-Erza CVPR’03] Observation: Tradeoff between Resolution and Exposure Time Motion blur exist in high resolution images. time Hi-Resolution Low Frame-rate Low-Resolution Hi-Frame-rate Our goal is to deblur the high resolution images with assistance from low resolution, high frame rate video. 9 / 25

Our Hybrid Camera Hi-Res: 1024 x768 resolution at 25 fps Low-Res: 128 x 96 resolution at 100 fps. A beam-splitter is use to align their optical axes Dual-video capture synchronized by hardware Low-Res Camera High-Res Camera Beam-splitter 10 / 25

Low-Resolution High Frame-rate Spatially-varying motion blur kernels can be approximated by motion vector from low resolution video Observation 1 Motion Blur Kernels K Hi-Resolution Low Frame-rate 11 / 25

The deblured image, after down-sampling, should look similar to the low resolution image Observation 2 Deblurred Hi-Resolution Image Low-Resolution Image 12 / 25

Bayesian ML/MAP model: I : Deblurred Image K: Estimated Blur Kernel I b : Observed High Resolution Blur Image I l : Observed Low Resolution Shape Image Sequences K o : Observed Blur Kernel from optical flow computation Our Formulation (Main Algorithm) 13 / 25

Optimization Procedure Global Invariant Kernel (Hand Shaking) Spatially varying Kernels (Object Moving) Deconvolution Eq. Low Resolution Reg. Kernel Reg. 14 / 25

Moving object appears sharp in the high-frame- rate low-resolution video Perform binary moving object segmentation in the low-resolution images Compose the binary masks with smoothing to approximate the alpha matte in the high-resolution image Moving Object Extraction Problem with mixing color 15 / 25

Results Image Deblurring: Hand-shaking Motion Blur (Global Motion) In-plane Rotational Motion Blur Translational Motion Zoom-in motion Video Deblurring Moving box: arbitrary in-plane motion Moving car towards camera: translational + zoom in motion 16 / 25

Hand Shaking (Motion blur = Global) Results Input[Fregus et. al. Siggraph’06][Ben-Ezra et. al. CVPR’03] Back ProjectionOur ResultGround Truth 17 / 25

Rotational Motion (Motion Blur = Spatially-varying) Results Input[Shan et. al. ICCV’07][Ben-Ezra et. al. CVPR’03] Back Projection Our ResultGround Truth 18 / 25

Translational Motion (Motion Blur = Global for object) Results Input[Fregus et. al. Siggraph’06][Ben-Ezra et. al. CVPR’03] Back ProjectionOur ResultGround Truth 19 / 25

Zoom-in motion (Motion Blur = Spatially-varying) Results Input[Fregus et. al. Siggraph’06][Ben-Ezra et. al. CVPR’03] Back ProjectionOur ResultGround Truth 20 / 25

Results (moving object) In-plane Rotation 21 / 25 [Show video]

Results (moving object) Out-of-plane Motion (zoom translate) 22 / 25 [Show video]

Limitations and Discussion High frequency lost during the convolution process cannot be recovered Small ringing artifacts cannot be removed Basic assumptions: Constant Illumination during exposure Rigid objects Moving objects are not overlapped Problems in separating moving objects from moving background 23 / 25

Hybrid camera framework Extended to spatially varying motion blur Extended to video Combined Deconvolution and Backprojection Effective in reducing ringing artifacts Effective in recovering motion blurred details Formulated into a Bayesian ML/MAP Solution Summary of Image/Video Deblurring 24 / 25

Personal Homepage: Thank you! (Question/Answers) 25 / 25