Patch Based Synthesis for Single Depth Image Super-Resolution (ECCV 2012) Oisin Mac Aodha, Neill Campbell, Arun Nair and Gabriel J. Brostow Presented By: Itzik Ben ShabatJanuary 2014

Contents  Problem & Motivation  SR General Overview  Related Work  The Proposed Method  Results  Qualitative  Quantitative  Future Work  Paper review

Problem & Motivation  How do we convert a Low Resolution (LR) image to High Resolution (HR) ?  Get a better camera (Sensor)

Problem & Motivation  Cant get a better camera? Super Resolve the image! (SR)

Problem & Motivation  Now do it in RGB-D ! ! ! PMD CamCube - 200X200 PointGrey BumbleBee x480 at 48fps MS Kinect - 640x480

SR General Overview Common approaches:  Take multiple LR images from different angles and reconstruct the additional information (requires multiple images)

SR General Overview Common approaches:  Use a LR to HR database (requires a database)  Focus on this approach

Related Work  Intensity Images  EbSR [15] - Freeman, W.T., Liu, C.: Example-based super resolution. In: Advances in Markov Random Fields for Vision and Image Processing. MIT Press (2011)  Similar:  Filter input  Normalized Patch matching  Solving minimum energy problem (using BP)  Different  Not Designed for RGB-D images  Matching HR and interpolated LR patches Create LR- HR database Perform bicubic interpolation on input image Desaturate and High- pass filtered Normalize contrast Solve patch minimum energy problem using BP Add back low frequency and color

Related Work  EbSR looking closer Ground trouth EbSR Output

 Intensity Images  ScSr [17] - Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Transactions on Image Processing (2010)  Similar:  Use patches and minimization problem  Different:  Not designed for RGB-D images  Two dictionaries  Database structure (sparse representation)  Solves 2 minimization problems separately – global and local  No noise reduction implementation Learn 2 dictionaries (LR- HR) For each input patch - Find LR representation Apply representation to HR pairs Solve global optimization problem Related Work

 Depth + Intensity Hybrids  Cross Bilateral [1] - Yang, Q., Yang, R., Davis, J., Nister, D.: Spatial-depth super resolution for range images. In: CVPR. (2007)  Similar:  Specific for RGB-D  Use bilateral filter  Different:  Requires additional input (destination resolution image)  Doesn’t use patches  Solves a fusion problem

Construct Multi- Resolution MRF grid Formulate optimization problem Bilinear filter for y initial guess Solve iteratively Related Work  Depth + Intensity Hybrids  MRF SR [25]- Diebel, J., Thrun, S.: An application of markov random fields to range sensing. In: NIPS. (2005)  Similar:  Uses MRF  Different:  Uses multi-resolution MRF  Requires additional input (destination resolution image)  Doesn’t use patches  Solves a fusion problem

The Proposed Method  Challenges  Construct database  Noise  “Flying pixels” at discontinuities  Wrong depths for specular or dark materials  Edges – jarring artifacts (different than rgb image)

The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

The Proposed Method – Database  Constructing the database  Less sources for database construction than rgb images  Considered synthetic Vs. Real datasets  Database uses 30 scenes of 800x800 (scenes flipped left to right) – 5.3 million patches  Pruning to remove redundant patches (planar surfaces) SyntheticLaser Scan

The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

The Proposed Method - Filtering  Noise Reduction  Assumption – High frequency=noise  A. Bilateral filter on input patches before patch normalization  Edge preserving  Noise reducing  Nonlinear  Weighted average of intensity values from nearby pixels  *Used in Adobe Photoshop “Blur” function  B. Bicubic filter on database HR patches before down-sampling

The Proposed Method - Filtering Input After Bilateral filtering

The Proposed Method - Filtering  Noise Reduction  Pro – Cleaner image for patching  Con – Some data is lost

The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

The Proposed Method – Matching Input Depth Image

N non overlapping low resolution input patches x i For each x i we wish to find its corresponding high resolution y i Patches are normalized The Proposed Method – Matching Input Depth Image

The Proposed Method – Matching High Resolution Database

Output Depth Image The Proposed Method – Matching Input Depth Image High Resolution Database

Output Depth Image High Resolution Database The Proposed Method – Matching Input Depth Image

Output Depth Image High Resolution Database The Proposed Method – Matching

Output Depth Image … The Proposed Method – Matching Input Depth Image High Resolution Database

Output Depth Image High Resolution Database … The Proposed Method – Matching Input Depth Image

Output Depth Image … The Proposed Method – Matching Input Depth Image High Resolution Database

Output Depth Image … The Proposed Method – Matching Input Depth Image High Resolution Database

Output Depth Image … … The Proposed Method – Matching Input Depth Image High Resolution Database

The Proposed Method - Matching  Matching Patches to database  Matching is done between LR patches  Kd tree is used for speeding up the process

The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

The Proposed Method - Reconstruction  Solving minimum energy problem  Solved using TRW-S algorithm (based on belief propagation)  E d -Unary Potential - Difference between normalized matching LR patches

The Proposed Method - Reconstruction  E s -Pairwise Potential - Difference between un-normalized HR patch overlaps

The Proposed Method - Reconstruction yiyi yjyj  E s - Pairwise Potential -

yiyi yjyj The Proposed Method - Reconstruction  E s - Pairwise Potential -

yiyi yjyj The Proposed Method - Reconstruction  E s - Pairwise Potential -

yiyi yjyj The Proposed Method - Reconstruction  E s - Pairwise Potential -

-- E s =( + ( )2)2 )2)2 yiyi yjyj The Proposed Method - Reconstruction

 Normalization – is un-normalized based on the input patch min and max values

The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

The Proposed Method – Filter Results  Noise reduction  C. Post processing Denoising – Outlier detection and correction using threshold Result Result after denoising input

Results - Qualitative  Exp 1:  Used Middleburry stereo dataset  Down-sampled the ground truth (X2,X4)  Reconstructed  Compared RMSE  Exp 2:  3 laser scans  Upsampled by 4  Ground touth comparison  Exp 3:  Use synthetic Vs. real database

Results - Qualitative  Proposed method Vs. Other Methods (Exp. 2) Proposed Method

Results - Qualitative  Proposed method - Real Vs. Synthetic training data (Exp. 3)

Results - Quantitative  Reminder: MRF RS Cross BilateralScSREbSR Upsampling factor Method used RMSE RGB-D image used Method used RGB-D image used

Results - Qualitative Results Movie

Conclusions  1 st or 2 nd best from intensity based methods  MRF RS, Cross Bilateral are better but require more data  Speed is not realtime compatible  Super resolving moving depth videos  Synthetic data exhibit better results than scanned data for training.

Future work  Extend to exploit temporal context (in video)  Exploit context when querying the database  Develop a sensor specific noise model for better results

Paper review  Pros  Self contained  Well referenced  Novel approach  Available resources – website, data, code  Good experimental results  Cons  Non chronological order of subjects (training after method, implementation notes after results, etc.)  Supplied code insufficient

Questions?

References  Some of the slides and images were taken from the original paper presentation, paper and website 

Appendix - filters  Bicubic filter -  Interpolate data points on a two dimensional regular grid  Smoother  Fewer interpolation artifacts  Uses 16 pixels (4X4) Bicubic Bilinear Nearest neighbor