1. Light Field Compression Using 2-D Warping and Block Matching Shinjini Kundu Anand Kamat Tarcar EE398A Final Project 1 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

2. Outline Motivation and Goals Overview of Our Method Results and Analysis Summary Future Work References 2 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

3. Motivation Light field images are used in computer graphics to compute new views of a scene without need for scene geometry model 1. Need to compress large set of images Exploit inter-view coherence to achieve compression. 1. M. Levoy and P. Hanrahan, “Light field rendering,” in Computer Graphics (Proceedings SIGGRAPH 96), August 1996, pp. 31-42. 3 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

4. Light Fields Represents a 3D scene or object from all viewing positions and directions – 2D array of 2D images – Difficult to Acquire – Very Large Perfect representation requires images of the order of the resolution

5. Light Field Views EE398A - Compression of Light Fields using 2-D Warping and Block Matching 5

6. Light Field Data Set 8.4 MB uncompressed data sets http://lightfield.stanford.edu/aperture.swf?lightfield=data/lego_lf/preview.zip&zoom=1 Credit: Andrew Adams 6 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

7. Related Work Intra-frame coding – Vector quantization, DCT coding, transform coding yield compression ratios of less than 30:1 Inter-frame coding (compression in the hundreds, thousands) – Disparity compensation – 3D geometry models – Blockwise Compression ideal: maximally use coherence between two images 7 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

8. Our Method: 2-D Warping Each consecutive view is a projection of the previous view due to constant predictable movement of camera Find this relation between the views by obtaining projection matrix for each pair of views Predict the view and encode the residual 8 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

9. Our Encoding Scheme 9 EE398A - Compression of Light Fields using 2-D Warping and Block Matching Reconstructed Previous View Previous Frame 2-D Warped Lagrangian Cost Function Cost=R1+λD1 Cost=R2+λD2 2D Warping Algorithm 2-D DCT for the Residual Residual and MV ? Input View -- Use for Reconstruction

10. Notes DCT used on 8x8 blocks to encode residual Laplacian distribution assumed for motion vectors Projection matrix was encoded by normalizing values with respect to 10, and assuming Laplacian distribution of bitrate. The min and max values are encoded separately using binary encoding. H = -0.578 0.005 -0.720 -0.003 -0.572 0.007 0.000 0.000 -0.582 EE398A - Compression of Light Fields using 2-D Warping and Block Matching 10

11. 1. Feature match by correlation 2. Projective matrix computed Lagrangian Mode Decision using two references 3. Clipped edges are interpolated using motion compensation 11 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

12. Getting a predicted projection: Step 1: Feature matching by Correlation Features detected by Harris corner detection algorithm, and matching points identified by maximum correlation 12 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

13. Computing the Homography Matrix A homography is an invertible transformation from the real projective plane to the projective plane that maps straight lines to straight lines EE398A - Compression of Light Fields using 2-D Warping and Block Matching 13

14. Results for 2-D Projection Warping EE398A - Compression of Light Fields using 2-D Warping and Block Matching 14

15. Results for 2-D Projective Warping EE398A - Compression of Light Fields using 2-D Warping and Block Matching 15

16. Results for 2D Projective Warping EE398A - Compression of Light Fields using 2-D Warping and Block Matching 16

17. Compression Ratios EE398A - Compression of Light Fields using 2-D Warping and Block Matching 17

18. Conclusion Advantages: decreased coding complexity, and increased rate/PSNR as well as compression Experimental results demonstrate improved coding efficiency with our 2D warp method when compared with MVC. 18 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

19. Future Work Possible Optimize the code to give better PSNR values and check performance by introducing extra modes like copy mode Explore other methods of using inter-view redundancy in detail like disparity compensation at sub-pel accuracy Run for larger data sets and optimize complexity of the algorithm EE398A - Compression of Light Fields using 2-D Warping and Block Matching 19

20. Summary Light fields represent a 3D scene using sequence of 2-D images Large amounts of data Can use redundancy between images using 2- D warping with motion compensated block matching Results in a sleek method for compression Performance wise.. EE398A - Compression of Light Fields using 2-D Warping and Block Matching 20

21. Acknowledgement Prof. Girod for pointing us in the right direction Mina Makar for his help Chuo-Ling Chang for DAPBT code Huizhong Chen and Derek Pang for their help Prof. Peter Kovesi for open source matlab function library Prof. Levoy’s group and Andrew Adams for access to light field images 21 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

22. Questions? EE398A - Compression of Light Fields using 2-D Warping and Block Matching 22

23. Other Projects Use Motion Compensation with Directional Transforms o Result: Gain in PSNR due to directionality is approximately 0.1dB at high Quantization; almost nil increase seen at low quantization So, We adapted the direction of out project to study a new approach of compression presented next. EE398A - Compression of Light Fields using 2-D Warping and Block Matching 23

24. Results with Motion Compensation and DAPBT for Crystal light field 24 EE398A - Compression of Light Fields using 2-D Warping and Block Matching

25. Results with Motion Compensation and DAPBT for Lego light field EE398A - Compression of Light Fields using 2-D Warping and Block Matching 25

26. This is how blocking is done and direction selection happens! IAP(DAT)+IRP(DCT) for QP=44, Crystal Light Field EE398A - Compression of Light Fields using 2-D Warping and Block Matching 26

27. For Lego light field IAP(DAT)+IRP(DCT) for QP=44, Crystal Light Field EE398A - Compression of Light Fields using 2-D Warping and Block Matching 27