2. Bernd Girod: Image Compression and Graphics 1 Image Compression and Graphics: More Than a Sum of Parts? Bernd Girod Collaborators: Peter Eisert, Marcus Magnor, Prashant Ramanathan, Eckehard Steinbach (all Stanford), Thomas Wiegand (HHI) Image, Video, and Multimedia Systems Group Information Systems Laboratory Stanford University

3. Bernd Girod: Image Compression and Graphics 2 2048 triangles Can 3-D Geometry Help to Compress Images? Conjecture: 3-d geometry models help compression, if a single 3-D model captures the dependencies between many views (or frames of a sequence). Conjecture: 3-d geometry models help compression, if a single 3-D model captures the dependencies between many views (or frames of a sequence).

4. Bernd Girod: Image Compression and Graphics 3 Outline of this Talk Compression of many simultaneous views (e.g. light-fields) –Encoding view-dependent texture maps with 4-d wavelets –Hierarchical image-domain light-field coder –Why image-domain encoding is (usually) superior to texture-map encoding Model-based compression of talking head sequences –Modeling and estimation of facial expressions –Avatars –Incorporate synthetic video into motion-compensated hybrid coding

5. Bernd Girod: Image Compression and Graphics 4 Multi-View Image Capture Coding schemes suitable for 2-plane parametrization Hemispherical image arrangement (arbitrary recording positions)

6. Bernd Girod: Image Compression and Graphics 5 Align Views by Mapping onto Object Surface Camera views: No correlation between corresponding pixels View- dependent texture map: Strong correlation between corresponding texels

7. Bernd Girod: Image Compression and Graphics 6 3-D Reconstruction from Many Views  processes all views simultaneously  exploits texture and silhouette information  yields solid 3-D voxel model Volumetric Reconstruction Subdivide object’s bounding box into voxels Generation of multiple hypotheses for each voxel Hypothesis elimination by projecting visible voxels into light-field images Iterate over all voxels until remaining hypotheses are “photo-consistent”

8. Bernd Girod: Image Compression and Graphics 7 Surface Representation Initial octahedral geometry Geometry refinement –determine vertex normals –move vertices to model surface –subdivide triangles 128 triangles512 triangles2048 triangles8192 triangles voxel model

9. Bernd Girod: Image Compression and Graphics 8 Warp of each image into a view-dependent texture map Texture map correlated in 4-d Interpolate missing texels Texture Map Encoding with 4-d Wavelets 4D Haar Wavelet Transform Embedded encoding of wavelet coefficients (4D-SPIHT) Arrange images into 2-d array

10. Bernd Girod: Image Compression and Graphics 9 Results: Wavelet Texture Map Encoder Reconstruction quality in luminance PSNR (dB)

11. Bernd Girod: Image Compression and Graphics 10 Results: Wavelet Texture Map Encoder

12. Bernd Girod: Image Compression and Graphics 11 28.6 dB 0.0076 bpp 26.3 Kbytes Progressive Decoding 36.6 dB 0.213 bpp 736 Kbytes

13. Bernd Girod: Image Compression and Graphics 12 Align Views by Model-aided Prediction ? Given: geometry model, reference images Render geometry for reference images and prediction image For each pixel: determine triangle, coordinates Find corresponding pixels in reference images Copy & average visible pixels

14. Bernd Girod: Image Compression and Graphics 13 Hierarchical Image Coding Order project camera positions on hemisphere subdivide into 4 quadrants INTRA-encode corner images encode center image - image prediction - residual error coding encode mid-side images subdivide into sub-quadrants encode center and mid-side images subdivide repeatedly

15. Bernd Girod: Image Compression and Graphics 14 Model-aided Image-Domain Light-Field Coder DCT Coefficients Light-Field Image I[u,v] Disparity Map Generation Multiframe Disparity Compensation Residual-Error DCT Coder Residual-Error Decoder Image Buffer Compressed Geometry Model 3-D Geometry Reconstruction Geometry Coder Geometry Decoder -

16. Bernd Girod: Image Compression and Graphics 15 Picture Quality original Mouse light field 257 RGB images, 384x288 pixels 81.3 Mbytes compressed 300:1 0.077 bpp (267 KBytes) 37.9 dB PSNR

17. Bernd Girod: Image Compression and Graphics 16 Model-aided vs. Texture Coding Model-aided Texture

18. Bernd Girod: Image Compression and Graphics 17 Natural vs. Synthetic Image Set 40 % 70 %

19. Bernd Girod: Image Compression and Graphics 18 2 dB 7 dB 9 dB Inaccurate Geometry

20. Bernd Girod: Image Compression and Graphics 19 Model-based videophone

21. Bernd Girod: Image Compression and Graphics 20 Modeling of Facial Expressions Head geometry composed of 101 triangular B-spline patches Facial expressions by superposition of 66 FAPs (Facial Animation Parameters) according to MPEG-4 standard FAPs act on control points of triangular B-spline patches

22. Bernd Girod: Image Compression and Graphics 21 Estimation of Facial Expressions Displacement field constrained by FAPs Linearize for small FAPs Optical flow constraint equation Solve overdetermined system by linear regression Apply iteratively in analysis-synthesis loop Incorporate spatial resolution pyramid

23. Bernd Girod: Image Compression and Graphics 22 Results: Peter Sequence: Peter, 230 frames, CIF resolution, 25 fps OriginalSynthesized Compressed 25,000:1 1.2 kbps - 32.8 dB PSNR

24. Bernd Girod: Image Compression and Graphics 23 Results: Eckehard Sequence: Eckehard CIF resolution, 25 fps OriginalSynthesized 1.1 kbps, 32.6 dB PSNR

25. Bernd Girod: Image Compression and Graphics 24 Results: Peter as Eckehard Sequence: Peter, 230 frames, CIF resolution, 25 fps OriginalSynthesized

26. Bernd Girod: Image Compression and Graphics 25 Results: Eckehard as Peter Sequence: Eckehard CIF resolution, 25 fps OriginalSynthesized

27. Bernd Girod: Image Compression and Graphics 26 Results: Peter as Akiyo Sequence: Peter, 230 frames, CIF resolution, 25 fps OriginalSynthesized

28. Bernd Girod: Image Compression and Graphics 27... But, What About Unknown Objects? Sequence: Clap OriginalSynthesized 1.2 kbps

29. Bernd Girod: Image Compression and Graphics 28 Model-Aided Coding: Incorporating Synthetic Video into MC Hybrid Coding Intraframe DCT Coder Intraframe Decoder Multiframe Motion Compensation - Decoder e Coder Control Control data (incl. motion vectors) DCT coefficients Input Video Model-based Decoder FAPs Model- based Coder

30. Bernd Girod: Image Compression and Graphics 29 R-D-Optimal Mode Decision Synthesized frame Previous decoded frame Selection Mask minimizing D+ R Predicted frame

31. Bernd Girod: Image Compression and Graphics 30 Results: Peter Sequence: Clap, 8.33 fps, CIF resolution H.263 (TMN-10) @ 12 kbpsModel-Aided Coder @ 12 kbps

32. Bernd Girod: Image Compression and Graphics 31 Results: Akiyo Sequence: Akiyo, 10 fps, CIF resolution H.263 (TMN-10) @ 10 kbpsModel-Aided Coder @ 10 kbps

33. Bernd Girod: Image Compression and Graphics 32 R-D Performance of Model-Aided Coder Sequence: PeterSequence: Akiyo ~ 40% ~ 35%

34. Bernd Girod: Image Compression and Graphics 33 Can 3-d geometry help to compress images? YES...... IF many views of the same 3-D object/scene shall be compressed. Applications in –Multiview image coding (light-field compression) –Compression of video sequences –Very high compression ratios (100:1... 25,000:1) Require accurate vision algorithms for 3-d reconstruction Image-domain compression more resilient against inaccurate geometry and hence more practical than texture-map encoding Conclusion

35. Bernd Girod: Image Compression and Graphics 34... THE END