1. Marc CHAUMONT ICIP 2003 Fully scalable object based video coder based on analysis- synthesis scheme Marc Chaumont, Nathalie Cammas 1 and Stéphane Pateux Temics, IRISA/INRIA, France, 1 France Telecom, France

2. Marc CHAUMONT ICIP 2003  What is scalability ?  ordering the coded information by decreasing significant order.  different scalability : object, SNR, temporal, spatial, bitstream.  Why looking for scalability ?  bitstream can be decoded at different bitrate and different qualities  Why using an object coder instead of an non object coder ?  object manipulation  bitrate allocation  better motion estimation (limit mesh degeneracy on occlusion boundaries)  good tradeoff between pixel based and model based approach. Context

3. Marc CHAUMONT ICIP 2003  Objectives :  scalability on the 3 informations : motions - textures - shapes  independent coding of the 3 informations  long term approach  Why those objectives :  scalability : …  independent coding : to allow a better rate distribution between motion -texture - shape.  long term approach : to allow an efficient coding with wavelet  Our proposed solution :  using wavelet (to allow scalability)  decorrelation (to allow independent coding)  long term motion estimation (to allow a long term approach) Proposed approach

4. Marc CHAUMONT ICIP 2003 Proposed approach analysis-synthesis scheme z-order shapes frames z-order shapes frames 1 - ANALYSIS 2 - CODING 3 - DECODING z-order NO SHAPE frames z-order frames shapes 1 - ANALYSIS 2 - CODING 3 - DECODING sequence SYNTHESISSYNTHESIS SEGMENTATIONSEGMENTATION obj1 obj2 reconstructed sequence

5. Marc CHAUMONT ICIP 2003 Proposed approach analysis-synthesis scheme z-order shapes frames z-order shapes frames 1 - ANALYSIS 2 - CODING 3 - DECODING z-order NO SHAPE frames z-order frames shapes 1 - ANALYSIS 2 - CODING 3 - DECODING sequence SYNTHESISSYNTHESIS SEGMENTATIONSEGMENTATION obj1 obj2 reconstructed sequence

6. Marc CHAUMONT ICIP 2003 Proposed approach analysis-synthesis scheme z-order shapes frames z-order shapes frames 1 - ANALYSIS 2 - CODING 3 - DECODING

7. Marc CHAUMONT ICIP 2003 Proposed approach analysis-synthesis scheme z-order shapes frames z-order shapes frames 1 - ANALYSIS 2 - CODING 3 - DECODING

8. Marc CHAUMONT ICIP 2003 Proposed approach analysis-synthesis scheme 1 - ANALYSIS 2 - CODING 3 - DECODING z-order shapes frames z-order shapes frames GOP

9. Marc CHAUMONT ICIP 2003 Proposed approach analysis-synthesis scheme long term motion estimation (active mesh) decorrelation thanks to motion and padding z-order shapes frames z-order shapes frames GOP SYNTHESISSYNTHESIS motions contours textures motions textures contours coding decoding coding decoding coding decoding GOP

10. Marc CHAUMONT ICIP 2003 Proposed approach analysis-synthesis scheme long term motion estimation (active mesh) decorrelation thanks to motion and padding z-order shapes frames z-order shapes frames GOP SYNTHESISSYNTHESIS motions contours textures motions textures contours coding decoding coding decoding coding decoding GOP

11. Marc CHAUMONT ICIP 2003 Long term motion estimation Motion estimation via active mesh GOP size = 8

12. Marc CHAUMONT ICIP 2003 Proposed approach analysis-synthesis scheme long term motion estimation (active mesh) decorrelation thanks to motion and padding z-order shapes frames z-order shapes frames GOP SYNTHESISSYNTHESIS motions contours textures motions textures contours coding decoding coding decoding coding decoding GOP

13. Marc CHAUMONT ICIP 2003 Decorrelation: motion projection & padding textures projected contours projected MOTION PROJECTION PADDING MOTION PROJECTION PADDING contours padded textures paddedinitial textures initial contours

14. Marc CHAUMONT ICIP 2003 Proposed approach analysis-synthesis scheme long term motion estimation (active mesh) decorrelation thanks to motion and padding z-order shapes frames z-order shapes frames GOP SYNTHESISSYNTHESIS motions contours textures motions textures contours coding decoding coding decoding coding decoding GOP

15. Marc CHAUMONT ICIP 2003 motion - texture - contour motion (mesh)textures projected and paddedcontours projected and padded 3 independent information GOP size = 8

16. Marc CHAUMONT ICIP 2003  Decorrelation allows :  independent coding of the 3 informations : motion -texture - shape.  allows independent lossy coding on each information  a better rate distribution between motion -texture - shape. distribution example for Foreman foreground CIF 15Hz at 85 Kb/s : texture : 76 % motion : 17 % shape : 7 %  long term approach  efficient coding with wavelet Example : our scheme on Erik sequence CIF 15Hz at 64Kb/s is better than H26L VM 8.4 (less than 1 dB)  scalability on the 3 information thanks to wavelet Benefits of the decorrelation

17. Marc CHAUMONT ICIP 2003 Proposed approach analysis-synthesis scheme long term motion estimation (active mesh) decorrelation thanks to motion and padding z-order shapes frames z-order shapes frames GOP SYNTHESISSYNTHESIS motions contours textures motions textures contours coding decoding coding decoding coding decoding GOP

18. Marc CHAUMONT ICIP 2003 Coding step textures contours motions Spatial decomposition (9/7 Daubechies filter) Temporal decomposition (5/3 lifting filter) Temporal decomposition (9/7 filter) Spatial decomposition (9/7 Daubechies filter) Temporal prediction IPB Spatial pyramidal decomposition Bit plan arithmetic coder EBCOT Bit plan arithmetic coder Spatio-temporal transformation

19. Marc CHAUMONT ICIP 2003 Results Background object : Foreground object : Shape : 3 Kb/s Texture : 39,7 Kb/s Motion : 7,3 Kb/s Texture + Motion : 11,5 Kb/s Reconstructed sequence at 62 Kb/s

20. Marc CHAUMONT ICIP 2003  With or without shape distortion Results Without shape distortionReconstructed sequence at 62 Kb/s (with shape distortion)

21. Marc CHAUMONT ICIP 2003  H264/AVC versus our object scalable scheme Results Our scheme 62 Kb/s PSNRtexture-foreground = 29.3 H264/AVC non object 62 Kb/s PSNRforeground = 27.9. 1 B frame,. RD optimization,. CABAC.

22. Marc CHAUMONT ICIP 2003  Spatial scalability Results Sequence 62 Kb/s CIF 15 Hz shape : 3 Kb/s background : text + mvt : 11,5 Kb/s foreground : text + mvt : 47 Kb/s Sequence 49 Kb/s QCIF 15 Hz shape : 1,7 Kb/s background : text + mvt : 9,6 Kb/s foreground : text + mvt : 37,5 Kb/s

23. Marc CHAUMONT ICIP 2003  Spatio-temporal scalability Results Sequence 49 Kb/s QCIF 15 HzSequence 36 Kb/s QCIF 7.5Hz shape : 1,2 Kb/s background : text + mvt : 6,7 Kb/s foreground : text + mvt : 27,9 Kb/s shape : 1,7 Kb/s background : text + mvt : 9,6 Kb/s foreground : text + mvt : 37,5 Kb/s

24. Marc CHAUMONT ICIP 2003  SNR scalability Results Sequence 49 Kb/s CIF 15 HzSequence 126 Kb/s CIF 15 Hz shape : 6,8 Kb/s background : text + mvt : 24,4 Kb/s foreground : text + mvt : 94,9 Kb/s shape : 1,7 Kb/s background : text + mvt : 9,6 Kb/s foreground : text + mvt : 37,5 Kb/s

25. Marc CHAUMONT ICIP 2003  Based on :  analysis-synthesis scheme  decorrelation of the 3 informations (active mesh - padding - z-order)  fully scalable  Benefits :  fully scalable (SNR, spatial, temporal & on each information)  independent coding of the 3 informations  better bitrate distribution  long term approach  allow the use of longer wavelet kernels  Future work :  improving texture coding  improving motion estimation in occlusion part Conclusion : A novel object based video coder