1. 1 Yu Liu 1, Feng Wu 2 and King Ngi Ngan 1 1 Department of Electronic Engineering, The Chinese University of Hong Kong 2 Microsoft Research Asia, Beijing, 100080, China ISCAS 2007, New Orleans, USA, 27-30 May 2007 3D Object-based Scalable Wavelet Video Coding with Boundary Effect Suppression

2. 2 Introduction Introduction Motivation Motivation Object-based Motion Threading Using The Lifting Structure Object-based Motion Threading Using The Lifting Structure 3D SA-DWT Via Lifting 3D SA-DWT Via Lifting Experimental Results Experimental Results Conclusion Conclusion Outline

3. Introduction Standard video coding paradigm Standard video coding paradigm Frame-based: MPEG-1/2, H.261/263/264Frame-based: MPEG-1/2, H.261/263/264 V.S. V.S. Object-based: MPEG-4/7Object-based: MPEG-4/7 Advantages: Advantages: enables the accessibility and manipulability of object within a video sequenceenables the accessibility and manipulability of object within a video sequence allows the structure of video content to survive the process of acquisition, editing and distributionallows the structure of video content to survive the process of acquisition, editing and distribution some important features still need more efforts some important features still need more efforts scalability,scalability, coding efficiency,coding efficiency, etc.,.etc.,. 3 DCT/ ICT

4. Introduction Alternative to traditional video coding standard — wavelet-based video coding Alternative to traditional video coding standard — wavelet-based video coding full spatio-temporal-quality scalabilityfull spatio-temporal-quality scalability non-redundant 3D subband decompositionnon-redundant 3D subband decomposition comparable with H.264-based JSVM schemecomparable with H.264-based JSVM scheme MCTF-based 3D scalable wavelet video coding MCTF-based 3D scalable wavelet video coding Motion Compensated Temporal Filtering (MCTF)Motion Compensated Temporal Filtering (MCTF) motion compensation is incorporated into temporal wavelet transform motion compensation is incorporated into temporal wavelet transform Spatial domain MCTF (SD-MCTF), T+2DSpatial domain MCTF (SD-MCTF), T+2D which performs MCTF prior to spatial wavelet transform which performs MCTF prior to spatial wavelet transform In-band MCTF (IB-MCTF), 2D+TIn-band MCTF (IB-MCTF), 2D+T which performs MCTF after spatial wavelet transform which performs MCTF after spatial wavelet transform 4

5. Motivation 3D Object-based Scalable Wavelet Video Coding 3D Object-based Scalable Wavelet Video Coding Object-based motion threading using the lifting structureObject-based motion threading using the lifting structure extends the motion threading technique from the frame- based coding to the object-based coding extends the motion threading technique from the frame- based coding to the object-based coding attracted by the unique advantages of the object-based coding that do not exist in other coding schemesattracted by the unique advantages of the object-based coding that do not exist in other coding schemes 3D SA-DWT via Lifting3D SA-DWT via Lifting to suppress the boundary effects to suppress the boundary effects which exist in spatial and temporal wavelet transforms due to the manner of object-based codingwhich exist in spatial and temporal wavelet transforms due to the manner of object-based coding 5

6. Object-based Motion Threading Using The Lifting Structure Previous motion threading techniques [Xu01, Luo04] Previous motion threading techniques [Xu01, Luo04] Motion Threading (MTh) [xu01]: An efficient implementation of MCTFMotion Threading (MTh) [xu01]: An efficient implementation of MCTF employs long wavelet filters to exploit the long-term correlation across frames along motion trajectory employs long wavelet filters to exploit the long-term correlation across frames along motion trajectory some limitations, such as many-to-one mapping, terminating and non-referred pixels some limitations, such as many-to-one mapping, terminating and non-referred pixels Advanced Motion ThreadingAdvanced Motion Threading [Luo04] [Luo04] many-to-one mapping: many-to-one mapping: well solvedwell solved terminating and non-referred terminating and non-referred not well solvednot well solved 6

7. Object-based Motion Threading Using The Lifting Structure Why object-based motion threading? Why object-based motion threading? object-based coding with arbitrary regions of supportobject-based coding with arbitrary regions of support more accurate motion trajectory in the video objectmore accurate motion trajectory in the video object Object-based MTh using the lifting structure Object-based MTh using the lifting structure aims to reduce the boundary effects of artificially terminating/ emerging threads in the previous MTh.aims to reduce the boundary effects of artificially terminating/ emerging threads in the previous MTh. 7

8. Object-based Motion Threading Using The Lifting Structure Object-based motion threading Object-based motion threading For the thread which is originally terminated in many-to-one mappingFor the thread which is originally terminated in many-to-one mapping it still can continue temporal filtering without being terminated by using lifting structure it still can continue temporal filtering without being terminated by using lifting structure 8

9. Object-based Motion Threading Using The Lifting Structure Object-based motion threading Object-based motion threading For the terminating and non-referred pixelsFor the terminating and non-referred pixels originally indicate the boundaries of the motion threads originally indicate the boundaries of the motion threads using the symmetric extension over the motion thread boundary using the symmetric extension over the motion thread boundary instead of assigning the motion vector from adjacent motion threadinstead of assigning the motion vector from adjacent motion thread 9 shape adaptive temporal wavelet transform

10. 3D SA-DWT Via Lifting Boundaries Boundaries Temporal domain: the boundary of motion threadTemporal domain: the boundary of motion thread Spatial domain: the boundary of arbitrarily shaped signalsSpatial domain: the boundary of arbitrarily shaped signals Approaches for transforming 2D arbitrarily shaped signals to wavelet domain Approaches for transforming 2D arbitrarily shaped signals to wavelet domain Extension over the boundaries [Li00]Extension over the boundaries [Li00] Boundary filters [Herley95]Boundary filters [Herley95] Projections onto convex sets [You87]Projections onto convex sets [You87] Biorthogonal symmetric filters with symmetric extensions over the boundaries [Li00] Biorthogonal symmetric filters with symmetric extensions over the boundaries [Li00] focus on the texture coding of 2D still image objectsfocus on the texture coding of 2D still image objects not for the coding of 3D video objectsnot for the coding of 3D video objects 10

11. 3D SA-DWT Via Lifting Efforts in developing coding techniques for arbitrarily shaped 3D video objects Efforts in developing coding techniques for arbitrarily shaped 3D video objects Opportunity provided by object-based motion threading techniqueOpportunity provided by object-based motion threading technique align a series of video object planes to form a 3D video object align a series of video object planes to form a 3D video object 3D SA-DWT = 3D SA-DWT = (1D Shape Adaptive Temporal Wavelet Transform + 2D Shape Adaptive Spatial Wavelet Transform) 2D Shape Adaptive Spatial Wavelet Transform) SD-MCTF (T+2D)SD-MCTF (T+2D) first apply 1D-SA-DWT in the temporal thread segment first apply 1D-SA-DWT in the temporal thread segment then apply 2D-SA-DWT in the spatial signal segment then apply 2D-SA-DWT in the spatial signal segment IB-MCTF (2D+T)IB-MCTF (2D+T) or vice versa or vice versa 11

12. 3D SA-DWT Via Lifting Spatial Signal SegmentTemporal Thread Segment Biorthogonal 9/7 lifting wavelet Biorthogonal 5/3 lifting wavelet Example of aligning a series of video object planes to form a 3D video object 12 symmetric extensions over the boundaries

13. 3D SA-DWT Via Lifting 3D multi-spatio-temporal resolution video object pyramid 3D multi-spatio-temporal resolution video object pyramid non-redundant decomposition structurenon-redundant decomposition structure full spatio-temporal-quality scalability for object-based codingfull spatio-temporal-quality scalability for object-based coding 3D EBCOT (Embedded Block Coding with Optimal Truncation) [Xiong05] 3D EBCOT (Embedded Block Coding with Optimal Truncation) [Xiong05] a bit-plane entropy coding scheme which encodes the wavelet coefficients coming from 3D SA-DWTa bit-plane entropy coding scheme which encodes the wavelet coefficients coming from 3D SA-DWT an extension of 2D EBCOT in the JPEG2000 image compression standardan extension of 2D EBCOT in the JPEG2000 image compression standard Extension to object-based coding: 3D OB-EBCOT Extension to object-based coding: 3D OB-EBCOT Only samples inside the object are encoded by 3D object-based EBCOTOnly samples inside the object are encoded by 3D object-based EBCOT If neighboring samples fall outside the object, their value is set to zero in context modelingIf neighboring samples fall outside the object, their value is set to zero in context modeling 13

14. Experimental Results The test object sequences, (a) Akiyo, (b) Bream, (c) Children, (d) Coastguard, and (d) Foreman 14 (a) (b) (c) (d) (e) The proposed object-based scalable wavelet video coder (3D OB-EBCOT) includes two versions: SD-MCTF-based: 3D OB-EBCOT (SD), IB-MCTF-based : 3D OB-EBCOT (IB).

15. Experimental Results Table I lists the average PSNR (in dB) results given by different coders on the four foreground object sequences. Table I lists the average PSNR (in dB) results given by different coders on the four foreground object sequences. 15 The comparisons of 3D OB-EBCOT with MPEG-4 in terms of PSNR plots (in dB) for the foreground objects, Children and Foreman. The comparisons of 3D OB-EBCOT with MPEG-4 in terms of PSNR plots (in dB) for the foreground objects, Children and Foreman.

16. Experimental Results Coding performance comparison of 3D OB-EBCOT with different coding schemes Coding performance comparison of 3D OB-EBCOT with different coding schemes 16 The 3D OB-EBCOT coder is an object-based scalable video coder and is able to achieve all of these different target bit rates in a single bitstream.

17. Conclusion 3D Object-based Scalable Wavelet Video Coder 3D Object-based Scalable Wavelet Video Coder object-based motion threading with lifting structureobject-based motion threading with lifting structure to improve the previous motion threading techniques for object-based coding to improve the previous motion threading techniques for object-based coding 3D SA-DWT via lifting in a unified framework3D SA-DWT via lifting in a unified framework to suppress the boundary effect problem existing simultaneously in spatial and temporal transforms of object-based coding to suppress the boundary effect problem existing simultaneously in spatial and temporal transforms of object-based coding 17

18. Thank you for your attention! Q&A 18