Object Based Video Coding - A Multimedia Communication Perspective Muhammad Hassan Khan

Overview Motivation for Video Coding Today’s Video Coding Problems with today’s video coding Desirable Features Solution to get desirable features  Object Based Video Coding  MPEG-4 Support  Model Based Coding Major Problem: Segmentation Segmentation by Graph Cuts Architecture to Incorporate this segmentation mechanism with MPEG-4 bit stream

Why Video Coding? Consider a 1 minute video with 60fps No of frames = 60 x 60 = 3600 Given that each color frame in the video was a 640 x 480 pixels The size of the raw video comes out to be?  3600 x 640 x 480 x 3 = 3,317,760,000 bits  Of the order of Gbs Now a days one might say that memory is no big deal… BUT What if we want to transfer this file from one node to the other node over a network!  Things would collapse very soon  Just imagine if the video was 1 hour duration rather than 1 minute!!!  I hope the need for video coding is now obvious

Today’s Video Coding YUV (lossy) MotionDCT Quantize (lossy) EntropyOrder Designed for natural scenes Higher frequency DCT coefficients are quantized Sharp edges are not well preserved

Problems with Today’s Video Coding Poor performance in case of  Anything with sharp edges  Highly textured regions  Texts (Channel Logos) The bit stream produced by today’s coders is also debatable in that weather it is the MOST optimal bit stream  In fact what is a most optimal bit stream is still a question

Desired Features Better compression Improved quality Interactivity and Manipulation of Content Error Resilience Processing of content in the compressed domain Identification and selective coding/decoding of the object of interest Facilitate Search / Indexing (MPEG-7)

Solution to Get Desirable Features MPEG-4  Support for Object Based Coding Rather than conventional block based coding for natural images The scene should be divided hierarchically into objects The scene will now be described by the objects placed in a hierarchical manner A sample is presented in the next slide

Hierarchical Description The scene divided into objects

The decoding process

Meshed Video 2D mesh tessellates the video into patches Motion vector for each vertex Motivation  Modeling (Motion and Shape)

Problems with Mesh Based Coding Works fine with previously known models and caters for a small class of objects The reliable tracking of features or control points along the video  E.g. FAPs A ready-made model is assumed, 2D or 3D model of the object has to be known A more general approach was required  Object Based Video Coding Shape, Color, and Motion

Requires a Major Step! Segmentation  Dividing the scene into objects  In simplest form these objects can be foreground and background  In more complex situations there can be multiple objects in the scene  Segmentation is required to extract the objects Computing Motion  Object Based Motion  Parameterized Motion Information

Segmentation by Graph Cuts Uses Max-Flow Min-Cut Algorithm from Graph Theory Divides the data into regions based on an energy function, usually employed to the intensities of the image A smoothness function is also used to make sure that the segmentation achieved is consistent Details will be provided in the final presentation

Architecture We will also propose mechanism to assign motion information to the segmented objects Our approach will be as consistent as possible with the support provided by MPEG

References Gary J. Sullivan, Pankaj Topiwala, Ajay Luthra SPIE Conference on Applications of Digital Image Processing XXVII, Special Session on Advances in the New Emerging Standard: H.264/AVC, August, 2004 Gabriel Antunes, Abrantes, Fernando Pereira, MPEG-4 Facial Animation Technology : Survey, Implementation and Results, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 9, No. 2, March 1999 Roger H Clarke, Image and Video Compression: A Survey Department of Computing and Electrical Engineering, Heriot-Watt University, Riccarton, Edinburgh EH14 4 AS, Scotland. Noel Brady, MPEG-4 Standardized Methods for the Compression of Arbitrarily Shaped Video Objects, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 9, No. 8, December 1999 Boykov, Y.; Veksler, O.; Zabih, R.; Fast approximate energy minimization via graph cuts, Pattern Analysis and Machine Intelligence, IEEE Transactions on Volume 23, Issue 11, Nov Page(s):