Shen-Chuan Tai, Chien-Shiang Hong, Cheng-An Fu National Cheng Kung University, Tainan City,Taiwan (R.O.C.),DCMC Lab Pacific-Rim Symposium on Image and Video Technology(PSIVT) 2010

Introduction Background Boundary Matching Algorithm (BMA) Hybrid MV Estimation Frame Concealment Algorithm (HMVE) Proposed Object-based Full Frame Concealment Algorithm Searching and Set MV to an object from Reference Frame Object Boundary Handling Overlapped Regions by Different Objects Uncovered Regions Handling Simulation Results Outline

Introduction With the rapid development of wireless network technologies, multimedia data are transmitted through wireless networks which are always error-prone. Errors caused by packet losses not only degrade the quality of current frame, but also propagate to succeeding frames. Error concealment methods reconstruct the damaged data based on high spatial and temporal correlations of the video sequences. According to the blocks in the same object should have near MVs, using the MV to conceal whole object will make more comfortable view for human vision.

Background The error concealment algorithm implemented in the test model of H.264 assumes that in a wireless environment. Generally, the temporal domain usually keeps more information than spatial domain. Here lists some of these algorithm: Boundary Matching Algorithm (BMA) Hybrid MV Estimation Frame Concealment Algorithm (HMVE)

Boundary Matching Algorithm (BMA) [3] T. Thaipanich, P. H, Wu, and C. C. J. Kuo, "Video error concealment with outer and inner boundary matching algorithm", SPIE, vol. 6696, pp , Based on the boundary smoothness assumption. The candidate MV set usually consists of neighboring MVs and the zero MV: D n BMA : the MAD of the nth candidate block P i N & P j C : boundary pixel values of neighboring block i and candidate block j n opt : the best candidate and used to conceal the lost block

Boundary Matching Algorithm (BMA) [3] T. Thaipanich, P. H, Wu, and C. C. J. Kuo, "Video error concealment with outer and inner boundary matching algorithm", SPIE, vol. 6696, pp , PiNPiN PjCPjC

Outer Boundary Matching Algorithm (OBMA) [3] T. Thaipanich, P. H, Wu, and C. C. J. Kuo, "Video error concealment with outer and inner boundary matching algorithm", SPIE, vol. 6696, pp , 2007.

Hybrid MV Estimation Frame Concealment Algorithm (HMVE) HMVE algorithm has three steps as following: Step.1 classify the pixels of the missing frame into three parts. Part A: Pixels are covered by least one extrapolated 4×4 block. Part B: Pixels belong to blocks overlapped by extrapolated 4×4 blocks, but are not covered by any concealed blocks. Part C: Pixels belong to blocks that are not covered by any extrapolated 4×4 blocks. [11][29] Q. Peng, T. Yang, and C. Zhu, “Block-based temporal error concealment for video packet using motion vector extrapolation,” in Proc. IEEE Int. Conf. Communications, Circuits and Systems and West Sino Expositions, Jun. 2002, vol. 1, pp. 10–14.

Hybrid MV Estimation Frame Concealment Algorithm (HMVE) [11] B. Yan, and H. Gharavi, "A Hybrid Frame Concealment Algorithm for H.264/AVC," IEEE Transactions on Image Processing, vol. 19 pp , Jan For example, Part A : Pixels {1, 2, 3, 4, 5, 6, 7, 9, 13, 14, 15} in BLK1 Part B : Pixels {8, 10, 11, 12, 16} in BLK1 Part C : Pixels in BLK2

Hybrid MV Estimation Frame Concealment Algorithm (HMVE) HMVE algorithm has three steps as following: Step.2 estimate MV a and MV m for each block that is covered by at least one extrapolated 4×4 block in the missing frame. [11] B. Yan, and H. Gharavi, "A Hybrid Frame Concealment Algorithm for H.264/AVC," IEEE Transactions on Image Processing, vol. 19 pp , Jan B n i : the concealed i th block in the missing frame n EB n j : the j th extrapolated 4×4 block in the reference frame to missing frame n

Hybrid MV Estimation Frame Concealment Algorithm (HMVE) HMVE algorithm has three steps as following: Step.2 estimate MV a and MV m for each block that is covered by at least one extrapolated 4×4 block in the missing frame. [11] B. Yan, and H. Gharavi, "A Hybrid Frame Concealment Algorithm for H.264/AVC," IEEE Transactions on Image Processing, vol. 19 pp , Jan BniBni EB n j W n i,j = 6

Hybrid MV Estimation Frame Concealment Algorithm (HMVE) HMVE algorithm has three steps as following: Step.3 conceal all pixels of a block. For each pixel in Part A: For each pixel in Part B: For each pixel in Part C: After the set MVS(P n x,y ) of pixel is obtained, the MV of this pixel is estimated by averaging. [11] B. Yan, and H. Gharavi, "A Hybrid Frame Concealment Algorithm for H.264/AVC," IEEE Transactions on Image Processing, vol. 19 pp , Jan Includes the MVs of all overlapped extrapolated blocks. The MVs pare in MVS(P n x,y ) would be discard if the MV distance of each other is bigger than a threshold.

Proposed Object-based Full Frame Concealment Algorithm Flowchart :

Searching and Set MV to an Object from Reference Frame The MV of first searching block is the stand MV of current object. Search the blocks which the difference MV with standard MV is less than a threshold and set these blocks as an object. Each object is bounded into 8x8 blocks. MV which has the most number with the same value will be used to recover this object: OB k : the k th object of reference frame B n k : t he n th block in k th object

Flowchart : Proposed Object-based Full Frame Concealment Algorithm

Object Boundary Handling The block-based concealment cause blocking artifact. A block will be defined as boundary if : 1. It locates at the boundary of object. 2. There are edge pixels in this block. (detected by sobel edge detection) For boundary block, it will be considered with one neighboring block that does not belong to this object.

Flowchart : Proposed Object-based Full Frame Concealment Algorithm

Overlapped Regions by Different Objects When two or more objects overlapped at one region, the MV nearest to the results of HMVE algorithm will be better selection.

Flowchart : Proposed Object-based Full Frame Concealment Algorithm

Uncovered Regions Handling Our algorithm uses a method that based on BMA. The MVs of the pixels concealed in this block and MVs of neighboring pixels are also the candidates for uncovered region. The candidate block that yield the smallest distortion is selected to conceal the uncovered region.

Flowchart : Check pixel by pixel,if the difference is larger than a 50, this pixel will use HMVE concealment as final result. Proposed Object-based Full Frame Concealment Algorithm

Simulation results Test environment: Joint Model (JM10.1) of H.264 CIF (352×288) videos The period of I frame reset is 15 and the number of the reference frames is 1.

Simulation results

Test environment: Proposed OFFC algorithm could also conceal the I frame when entire I-frame is lost. Missing I-frame is not representing a scene cut.

Simulation results The region used by HMVE algorithm result is fewer than 2%. Red region is chosen by HMVE result, because the sobel edge detect could not detect boundary correctly.