1שידור ווידיאו ואודיו ברשת האינטרנט Dr. Ofer Hadar Communication Systems Engineering Department Ben-Gurion University of the Negev URL: Based on: 1 Lecture of Osnat Bar Nir from Scopus 2. Motion estimation lecture: www-ict.its.tudelft.nl/html/education/courses/et10-38/hc/hc14 3. Image communication course, lec.10, by Girod 4. Block matching algorithms, from: O. Hadar Motion Estimation for compression January, 4, 2011 דחיסת מולטימדיה Motion Estimation for compression January, 4, 2011 דחיסת מולטימדיה

2 שידור ווידיאו ואודיו ברשת האינטרנט Block Matching Approaches

3 שידור ווידיאו ואודיו ברשת האינטרנט What Is Motion Estimation ? Example: Frame 1Frame 2

4 שידור ווידיאו ואודיו ברשת האינטרנט Macro-Block to be coded GOOD MATCH FAIR MATCH POOR MATCH Block Matching

5 שידור ווידיאו ואודיו ברשת האינטרנט Residual Error Picture Desired Picture Predicted Picture Residual Error Picture The residual Error Picture contains the differences between the desired picture and the predicted one. This is the picture to be coded and transmitted. Reference Picture

6 שידור ווידיאו ואודיו ברשת האינטרנט Why Motion Estimation ? For each block in current frame find a matching block from consecutive frame, so that only the (relatively small) Residual Error Picture and its corresponding Motion Vectors are encoded. The basic principle of motion estimation is that in most cases, consecutive video frames will be similar except for changes induced by objects moving within the frames.

7 שידור ווידיאו ואודיו ברשת האינטרנט How Motion is estimated ? Motion estimation applies a comprehensive two- dimensional spatial search in order to find the best match for each block. The problem for motion estimation to solve is how to effectively represent the changes between two video frames. MPEG does not define how this search should be performed.

8 שידור ווידיאו ואודיו ברשת האינטרנט Block Matching - 1 n Block Matching u Assumes the 2D translation motion model. u Subdivide the frame (image information) into square (NXN) blocks. u Image sequence model. S t-1 t time Observe notation!

9 שידור ווידיאו ואודיו ברשת האינטרנט Criterions for Block Matching - 1 n For each NXN block S, a single motion vector (d h,d v ) is found by minimizing an error measure- finding the best “match” which minimizes the error measure. n Mean Square Error (MSE) match criterion- n Mean Absolute Difference (MAD): n Full search block matcher calculates M(d) for all d within an MXM search area.

10 שידור ווידיאו ואודיו ברשת האינטרנט Criterions for Block Matching - 2 n Pixel Difference Classification (PDC): u Compares each pixel of the target block with its counterpart in the candidate block and classifies each pixel pair as either matching or not. “Match” if the difference is less than some threshold (th). The grater the number of matching pixels, the better the match is.

11 שידור ווידיאו ואודיו ברשת האינטרנט Criterions for Block Matching - 3 n Integral projection u Sums the values of pixels from each column and each row of the block.

12 שידור ווידיאו ואודיו ברשת האינטרנט Full Search All blocks in the search window (part of reference frame) are compared against current block (in currently coded frame), in order to select the most representative one. The pointer to the selected block is called Motion Vector. Usually requires long processing time, therefore not applicable in real-time applications currently coded framereference frame

13 שידור ווידיאו ואודיו ברשת האינטרנט Full search block matching - 1 n All possible displacements within the search range are compared => n Computationally complex. n Noise sensitive. n Estimated vector field is not very smooth. Full search/brute force/exhaustive block matching N N M M d v,max d h,max x(m,t) dvdv dhdh

14 שידור ווידיאו ואודיו ברשת האינטרנט Sub-Optimal Motion Estimation Full search is not applicable DSP computation power is limited … Motion estimation is very expensive computationally

15 שידור ווידיאו ואודיו ברשת האינטרנט Sub Optimal Motion Estimation Sub Optimal Motion Estimation For a search window of ±32 pixels, full search will take more than 4000 points per MB. DSP enables a budget of about 250 points per MB. A way must be found to estimate motion in much more efficient way: A S ub-optimal Motion Estimation is needed

16 שידור ווידיאו ואודיו ברשת האינטרנט Sub-Optimal Motion Estimation Main approaches to reduce computation of motion estimation: Pel sub-sampling: A sub-sample of the MB is compared with sub-sampled blocks in search window. Adaptive search area Sub-sampling of the motion displacement space Hierarchical & multi-resolution motion estimation Predictive Motion Estimation

17 שידור ווידיאו ואודיו ברשת האינטרנט Sub-sampling of the motion displacement space Compare a sub-set of blocks against current MB Algorithms based on a coarse-to-fine approach: Step size of search is reduced by ½ with every search step. Many algorithms, using various searching orders.

18 שידור ווידיאו ואודיו ברשת האינטרנט Sub-sampling of the motion displacement space example: 3-Step Search:

19 שידור ווידיאו ואודיו ברשת האינטרנט Reduced complexity block matching - 1 n To reduce the complexity of full search block matching, various restricted search patterns have been proposed. n “One at a time search”: u Horizontal stage: the point of minimum distortion on the horizontal axis is found. u Vertical stage: Starting with this point, the minimum distortion in the vertical direction is found. dvdv dhdh

20 שידור ווידיאו ואודיו ברשת האינטרנט Reduced complexity block matching - 2 n N-step search or cross-search u A number of positions around a center are tested. The position of minimum distortion becomes the center of the next stage. u Principle of locality: First, examine a number of sparsely spaced candidate blocks, from the search area, and choose the best match from these to be the center of a finer (denser) search. u Special case: Quadrant Monotonic: candidate blocks close to the optimal block have better matches and the value of distortion is a function of the distance from the optimal position. dvdv dhdh

21 שידור ווידיאו ואודיו ברשת האינטרנט Reduced complexity block matching - 3 n 2D logarithmic search u Iterative comparison of error measure values at 5 neighboring points u Logarithmic refinement of the search pattern if: F The best match is in the center of the 5-point pattern. F The center of the search pattern touches the border of the search range. u Computational complexity: Example: max. horizontal, vertical displacement = 6 integer-pix accuracy. (a- for special vector, b- worst case).

22 שידור ווידיאו ואודיו ברשת האינטרנט D Logarithmic Search We will discuss two approaches here: n This approach was published in a paper by Jain and Jain. u Iterative comparison of the error measure (MAE) at five neighboring points u Logarithmic refinement (divide by 2) of the search pattern if F Best match is in the center of the 5-point pattern (right figure) F OR, center of search pattern touches the border of the search range (left figure) Search edge

23 שידור ווידיאו ואודיו ברשת האינטרנט Reduced complexity block matching - 5 n Orthogonal search algorithm (OSA): u Hybrid of the 2D logarithmic search and the 3 step search: u First stage- horizontal: F Compute distortion in 3 points (0,0), (s,0), (-s,0), where s is the step size. F Position of minimum distortion becomes the center of the vertical stage. u Second stage- vertical: F Two point- above and below the new center are examined, and the value of the distortion function at these 3 positions are compared. F The position with the minimum distortion becomes the center of the next stage. u After one horizontal and vertical iteration, the step size s is halved, if it is greater than 1, and the algorithm proceeds from step one.

24 שידור ווידיאו ואודיו ברשת האינטרנט Reduced complexity block matching - 6 n Orthogonal search algorithm (OSA), cont.

25 שידור ווידיאו ואודיו ברשת האינטרנט Pel Subsampling Pel Subsampling 16x16 Pixel Array4:1 Subsampling2:1 Subsampling Reduces the number of pels taken into account for the distance calculation at every search position.

26 שידור ווידיאו ואודיו ברשת האינטרנט Adaptive search area Performs a Spiral Search for the matching block. Search window size is data dependent, and set in run-time Example Spiral Search:

27 שידור ווידיאו ואודיו ברשת האינטרנט Predictive Motion Estimation Motion vectors can be predicted a priori, in order to gain an initial guess of the next motion vector. Often there is a high correlation between motion of neighboring blocks.

28 שידור ווידיאו ואודיו ברשת האינטרנט Hierarchical & multi-resolution motion estimation Find motion vector in a coarse resolution, and refine the predicted motion vector to obtain the motion vector in a finer resolution. Usually uses 3 resolution levels. Requires extra computation for multi-resolution representations of the pictures.

29 שידור ווידיאו ואודיו ברשת האינטרנט Hierarchal Block Matching - 1 n Start with estimation on low resolution frames. n Propagate (intermediate) vectors to higher resolution levels. n Allow only refinement of propagated motion vectors u Limited complexity even for large search areas. u Implicit smoothing of motion field. Resolution level 1 Resolution level 2 Resolution level 3

30 שידור ווידיאו ואודיו ברשת האינטרנט Hierarchal Block Matching Block matcher (e.g. N-step) on small blocks 2.b. refine motion field 3.b. refine motion field 2.a. propagate motion field Frame t-1Frame t

31 שידור ווידיאו ואודיו ברשת האינטרנט Hierarchal Block Matching - 3 propagate Search area Search area (only refinement allowed) Found motion vector Resolution level 2 Resolution level 1 Smoothness is obtained because the same propagated vector is used for multiple blocks at higher levels.

32 שידור ווידיאו ואודיו ברשת האינטרנט Block matching for other models n For other than traditional models, block matching is still possible, but becomes very computationally expensive. n Example: Rotation Ф=0Ф=0.15Ф=0.30Ф=0.70 For a very large number of rotations, the match criterion has to be calculated.

33 שידור ווידיאו ואודיו ברשת האינטרנט Summary After reviewing the above sub-optimal motion estimation method, we found out that the best results were achieved while using an algorithm which combines sub-sampling of the motion displacement space with adaptive search area and predictive motion vectors. Results were close enough to the results obtained by full search.