Motion Estimation Multimedia Systems and Standards S2 IF Telkom University

Overview video compression schemes Inter-frame vs. Intra-frame Intra-frame (e.g. JPEG, DCT, Wavelet)  more space  Independent  easy for editing Inter-frame (long group of picture/ motion estimation, e.g. MPEG (1, 2, 4), H.261, and H.263 )  there is relation between the frames,  better for compression 2

Motion estimation Exploiting similarities between the regions boundaries in successive video frames Predictive differential coding: to code the changes incurred on the boundaries of detected regions from one to another. Trade-off: coding efficiency vs algorithmic complexity. minimal complexity: image segmentation could only be utilised for each video frame with no consideration given to temporal redundancies of shape and texture information. 3

Motion estimation Motion Estimation Motion Vector Motion Compensation Motion Model Motion Estimation Encoding and Decoding Matching Algorithm Searching Algorithm Evaluation Matrices 4

Motion Vector [1] 5 Definition : (H.264/MPEG-4 AVC) : A two-dimensional vector used for inter prediction that provides an offset from the coordinates in the decoded picture to the coordinates in a reference picture.

Motion vector [2] Motion representation 6

Motion Compensation Applying the motion vectors to a frame to synthesize the transformation to the next frame Many frames on the movie only have little different between frames (camera or object moving) 7

8

Motion Model Motion in 3-D space corresponds to displacements in the image plane Motion compensation is conducted to provide a prediction signal for efficient video compression Efficient motion compensation often uses side information to transmit the displacements Displacement must be efficiently represented for video compression 9

Motion Estimation [1] The process of determining motion vectors that describe the transformation from one 2D image to another, usually from adjacent frames in a video sequence Block-matching with previous frame Space for saving bit motion vector is smaller than full block Efficient motion estimation reduces the energy in the motion- compensated residual frame and can dramatically improve compression performance. Motion estimation can be very computationally intensive and so this compression performance may be at the expense of high computational complexity. 10

Motion Estimation [2] „„Key performance issues  Coding performance  Complexity  Storage and/or delay The motion estimation creates a model by modifying one or more reference frames to match the current frame as closely as possible. The current frame is motion compensated by subtracting the model from the frame to produce a motion-compensated residual frame. This is coded and transmitted, along with the information required for the decoder to recreate the model (typically a set of motion vectors). 11

Motion Estimation [3] At the same time, the encoded residual is decoded and added to the model to reconstruct a decoded copy of the current frame( which may not be identical to the original frame because of coding losses). This reconstructed frame is stored to be used as reference frame for further predictions. 12

Motion Estimation [4] 2-D spatial search : 13

Motion Estimation [5] Residual Error 14

Motion Estimation [6] Choice of reference frames 1.Forward Prediction o Using an older encoded frame 2.Backwards Prediction o Using a future frame o Encoder to buffer coded frame o Out of temporal order 3.Bidirectional Prediction Note : by using multiple reference frames will make encode and decode complexity and storage requirement increase 15

Encoding and Decoding [1] 16

Encoding and decoding [2]  Encoding 1. Find a block in previous frame(reference frame). 2. Choose a region that have smallest error 3. Subtraction the matching region with current block to get difference block. 4. Encode and transmit difference block. 5. Encode and transmit motion vector that indicated position of matching region.  Decoding 1. Decoding the difference between block and motion vector 2. Sum the difference block with matching region in the reference frame. A region that was pointed by motion vector. Fact: 60% computational complexity at encoder side. Fast algorithms are necessary for real time applications. 17

Matching Algorithm

Matching Algorithm [1] 1.Dividing current and previous frame into non-overlapping block MxN 2.For each block in current frame: Finding in the region that have size 2p+N at the previous frame. Finding the nearst block by using evaluation matrices 19

Matching Algorithm [2] 1.Slow Motion Object (reduce number of search point) Cross Search Algorithm (CSA) Full Search Algorithm (FSA) Spiral Search Three Step Search Two Dimensional Logarithmic Search (TDL) Binary Search (BS) Four Step Search (FSS) Orthogonal Search Algorithm (OSA) 2.Fast Motion Object Hierarchical Search Algorithm (HSA) 20

Matching Algorithm [3]  Search Strategies I : full search All possible displacements within the search range are compared Computationally expensive 21

Matching Algorithm [4] Search Strategies II: Three Step Search Algorithm Step 1 Select an initial step size (s) equal or slightly larger than half of the maximum search range Calculate the error for the block at the center of search area and 8 square neighborhood point at the distance of s from center Step 2 Move the center to the point with minimum distortion Reduce step size by a factor of two If the step size is greater than one, repeat the step1, otherwise go to step 3 Step 3 Final point with minimum distortion is the result 22

Matching Algorithm [5]  Search Strategies III : 2D logarithmic search Anil K. Jain It can be more accurate than TSS The search window is large  Algorithm Step1 Select an initial step size (s) Calculate the error for the block at the center of search area and four point at x and y axis at distance of s from center Step2 If the position of best match is at center keep the center unchanged and reduce the step size by half, otherwise the best match becomes the center Then step 1 is repeated Step3 When the step size becomes 1 all the 8 neighbor blocks around the center will be checked for finding the best match 23

Matching Algorithm [6] Search Strategies IV : Diamond Search 24

Matching Algorithm [7] 25 NoNo AlgorithmAdvantagesDisadvantages 1Three Step SearchLow Complexity in terms of selected candidate locations Good regularity in terms of motion vector generation Complexity factor increases with search area size High data bandwidth 22-D Logarithmic Search Suitable for fast motionThe accuracy of algorithm is low when the motion vector is at an angle to x-y axis 3Four Step SearchVery low comp, More accurate than TTS, memory saving, for small motion Risk of local minima if motion far away from center 4Orthogonal SearchVery fast for small motionsLocal minima may stop the search very soon 5One at Time SearchLow computationalLocal Maxima 6Cross SearchLow computationalPerformance is worse than Three-step search

Evaluation Matrices [1] 26 Mean squared error (MSE) Mean absolute error (MAE) where : f(m,n)= a pixel on coordinate (m,n) of NxN pixel block g(m+i,n+j) = a pixel on coordinate (m+i,n+j) that corresponding with previous frame.

Evaluation Matrices [2] 27 Current block reference area MSE at (0,0) {(1-4) 2 + (3-2) 2 + (2-3) 2 + (6-4) 2 + (4-2) 2 + (3-2) 2 + (5-4) 2 + (4-3) 2 + (3-3) 2 }/9 = 2.44

Evaluation Matrices [3] 28 Sum of absolute differences (SAD) Matching the template into search image Sum all of number in same matrix Example: 20, 25, and 17