Presentation is loading. Please wait.

Presentation is loading. Please wait.

Digital 3D Video: Chapter 8 Motion Estimation and Compensation

Similar presentations


Presentation on theme: "Digital 3D Video: Chapter 8 Motion Estimation and Compensation"— Presentation transcript:

1 Digital 3D Video: Chapter 8 Motion Estimation and Compensation
楊 家 輝 Jar-Ferr Yang 電腦與通信工程研究所 電機工程學系 國立成功大學 Institute of Computer and Communication Engineering Department of Electrical Engineering National Cheng Kung University, Tainan, Taiwan

2 Interframe Compression Methods
Video signal, which is a sequence of images, can be treated as a 3-D signal varying along x-axis, y-axis, and t-axis. Image Time x y Interframe compression can be further improved than intraframe compression, since we can further use temporal information (i.e., previously-coded images)

3 Motion-Compensated Coding
MC-DPCM Previously-coded Frame at t-1 f(x+△ x, y+△y, t-1) ^ Current Frame at t Extra Coding Parameter, Motion vector: (Dx, Dy) f(x, y, t) Difference: d(x, y, t) = f(x, y, t) – f(x+Dx, y+Dy, t-1) ^ f(x,y,t) Lossless Encoder PCM d(x,y,t) ^ Image Buffer f(x,y, t) f(x+Dx,y+Dy,t-1) + _ d(x,y,t) ^ c(x,y,t) DPCM Decoder Image Buffer f(x,y,t) ^ f(x+Dx,y+Dy,t-1) + Motion Estimation (Dx, Dy, t) (Dx, Dy, t)

4 MC-Transform Hybrid Coding
World Video Standards: Motion Compensated-DCT Hybrid Coding Methods 1. H.261/H.262/H.263 ITU (CCITT) for Visual Telephony 2. MPEG-1/MPEG-2/MPEG-4 ISO Moving Picture Expert Group To be discussed in the following chapters! How to transmit motion information? DCT- based Coder d(x,y,t) Image Buffer f(x,y, t) ^ f(x,y, t) Lossless Encoder c(x, y, t) DCT- based Decoder + Motion Estimation _ + + ^ + f(x+Dx,y+Dy,t-1) + ^ f(x+Dx,y+Dy,t-1) f(x,y,t) ^ Image Buffer (Dx, Dy,t) (Dx, Dy,t)

5 Video Compression Concepts
Spatial Redundancies => DCT and Quantization *Adjacent pixels with similar information Spatial Redundancy Temporal axis Time, T-1 Time, T Conceptually, the temporal redundancy contains in consecutive frames. If we can properly estimate the motion of moving target, the subtraction of the target block in the current frame and the estimated block in the previously-encoded frame become very small. On the other hand, the spatial redundancy contains in similar value in adjacent pixels. Due to energy compaction property, a few DCT coefficient can sufficient represent the spatial information. Temporal Redundancies =>Motion Compensation * Frame to frame redundancies

6 MC in Block-based Approach
If the motion parameters is transmitted on pixel bases, it might take too many information bits. Block-base motion information is more reasonable!! In current frame, the image is first divided into several target blocks! Current Frame In previously-coded frame, we need to find the best-matched block for each target block! Encoded Previously-coded Frame

7 Block-Motion Models (1)
1. Bilinear Transformation Bilinear x’1 = a1x1 + a2x2 + a3x1x2 + a4 x’2 = a5x1 + a6x2 + a7x1x3 + a8 Eight Motion Parameters: a1, a2, a3, a4, a5, a6, a7, a8

8 Block-Motion Models (2)
2. Prospective Transformation Prospective a1x1 + a2x2 + a3 a7x1 + a8xa + 1 a4x1 + a5x2 + a6 a7x1 + a8xa + 1 x’1 = x’2 = Eight Motion Parameters: a1, a2, a3, a4, a5, a6, a7, a8

9 Block-Motion Models (3)
3. Affine Transformation Rotation & Shift x’1 = a1x1 + a2x2 + d1 x’2 = a3x1 + a4x2 + d2 Six Motion Parameters: a1, a2, a3, a4, d1, d2

10 Block-Motion Models (4)
4. Pixel Coordinate Transformation Shift x’1 = x1 + d1 x’2 = x2 + d2 Two Motion Parameters: d1, d2 The simplest block motion model, which is used for block-based motion compensation mostly!

11 Concept of Motion Estimation
Current block Motion Estimation Motion Vectors Previous block (encoded) Residual block (px,py) Current frame (px,py) (cx,cy) Motion Vector (Dx, Dy) =(cx-px, cy-py) The motion estimator with the input of the current frame and the previous encoded frame outputs the motion vectors and residual frame. To search the best matched block, the motion estimation requires a search criterion and a search strategy. Previous frame (encoded) Motion Estimation needs a search criterion and a search strategy!

12 Block-Matching Algorithms
Definition of Best Match: t-1 Initial Point M2 N1 M2 N2 N1 t Current frame N2 M1 M1 Previously-encoded frame Within a pre-defined search range and with a search criterion, we need to find the best match block!

13 Block Matching Criteria (1, 2)
1. Minimum Means Square Error(MSE) MSE = 1 N1N2 (n1,n2)B [ s(n1, n2, t) - s(n1+d1, n2+d2, t-1) ]2 [d1*, d2*] = Arg Minimum MSE [d1 d2] (Search one block by one block) 2. Minimum Mean Absolute Difference (MAD) MAD = 1 N1N2 (n1,n2)B  [ s(n1, n2, t) - s(n1+d1, n2+d2, t-1) ] [d1*, d2*] = Arg Min MAD(d1,d2) [d1 d2] Mostly use: Sum of Absolute Difference (SAD) fixed block size

14 Block Matching Criteria (3, 4)
3. Maximum Matching Count (MMC) 1, if | s(n1,n2,t) - s(n1+d1,n2+d2,t-1) |≦C T(n1,n2,d1,d2) = 0, others MMC (d1,d2) = (n1,n2)  T(n1,n2,d1,d2) [d1*, d2*] = Arg Max MMC (d1, d2) [d1 d2] 4. Generalized Maximum Matching Count (MMC) T(n1,n2,d1,d2) =

15 Block-Matching Search
Search Range : -M1≦d1 ≦M1, -M2≦d2 ≦M2 -M1 M2 -M2 M1 (2M1 + 1) (2M2 + 1) Search Points: (2M1 + 1) ╳ (2M2 + 1) For MSE → (N1N2 Multiplications) × (2M1 + 1) × (2M2 + 1) For MAD → (2 × Additions) × (2M1 + 1) × (2M2 + 1) For MMC → (1 × Addition) × (2M1 + 1) × (2M2 + 1) For GMC → (1 × Addition) × (2M1 + 1) × (2M2 + 1)

16 Block Matching Search Algorithms
Block Matching Algorithms With Selected Block Matching Criteria -- MSE -- MAD -- MMC Block Match Search procedures Full-search Method (Exhaustive search) 1. Full Search (FS) ---(2M1+1)*(2M2+1) Example: M1=7﹐M2=7 (2M1+1) x (2M2+1) = 225 (Search Points)

17 Block Matching Search Algorithms
1. Full Search Method (FSM) 2. 1-D Row Column Search Method 3. Three Step Search (TSS) Method 4. Four Step Search (4SS) Method 5. Cross Search (CS) Method 6. Diamond Search Method Novel TSS Method Hexagonal Search Method Other improved Algorithms

18 2. Row-Column Search Algorithm
15 -7 +7 15

19 3. Three Step Search (TSS) Method
Step 1:”O”﹐9 search points  find min point (1st MSB) (x, y) = (a 0 0, b 0 0) a and b could be +1, 0, or -1 Step 2:””﹐8 search points  find min point (2nd MSB) (x, y) = (-1 c 0, +1 d 0) c and d could be +1, 0, or -1 Step 3: ”□” 8 points  find min point TOTAL = ( ) = 25 The TSS method is called as the logarithmic search method!!

20 Third-step Search Points
3. Three Step Search (TSS) Method First-step Search Points Second-step Third-step Search Points

21 Search Points of Three Step Search
-7 -6 -5 -4 -3 -2 -1 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 1 2 3 4 5 6 7 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25

22 4. Four Step Search (4SS) Step 1. Search 9 points (Start from 5x5 : same as the TSS) 1. Corner: 2. Center: 3. Side: Go to Step 2

23 4. Four Step Search (4SS) Step 2/3 If Corner: (+5) If Side: (+3)
If Center: (8-point step) Go to Step 3 if Not Center Step 4 if Center

24 4. Four Step Search (4SS) Step 4. Search window reduced to 3x3 (same as Center case)  1. Minimun search points = = 17 Step 1(Center) + Step 4 (Center)  2. Maximun search points = = 27 Step 1(Center) + Step 2 (Corner) + Step 3 (Corner) + Step 4 (Center)

25 Example for 4SS Step.1 =>Step.2 Side => Step.3 Corner
-7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 Step.1 =>Step.2 Side => Step.3 Corner => Step.4 Center Total Search =25 Points

26 Worst Cases for 4SS Step.1 =>Step.2 Corner => Step.3
-7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 Step.1 =>Step.2 Corner => Step.3 => Step.4 Center Total Search =27 Points

27 Search Points of Four Step Search
-7 -6 -5 -4 -3 -2 -1 27 26 27 27 27 27 27 25 23 25 27 27 25 22 20 22 25 26 23 20 17 20 23 26 1 2 3 4 5 6 7 25 22 20 22 25 27 27 27 25 23 25 27 27 17 27 27 26 27

28 5. Cross Search Method The search pattern is cross shape:
Step 1:”+”search (5 points) Step 2: (a) center (+8 points) stop (b) side (+3 points) Step N: center (+8 points) stop Search Shapes (Change 9 points to 5 Points)

29 5. Cross Search Method  1. Minimun search points = 5 + 8 = 13
2  1. Minimun search points = = 13 Step 1(Center)  2. Maximun search points = ???

30 N=4 for search range -7 to +7
5. Cross Search Example -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 Step.1 => Step.2 Side => Step.3 => Step.4 Center Total Search =19 Points Normal Worse Case N=4 for search range -7 to +7

31 Search Points of Cross Search
-7 -6 -5 -4 -3 -2 -1 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 16 19 16 19 19 16 16 16 16 16 16 19 16 13 16 19 19 19 19 13 13 13 19 19 19 1 2 3 4 5 6 7 19 16 13 16 19 19 16 16 16 16 16 16 16 19 19 16 19 16 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19 19

32 6. Diamond Search Step 1: 9 points ==> Three Cases 1. Center
2. Side 3. Corner

33 6. Diamond Search Step. 2/3 (1) Center (2) Side : (3) Corner :

34 6. Diamond Search Final step: with shirked diamond (same as Center case)  1. Minimun search points = = 13 Step 1(Center)  2. Maximun search points = ? (33)

35 6. Diamond Search Example
-7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 Step.1 =>Step.2 Side => Step.3 => Step.4 => Step.5 Total Search =29 Points

36 Search Points of Diamond Search
-7 -6 -5 -4 -3 -2 -1 25 24 23 24 25 25 25 22 24 21 23 21 24 22 25 25 22 22 19 21 21 19 22 22 24 19 19 18 19 19 24 21 21 16 16 21 21 23 23 18 13 18 18 23 23 1 2 3 4 5 6 7 21 21 16 16 21 21 24 19 19 18 19 19 24 22 22 19 21 21 19 22 22 25 25 25 25 22 24 21 23 21 24 22 25 24 23 24 25

37 7. Novel TSS Algorithm Step 1. TSS points (9 points)
+ around center points (8 points) 1. Center_center 2. Center_side 3. Center_corner 4. Outside Center_center Center_side Center_corner Outside point

38 7. Novel TSS Algorithm Step. 2
(1) Center ==> (0,0) Stop! (17 points) minimum search (2) Center-side  Perform 3 points search (3) Center-corner  Perform 5 points search Stop! (4) Outside point Perform regular TSS algorithm Center_corner Center-side 25+8=33(worst case)

39 Search Points for Novel 3-Step Search
-7 -6 -5 -4 -3 -2 -1 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 22 22 20 22 22 33 33 33 33 22 22 20 22 22 33 33 33 33 33 20 20 17 20 20 33 33 33 1 2 3 4 5 6 7 33 33 22 22 20 22 16 33 33 33 33 22 22 20 16 22 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33

40 Search Points Comparison
Methods min max average speed-up FS TSS 4SS NTSS Diamond Method Ave criterion Ave distance (FS) Optimality FS % TSS % 4SS % NTSS % Diamond %

41 Hexagon-Based Search Algorithm
Step 1: The large hexagon with seven checking points is centered at (0,0), the center of a predefined search window in the motion field. If the MBD point is found to be at the center of the hexagon, go to Step3 (Ending); otherwise, go to Step 2 (Searching). Step 2: With the MBD point in the previous search step as the center, a new large hexagon is formed. Three new candidate points are checked, and the MBD point is again identified. If the MBD point is at the center point of the new hexagon, otherwise, repeat this step continuously. Step 3: Switch the search pattern from the large to the small size of the hexagon. The four points covered by the small hexagon are evaluated to compare with the current MBD point. The new MBD point is the final solution of the motion vector.

42 Hexagon Search Path First step Second step Third step Final step

43 Number of Search Points
Diamond Search: Minimum search points =9+4=13 Number of search points NDS= 9+Mxn’+4, M=5 or n’: number of steps Hexagon Search : Minimum search points =7+4=11 Number of search points NHEXBS=7+3xn+4 n: number of steps

44 One More Step Hexagon Search
Minimum Distortion

45 Search Points of HEXBS Method
-7 -6 -5 -4 -3 -2 -1 20 17 17 17 20 20 20 17 17 17 17 17 17 17 20 20 20 17 17 17 14 17 17 17 20 17 17 14 14 14 14 14 17 17 17 17 14 14 11 14 14 17 17 17 17 14 14 11 11 11 14 14 17 17 1 2 3 4 5 6 7 17 17 14 14 11 14 14 17 17 17 17 14 14 14 14 14 17 17 20 17 17 17 14 17 17 17 20 20 20 20 20 17 17 17 17 17 17 17 20 17 17 17 20

46 Comparisons of HEXBS with DS
Block Size:16x16, Search Window:+/-7

47 Enhanced Hexagon Search
Enhanced Hexagon Search: Reduced inner search points by looking for the second and the third smallest points! Compared with the conventional inner search in the original/ predictive HEXBS algorithm The enhanced HEXBS will save two points for four sides and one point for another two sides. by uniformly averaging, 2x(4/6) + 1x(2/6) ~ 1.67 search points will be saved if we assume each side wins with equal probability.

48 New Search Mechanism: Prediction
Motion Vector Prediction: a. Due to the spatial correlation, the motion vector of the current block is close to those in nearby blocks. Previous coded blocks Predictor Example 1: vi -1,j-2 vi -1,j-1 vi -1,j vi -1,j +1 vi,j-2 vi,j-1 Predictor Example 2: vi,j of current block Usage of Predictor: Initial search point DPCM coding of motion vector

49 Temporal Search Prediction
Motion Vector Prediction: b. Due to the temporal correlation, the motion vector of the current block is close to those in nearby blocks in the previous frame. Current Frame Previous Frame (all are coded) Uncoded blocks of current block Predictor Example:

50 Early Stop Mechanism Early Stop Mechanism: Application:
During computation of motion estimation criterion (SAD), we don’t need to compute all Sub-SAD if the current accumulated SAD has larger than the minimum criterion Once a certain SAD is reached, the quantization will removed all residual errors, the further search could be avoided. Application: Spiral full search starting from prediction Prediction -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 Starting from most possible points, you can find minimum SAD. So, the early stop can be applied.

51 Computation Reduction Method
DCT and Quantization: DCT: Quantization: Divided by 2Q for DCT coefficients Quantized Results: Zero DCT coefficients after quantization if where the dead zone parameter is DZ = Q/2 Now, let us to look the details of DCT and quantization processing. After the DCT, we get the DCT coefficient F(u,v). However, if the DCT value is less than the quantization step, 2Q.

52 Computation Reduction for DCT
“Head-and-Shoulder” Sequences : Often used in video phone and video conference Static background and small motion of body High data correlation and low prediction residues Low rate and large quantization parameter Many all-zero blocks and zero motion macroblocks Computation Reduction for DCT and Quantization All-zero Block Condition All-zero Macroblock Condition Speed Up SAD Computation Computation Reduction Algorithm Just reverse of IDCT processor, if we can predict that the block after the DCT and quantization will be zero. We call this block as the all-zero block. Professor Ming-Ting Sun found that once we calculate the SAD in the motion estimation in statistical ways. We then can predict the all zero block. For some small SAD, actually, we don’t need to perform the DCT, IDCT, quantization, and inverse quantization.

53 All-Zero Block Condition
All-zero DCT Block Condition: After quantization, if the magnitude of DCT coefficient F(u, v) is less than the quantization step, it will be quantized to zeros: with dead zone DZ = Q/2 for For DC term, , the zero condition is Zero DC Condition: All-Zero DCT Block Condition: If we can prove that all F(u,v) are less than 2.5Q, then, the block is an all-zero DCT block. For computation of the DC term of DCT, this equation shows that the DC value is less than one eighth of SAD. That means that once the SAD is less then 20Q The whole block will be detected as the all zero block, which will not need further computation of DCT, IDCT, and quantization, and inverse quantization.

54 Correct Rate and Detected Rate
All-zero DCT Block Condition: TH=18Q TH=20Q TH=22Q B = 9600 99.84 % (68.12 %) 99.79 % (72.44 %) 99.73 % (76.09 %) B= 12800 99.77 % (60.68 %) (65.93 %) 99.64 % (70.28 %) B= 19200 (50.56 %) 99.59 % (56.91 %) 99.41 % (62.77 %) B= 28800 99.67 % (42.46 %) 99.55 % (49.11 %) 99.35 % (54.91 %) B= 33600 99.63 % (41.19 %) 99.50 % (47.84 %) 99.34 % (53.62 %) Correct Rate (Detected Rate) B: bit/sec Akiyo QCIF By experimental simulations, the correct rate and detected rate by threshold of 18Q, 20Q, and 22Q are shown in this table. For lower bit rate, we will use higher quantization, the detection rate will a higher. However, for larger bit rate, the lower quantization will make the detected rate smaller. If the sum of absolute difference (SAD) is less than the quantization-dependent threshold, 20Q, we can correctly find the all-zero DCT blocks.

55 All-Zero Macroblock Condition for ME
16 SAD (MB) 8 SADB1 SADB4 SADB3 SADB2 All-Zero Macroblock Condition: During the motion estimation, we can compute four SAD for each marcoblock. Once the SADs of four blocks are less than 20Q, we could treat the whole marcoblock as the all-zero marcoblock. The all-zero marcoblock detection with four comparison operation does not increase any other computation. if ( SADB1 < 20Q && SADB2 < 20Q && SADB3 < 20Q && SADB4 < 20Q ) { All-zero Macroblock }

56 Computation Reduction ME Algorithm
Step 1:Compute SAD of initial search point (x0,y0) If it satisfies all-zero MB condition, then stop search and set MV = (x0, y0) Else go to Step 2 Step 2: Continue search using any search algorithm A and check each search point: stop search and set MV = (x, y) search until end of Algorithm A。 To link with any motion estimation algorithm, we propose the computation reduction algorithm for the motion estimation. Basically, the proposed algorithm is completely the same as any other ME search algorithm. Once the motion search reaches the all-zero marcoblock condition, we will stop the search. With a good initial search points, we can dramatically reduce the search computation.

57 Computation Reduction (9600 bps)
PPMB Irene Miss_am Akiyo Claire Carphone FS 787.88 787.88 787.88 787.88 787.88 FSZ 282.72 196.82 355.82 255.81 448.35 TMN8 17.07 18.25 13.91 18.78 21.11 TMN8Z 7.89 5.10 6.05 5.05 13.36 TSS 28.91 28.91 28.89 28.90 29.16 TSSZ 11.25 9.33 13.24 10.15 18.77 DS1 7.01 6.10 5.01 5.29 9.27 DS1Z 4.37 3.00 2.80 2.63 7.29 DS2 13.98 13.18 12.14 12.50 16.42 This table shows that all search algorithms with all zero-marcoblock detection reduces their search points about one half for all sequences. DS2Z 6.72 4.79 5.97 4.83 10.88 BBGDS 14.93 13.92 12.55 12.92 17.86 BBGDSZ 7.41 5.26 6.03 4.99 12.31 The number of search Points Per MarcoBlocks (PPMB)

58 Performance Degradation (9600 bps)
PSNR(dB) Irene Miss_am Akiyo Claire Car phone FS 29.13 35.50 33.33 34.24 27.88 FSZ 29.05 35.06 33.39 34.23 27.73 TMN8 29.18 35.56 33.32 34.22 27.78 TMN8Z 29.06 35.25 33.27 27.68 TSS 29.07 35.09 33.31 34.16 27.75 TSSZ 34.81 34.15 27.65 DS1 29.15 35.44 27.77 DS1Z 29.09 34.76 27.67 DS2 29.17 35.60 34.28 27.80 DS2Z 34.72 33.28 34.17 27.66 BBGDS 29.12 27.81 BBGDSZ 29.10 34.83 34.18 27.70 As to the performance degradation, this table demonstrates that the proposed algorithm works well for all search algorithms, specially for the low rate video.

59 Reduction for DS1 Algorithm
Sequences Irene Akiyo Claire Miss_am Performance PPMB PSNR DS1 7.01 31.95 5.01 38.07 5.29 38.96 6.10 39.51 DS1Z 5.22 31.92 3.97 3.51 4.43 39.49 Fixed Threshold Method F600 5.60 31.93 2.30 2.39 38.94 3.38 39.44 F800 4.85 1.97 38.04 2.11 38.93 3.00 39.39 F1000 4.49 1.75 1.98 38.89 2.68 39.26 F1200 4.32 31.91 1.59 1.87 2.44 39.18 Referenced R200 3.87 1.55 1.95 3.05 39.37 R400 3.61 31.89 1.43 38.09 1.76 2.72 39.29 R600 3.41 31.86 1.36 38.05 1.65 38.83 2.29 39.03 R800 3.29 31.84 1.32 38.01 1.56 38.80 2.09 38.90 Mixed R200,F800 4.07 1.54 38.06 1.88 2.81 39.36 R200,F1000 3.80 1.52 1.85 38.92 2.67 39.31 R400,F800 3.58 1.40 1.73 2.54 39.22 R400,F1000 31.88 1.42 1.71 38.87 39.17 Original All-Zero Block By using the mixed threshold method, we can reduce diamond search method to about 1.4 – 4.07 search points without degradation of PSNR performances. DS1= Diamond Search with Step Size 1, B= bps

60 Computation Reduction
All-zero macroblock detection can reduce the search point of any ME algorithm, it also can save all the computation of DCT, Q, Q-1, and IDCT. Rate Control Ordered Source Pictures Residual Motion Estimator DCT Lossless Coder Q Buffer _ + Q-1 Prediction Coded Video Bit Steam MC Mode Decision + + Inverse DCT The all-zero marcoblock detection method can help to reduce the computation of motion estimation. It also can reduce all the computation required by the DCT, IDCT and quantization and inverse quantization. MV MC Modes Picture Predictor & Store Decoded Picture


Download ppt "Digital 3D Video: Chapter 8 Motion Estimation and Compensation"

Similar presentations


Ads by Google