1 Efficient Reference Frame Selector for H.264 Tien-Ying Kuo, Hsin-Ju Lu IEEE CSVT 2008

2 Outline Introduction Introduction H.264 Inter-Coding H.264 Inter-Coding Proposed Method Proposed Method Experimental Results Experimental Results Conclusions Conclusions

3 Introduction Background Knowledge (1/4) H.264 features for inter-frame coding H.264 features for inter-frame coding Variable block size motion compensation Variable block size motion compensation Subpixel motion estimation Subpixel motion estimation Multiple reference frame motion compensation Multiple reference frame motion compensation N - 5 N - 2N - 1 Current frame N

4 Introduction Background Knowledge (2/4) Why do Multiple reference frame help predictions ? Why do Multiple reference frame help predictions ? Uncovered background Uncovered background Non-integer pixel displacement Non-integer pixel displacement Lighting change Lighting change Camera shaking Camera shaking Noises in the source signal Noises in the source signal Noise effect

5 Introduction Background Knowledge (3/4) Sampling/noise effect of MobileCalendar Sampling/noise effect of MobileCalendar Original block (N-1) Ref block(N-2) Ref block (N-1) Residual block (N-2) Residual block MSE = MSE = 32.4

6 Introduction Background Knowledge (4/4) Drawback of MRF-ME Drawback of MRF-ME High computational complexity High computational complexity In order to reduce complexity In order to reduce complexity Reduce search points Reduce search points Continuous tracking technique Continuous tracking technique Early stop criteria Early stop criteria

7 Introduction Main Purpose (1/1) A simple and effective method of selecting proper reference frames in MRF-ME. A simple and effective method of selecting proper reference frames in MRF-ME. It enables working with any existing ME algorithms. It enables working with any existing ME algorithms. Experimental results demonstrate the effectiveness of proposed algorithm. Experimental results demonstrate the effectiveness of proposed algorithm.

8 H.264 Inter-Coding Overview (1/3) Variable block size Variable block size Macroblock partition : 16*16, 16*8, 8*16, 8*8 Macroblock partition : 16*16, 16*8, 8*16, 8*8 Submacroblock partition : 8*8, 8*4, 4*8, 4*4 Submacroblock partition : 8*8, 8*4, 4*8, 4*4 16*16 16*8 8*16 8*8 8*4 4*84*4

9 H.264 Inter-Coding Overview (2/3) Reference parameter REF Reference parameter REF Signal to which frame referred Signal to which frame referred Coded only once for each submacroblock partition Coded only once for each submacroblock partition All subblock smaller than 8*8 in the same subpartition must refer to the same frame

10 H.264 Inter-Coding Overview (3/3) Rate-distortion cost of each possible partition Rate-distortion cost of each possible partition REF : Reference parameter m( REF ) : The motion vector in the reference frame p ( REF ) : The predicted motion vector from the neighbors s : Original video signal c : Coded video signal R : Rate function of motion vectors SA(T)D : SAD or SATD

Reference frame selector Reference frame selector Treating 8*8 block as a minimal unit Treating 8*8 block as a minimal unit Making the selection using a mode 8*8 motion search Making the selection using a mode 8*8 motion search 11 Proposed Method Multiple Reference Frame Selection (1/9)

Flow chart Flow chart 12 Proposed Method Multiple Reference Frame Selection (2/9) Input a MB and perform four 8*8 block motion search on reference frame t-1 Variance of four MVs of 8*8 blocks ≦ TH? Perform motion estimation on reference frame t-1 Enter 2nd stage Yes No Output results First Stage

Flow chart Flow chart 13 Proposed Method Multiple Reference Frame Selection (3/9) Perform four 8*8 block motion search of the MB on the reference frame t-i Yes No i = 2 Is i ≧ N? i + 1 Is ref-frame t-k reffered by one of the four 8*8 blocks? Drop the non-referred ref frame t-k Is k ≧ N? Perform motion estimation on each valid reference frame Output results Second Stage Yes No

First stage First stage Disjoint 16*16 macroblock into four 8*8 blocks Disjoint 16*16 macroblock into four 8*8 blocks Motion search on ref-frame t-1 and check threshold Motion search on ref-frame t-1 and check threshold 14 Proposed Method Multiple Reference Frame Selection (4/9) Current Frame t Reference frame t MVs and corresponding minimal R-D cost

Motion vectors Motion vectors R-D cost R-D cost Check x and y components of V t-1 Check x and y components of V t-1 Less than a small threshold ? Less than a small threshold ?  Encoder terminate early by designating only the  Encoder terminate early by designating only the previous frame as reference frame previous frame as reference frame 15 Proposed Method Multiple Reference Frame Selection (5/9) Early stop criteria

The design of threshold depends upon the computational capacity of the encoder The design of threshold depends upon the computational capacity of the encoder Larger threshold, lower complexity, but worse coding efficiency Larger threshold, lower complexity, but worse coding efficiency 16 Proposed Method Multiple Reference Frame Selection (6/9) Determine upper bound of the coding efficiency and the worst case of complexity

Second stage Second stage Motion search of mode 8*8 should be tested on all of the remaining reference frames Motion search of mode 8*8 should be tested on all of the remaining reference frames 17 Proposed Method Multiple Reference Frame Selection (7/9) N : The maximal number of reference frames

For a given block index i, we let k be set as For a given block index i, we let k be set as The frame selector sets the ref-frame n-k as the valid, qualified reference frame The frame selector sets the ref-frame n-k as the valid, qualified reference frame 18 Proposed Method Multiple Reference Frame Selection (8/9) Block i has the best motion vector with the lowest cost, by referring to reference frame n-k

Drop unqualified frames Drop unqualified frames 19 Proposed Method Multiple Reference Frame Selection (9/9) Current frame t Frame t-2Frame t-3Frame t-4 Frame t-5 N = 5

Verifying the effectiveness of frame selector Verifying the effectiveness of frame selector Measuring the hit rate by comparing retained frames with the actual frames used via exhaustive search method (hit rate = 88% ~ 95%) Measuring the hit rate by comparing retained frames with the actual frames used via exhaustive search method (hit rate = 88% ~ 95%) 20 Proposed Method Analysis hit rate and Frame usage(1/6)

Even with the high hit rate, we expect that frame selector can drop as many frames as possible. Even with the high hit rate, we expect that frame selector can drop as many frames as possible. The false alarm ranges from 13%~32% The false alarm ranges from 13%~32% 21 Proposed Method Analysis hit rate and Frame usage(2/6)

Reference frame usage of the proposed frame selector for various video sequence Reference frame usage of the proposed frame selector for various video sequence 22 Proposed Method Analysis hit rate and Frame usage(3/6)

Why using mode 8*8 but not other modes in frame selection? Why using mode 8*8 but not other modes in frame selection? Hit rate Hit rate False alarm False alarm Motion estimation time spending Motion estimation time spending 23 Proposed Method Analysis hit rate and Frame usage(4/6)

Analysis of hit rate and false alarm using different modes on various sequence Analysis of hit rate and false alarm using different modes on various sequence 24 Proposed Method Analysis hit rate and Frame usage(5/6)

The motion estimation time spending on a MB The motion estimation time spending on a MB Judging from the hit rate, false alarm and complexity Judging from the hit rate, false alarm and complexity 8*8 is the best choice 8*8 is the best choice 25 Proposed Method Analysis hit rate and Frame usage(6/6) Mode16*1616*88*168*88*44*84*4 Time

Reference software : JM 9.2 Reference software : JM 9.2 Intel Pentium GHz with 512MB RAM Intel Pentium GHz with 512MB RAM Encoder parameters Encoder parameters 26 Experimental Results Overview (1/2)

Using Fast Full Search (FFS) and Fast Motion Estimation (FME) in JM 9.2 Using Fast Full Search (FFS) and Fast Motion Estimation (FME) in JM 9.2 Comparing the results with a frame selection method of Li.[1] Comparing the results with a frame selection method of Li.[1] 27 Experimental Results Overview (2/2) [1] X. Li, E. Q. Li, and Y. K. Chen, “Fast multi-frame motion estimation algorithm with adaptive search strategies in H.264,”in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., May 2004, vol.3, pp.369–372.

R-D curve comparison (Foreman) R-D curve comparison (Foreman) 28 Experimental Results

R-D curve comparison (Mobile) R-D curve comparison (Mobile) 29 Experimental Results

Using BDPSNR and BDBR to measure the performance difference between the methods Using BDPSNR and BDBR to measure the performance difference between the methods Calculates average PSNR and bitrate distance between two RD curves of two method, respectively. Calculates average PSNR and bitrate distance between two RD curves of two method, respectively. 30 Experimental Results

Discussing the computational complexity Discussing the computational complexity 31 Experimental Results

Speed up of SAD and SATD calculations Speed up of SAD and SATD calculations 32 Experimental Results

Number of MBs references in each reference frame (Mobile) Number of MBs references in each reference frame (Mobile) 33 Experimental Results Experimental Results

Number of MBs references in each reference frame (Foreman) Number of MBs references in each reference frame (Foreman) 34 Experimental Results Experimental Results

An efficient reference frame selector is proposed for the h.264 encoder to deal with the complexity issue pertaining to MRFME. An efficient reference frame selector is proposed for the h.264 encoder to deal with the complexity issue pertaining to MRFME. The experimental results demonstrate that the proposed algorithm can reduce significantly the complexity of ME at the encoder end, while keeping almost the same R-D performance as FFS. The experimental results demonstrate that the proposed algorithm can reduce significantly the complexity of ME at the encoder end, while keeping almost the same R-D performance as FFS. 35 Conclusions Conclusions