Frame by Frame Bit Allocation for Motion-Compensated Video Michael Ringenburg May 9, 2003

The Problem… Given a maximum bit budget B and a video with F frames, how many bits b i should we allocate to each frame f i in order to maximize the overall quality of the video? Formally, our constraint is: We assume a lossy, embedded coding scheme. Thus we can choose the exact number of bits to allocate to each frame.

Rate-Distortion Curves If we increase the number of bits allocated to a frame (the frame’s bit rate) and hold everything else constant, the frame’s distortion (the mean squared error or MSE) decreases. The distortion decays exponentially (2 -b )

Motion Compensation Each frame is predicted by the previous frame. We find blocks in the previous frame which are similar to blocks in the current frame, and calculate motion vectors which estimate the disparity between the previous and the current blocks. We only encode the difference, or the “residue”, between the predicted and actual frame. This complicates the task of bit allocation, because the quality of a frame depends not only on its rate, but also on the rates of all of the previous frames.

Measuring Video Quality We can measure the quality of individual frames with MSE (distortion) or PSNR (Peak Signal-to-Noise Ratio), but how do we measure the overall quality of the whole video? Method 1 (MMSE): Minimize the Mean MSE. Method 2 (MINMAX): Minimize the Maximum MSE. Leads to constant quality, which may be more visually appealing.

Outline of Talk Cheng-Li-Kuo ‘97 algorithm for MMSE Yang-Hemami ‘99 algorithm for MINMAX If time permits: –Adapting these algorithms for Group Testing for Video (GTV) –A new algorithm for the MINMAX metric

Preliminaries I frames: “Independent” frames - not predicted. Distortion of an I frame depends only on its own bit rate. P frames: “Predicted” frames. Distortion depends on own bit rate, plus bit rate of most recent I frame and all P frames in between. Group of Pictures (GOP): I frame, followed by some number of P frames. Both algorithms optimize individual GOP’s.

Cheng-Li-Kuo ‘97 Algorithm From “Rate Control for an Embedded Wavelet Video Coder”, by Po-Yuen Cheng, Jin Li, and C.-C. Jay Kuo. Minimizes the MMSE of each Group of Pictures. Based on experimental observations of rate- distortion and motion-compensation behavior of typical videos. Uses Lagrange Multiplier method to derive optimal allocations.

Rate-Distortion Curves Authors experimentally determined that Rate- Distortion curves can be approximated by: D max is distortion at rate 0. This is equivalent to the variance of the wavelet coefficients.

More on the ß parameter Larger ß indicates more efficient coding. ß I is typically larger than ß P, because the I frame quality is dependent only on its allocation. The ratio ß I / ß P is usually between 1.1 and 1.4 Examples: –Flower: ß I = 2.07, ß P = 1.50 –Mobile: ß I = 1.65, ß P = 1.50 –Tennis: ß I = 1.08, ß P = 0.86 –Cheer: ß I = 1.72, ß P = 1.43

Frame Dependency If e is the residue of motion compensation, f is the predicted frame, d is the displacement, and g is the reference frame after lossy encoding, then: Then if E represents the expected value, the variance is:

Frame Dependency Let a be the actual reference frame (as opposed to the reference frame after lossy encoding): This is the residue with respect to the original reference frame, plus the coding error of the reference frame. We assume they are not correlated.

Frame Dependency Let  a 2 be the variance if the reference frame was perfectly coded: The second part is the distortion of the motion-translated reference frame, which is linearly related to the actual distortion of the non-translated reference frame. Thus:

More on the  parameter Typically ranges from 0.5 to 0.9 Higher indicates better quality motion compensation. Decreases if there is violent motion or a scene change.

Lagrange multiplier method Using the experimentally observed rate distortion model and the frame dependency we just derived, we can solve for the optimal allocation using the Lagrange multiplier method. Let R GOP be the number of bits assigned to a group of pictures. We minimize:

Solution The authors solve the minimization, and derive:

Parameters We need to determine the variance  a 2, the coding efficiency ß, and the dependency  for every frame in the Group Of Pictures before we begin coding it. Alternatively, if this is too expensive, we can estimate the values using the previous GOP. K is an adjustable parameter. We perform a binary search until the rate constraint is met.

Experimental Results

Yang-Hemami ‘99 Algorithm From “MINMAX Frame Rate Control Using a Rate-Distortion Optimized Wavelet Coder”, by Yan Yang and Sheila Hemami. Minimizes the maximum distortion of any frame in the Group of Pictures. Leads to constant quality within a GOP.

Outline of Algorithm Let N be the number of frames in a GOP. Recall the first frame is an I frame, the rest are P frames. Step 1: Find rates R I and R P such that: R I + (N-1)R P =R t and D 1 (R I ) = D 2 (R P )=D. Step 2: Code the rest of the frames to distortion D. If the rate starts to get too high, adaptively raise D. If the rate starts to get too low, adaptively lower D.

Finding the Initial Rates Initially assume that all P frames have identical Rate-Distortion curves. Binary search: –Let R(D) = R 1 (D) + (N-1)R 2 (D) –Find a D 1 and D 2 such that R(D 1 ) < R t < R(D 2 ) –Repeat until |R(D) - R t | <  = R t x 1%: Let D = (D 1 + D 2 )/2 If R(D) < R t let D 1 = D Else, let D 2 = D Variant – force D I to be slightly less than D P

Adaptive Adjustment Algorithm In reality, the P frames are not identical. Iterate over the P frames, coding each to the current target distortion D P. Let  be the mismatch between jR P and the number of bits actually used to code the first j P frames. Let  up and  low be upper and lower bounds on the allowable mismatch. If  >  up then raise D P using update algorithm. If  <  low then lower D P using update algorithm.

Update Algorithm Code current P frame j at rate R j = R j-1 -  /(N - j). If  is still outside the allowable range, we use the update algorithm again on the next frame. Once  is back within the acceptable range, we update the target distortion D P.

Experimental Results

Adapting for GTV Both algorithms optimize GOP’s. Group Testing for video doesn’t have GOP’s in the traditional sense – just a single I frame at the beginning of the video, and then only P frames. But GTV does have “pseudo”-I frames. When there is a scene change, or violent motion, the frame is not predicted very well, and thus it behaves like an I frame. They are usually much less frequent than traditional I frames, though, and not evenly distributed. Detect these frames by the residual magnitude.

Adapting Yang-Hemami ‘99 Approach 1: Run initial rate algorithm on first two frames. Run adaptive adjustment algorithm until we encounter a “pseudo”-I frame. Repeat. Approach 2: Create N frame GOP’s at the beginning and at every “pseudo”-I frame. Use constant bit-rate coding outside the GOP’s.

Adapting Cheng-Li-Kuo ‘97 Approach 1: Start a new GOP after N frames or at the next “pseudo”-I frame, whichever comes first. Even if the first frame of the new GOP is a P frame, it will behave like an I frame with a low ß value, because all previous allocations are fixed. Approach 2:Create N frame GOP’s at the beginning and at every “pseudo”-I frame. Use constant bit-rate coding outside the GOP’s.

New algorithm Set targetD to a small value. Repeat: –Encode all frames in GOP to distortion targetD –If |bits_used – max_bits| < , break. –Scale all allocations by max_bits/bits_used and encode GOP. –If variance of the frame distortions is less than max_variance, break. –Set targetD to average distortion.

My Project Implement Cheng-Li-Kuo, Yang- Hemami, and my new algorithm in the context of GTV. Compare the speed and quality of the three algorithms.