1. An Algorithm For Constant- Quality Compressed Video Michael F. Ringenburg Richard E. Ladner Eve A. Riskin UW CSE Industrial Affiliates Meeting February 26, 2004

2. The Quick Introduction to Video Compression 4 Steps: 4 Steps: Motion compensation. Motion compensation. Outputs motion vectors, which predict the frame from the previous frame. Outputs motion vectors, which predict the frame from the previous frame. Actual - Predicted = Residual Actual - Predicted = Residual Transform the residual. Transform the residual. Concentrates the energy (information). Concentrates the energy (information). Lossy encoding of the transformed residual. Lossy encoding of the transformed residual. Decoding. Decoding. Generates an approximation of the original frame. Generates an approximation of the original frame.

3. Motion Compensation T=1 (reference)T=2 (current) Thanks to Gidon Shavit

4. Residual Examples (Boosted) OriginalResidual Thanks to Gidon Shavit

5. The Quick Introduction to Video Compression 4 Steps: 4 Steps: Motion compensation. Motion compensation. Outputs motion vectors, which predict the frame from the previous frame. Outputs motion vectors, which predict the frame from the previous frame. Actual - Predicted = Residual Actual - Predicted = Residual Transform the residual. Transform the residual. Concentrates the energy (information). Concentrates the energy (information). Lossy encoding of the transformed residual. Lossy encoding of the transformed residual. Decoding. Decoding. Generates an approximation of the original frame. Generates an approximation of the original frame.

6. Transform Example (DCT) Thanks to Gidon Shavit

7. The Quick Introduction to Video Compression 4 Steps: 4 Steps: Motion compensation. Motion compensation. Outputs motion vectors, which predict the frame from the previous frame. Outputs motion vectors, which predict the frame from the previous frame. Actual - Predicted = Residual Actual - Predicted = Residual Transform the residual. Transform the residual. Concentrates the energy (information). Concentrates the energy (information). Lossy encoding of the transformed residual. Lossy encoding of the transformed residual. Quantization vs. Embedded Coding Quantization vs. Embedded Coding Decoding. Decoding. Generates an approximation of the original frame. Generates an approximation of the original frame.

8. Embedded Bit Plane Coding The n-th bit plane consists of the n-th bit of each coefficient. The n-th bit plane consists of the n-th bit of each coefficient. Each bit plane is compressed with an embedded, lossless code. Each bit plane is compressed with an embedded, lossless code. Send the bit planes in order of significance. Send the bit planes in order of significance. The most important information is sent first. The most important information is sent first. Can terminate the stream between or within bit planes. Can terminate the stream between or within bit planes.

9. The Quick Introduction to Video Compression 4 Steps: 4 Steps: Motion compensation. Motion compensation. Outputs motion vectors, which predict the frame from the previous frame. Outputs motion vectors, which predict the frame from the previous frame. Actual - Predicted = Residual Actual - Predicted = Residual Transform the residual. Transform the residual. Concentrates the energy (information). Concentrates the energy (information). Lossy encoding of the transformed residual. Lossy encoding of the transformed residual. Decoding. Decoding. Generates an approximation of the original frame. Generates an approximation of the original frame.

10. The GTV Coder An embedded bit plane video coder developed at the University of Washington. An embedded bit plane video coder developed at the University of Washington. Written by Jeff West, Gidon Shavit and Michael Ringenburg, with Richard Ladner and Eve Riskin. Written by Jeff West, Gidon Shavit and Michael Ringenburg, with Richard Ladner and Eve Riskin. Motion Compensation on 16 x 16 blocks Motion Compensation on 16 x 16 blocks 8 x 8 Discrete Cosine Transform 8 x 8 Discrete Cosine Transform Lossy coding based on the GT-DCT image coder developed by Ed Hong with Richard Ladner and Eve Riskin Lossy coding based on the GT-DCT image coder developed by Ed Hong with Richard Ladner and Eve Riskin All results in this talk use the GTV coder All results in this talk use the GTV coder

11. Interframe Bit Allocation Given : An embedded coder, a maximum bit budget B and a video with F frames. Given : An embedded coder, a maximum bit budget B and a video with F frames. How many bits b i should we allocate to each frame f i in order to achieve constant quality video? How many bits b i should we allocate to each frame f i in order to achieve constant quality video? Formally, our constraint is: Formally, our constraint is:

12. Why Is This Important? Some frames require more bits (are harder) to encode, because … Some frames require more bits (are harder) to encode, because … Scene changes prevent good motion compensation. Scene changes prevent good motion compensation. As do lighting changes. As do lighting changes. And, to a lesser extent, dramatic actions. And, to a lesser extent, dramatic actions. For example, set your TiVo to it’s lowest quality/highest compression setting. Watch the scene change blockiness at playback For example, set your TiVo to it’s lowest quality/highest compression setting. Watch the scene change blockiness at playback

13. Why Is This Difficult? Motion Compensation!!! Motion Compensation!!! The quality of frame n is a function of the allocation to frame n, and the quality of frame (n-1). The quality of frame n is a function of the allocation to frame n, and the quality of frame (n-1). Quality of frame (n-1) is a function of the allocation to frame (n-1) and the quality of frame (n-2). Et cetera. Quality of frame (n-1) is a function of the allocation to frame (n-1) and the quality of frame (n-2). Et cetera. Thus, the quality of frame n is a function of n variables. Thus, the quality of frame n is a function of n variables. The quality function (a.k.a. rate-distortion curve) is different for every frame. The quality function (a.k.a. rate-distortion curve) is different for every frame. The rate-distortion curves are not monotonic. The rate-distortion curves are not monotonic. Approximately monotonic, though. Approximately monotonic, though.

14. MultiStage Algorithm A global allocation algorithm (allocates across the entire video). A global allocation algorithm (allocates across the entire video). Alternates between two stages which provide feedback to each other. Alternates between two stages which provide feedback to each other. The Constant Quality Stage The Constant Quality Stage The Target Allocation Stage The Target Allocation Stage After every stage, check if we have constant quality and the target allocation. After every stage, check if we have constant quality and the target allocation.

15. Example, Step 1 Constant Quality T=2000

16. Example, Step 2 Target Allocation T=1000

17. Example, Step 3 Constant Quality T=800

18. Example, Step 4 Target Allocation T=1000

19. Method Comparison

21. Conclusions Developed two new global interframe bit allocation algorithms for embedded video coders Developed two new global interframe bit allocation algorithms for embedded video coders MultiStage algorithm MultiStage algorithm Ratio algorithm (not presented here) Ratio algorithm (not presented here) MultiStage has the lowest variance, and the highest minimum PSNR. Nearly constant quality. MultiStage has the lowest variance, and the highest minimum PSNR. Nearly constant quality. Full paper at: Full paper at: http://www.cs.washington.edu/homes/miker/Ringenbu rgAllocation.pdf http://www.cs.washington.edu/homes/miker/Ringenbu rgAllocation.pdf