1. Video Coding For Compression... and Beyond Bernd Girod Information Systems Laboratory Department of Electrical Engineering Stanford University

2. Bernd Girod: Video Coding for Compression and Beyond 2 Bit Consumption of US Households Total for 70M households~230 Exabyte/year Television94% Radio1.7% Recorded Music0.4% Newspaper0.0003% Books0.0002% Magazines0.0002% Home video3.3% Video games0.6% Internet0.0003% [Source: UC Berkeley: How much Information] Bit equivalent, assuming state-of-the-art compression, year 2000

3. Bernd Girod: Video Coding for Compression and Beyond 3 Desirable Compression Ratios DSL ~200 kbps ~ 1,000 : 1 Dial-up modem, wireless link ~ 20 kbps ~ 10,000 : 1 ITU-R 601 166 Mbps CIF QCIF SDTV broadcasting ~2 Mbps ~ 100 : 1

4. Bernd Girod: Video Coding for Compression and Beyond 4 Outline Video compression – state-of-the-art Beyond compression –Rate-scalable video –Wavelet video coding –Error-resilient video transmission –Unequal error protection –Optimal scheduling for packet networks –Distributed video coding

5. Bernd Girod: Video Coding for Compression and Beyond 5 Outline Video compression – state-of-the-art Beyond compression –Rate-scalable video –Wavelet video coding –Error-resilient video transmission –Unequal error protection –Optimal scheduling for packet networks –Distributed video coding

6. Bernd Girod: Video Coding for Compression and Beyond 6 “It has been customary in the past to transmit successive complete images of the transmitted picture.” [...] “In accordance with this invention, this difficulty is avoided by transmitting only the difference between successive images of the object.”

7. Bernd Girod: Video Coding for Compression and Beyond 7 Motion-Compensated Hybrid Coding Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in

8. Bernd Girod: Video Coding for Compression and Beyond 8 Motion-Compensated Hybrid Coding Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in ¼-pixel accuracy

9. Bernd Girod: Video Coding for Compression and Beyond 9 Motion-Compensated Hybrid Coding Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in Adaptive block sizes...

10. Bernd Girod: Video Coding for Compression and Beyond 10 Motion-Compensated Hybrid Coding Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in Multiple Past Reference Frames

11. Bernd Girod: Video Coding for Compression and Beyond 11 Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Motion-Compensated Hybrid Coding Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in Generalized B-Frames

12. Bernd Girod: Video Coding for Compression and Beyond 12 Rate-Distortion Optimized Coder Control Minimize Lagrangian cost function Strategy: minimize J i for each block i separately, using a common Lagrange multiplier Total distortion Total bit-rate Distortion for block i Rate for block i Lagrangian cost for block i

13. Bernd Girod: Video Coding for Compression and Beyond 13 Multiple Reference Frames in H.264/AVC Mobile & Calendar (CIF, 30 fps) 01234 26 27 28 29 30 31 32 33 34 35 36 37 38 R [Mbit/s] PSNR Y [dB] PBB... with generalized B pictures PBB... with classic B pictures PPP... with 5 previous references PPP... with 1 previous reference ~15%

14. Bernd Girod: Video Coding for Compression and Beyond 14 Mobile & Calendar (CIF, 30 fps) 01234 26 27 28 29 30 31 32 33 34 35 36 37 38 R [Mbit/s] PSNR Y [dB] PBB... with generalized B pictures PBB... with classic B pictures PPP... with 5 previous references PPP... with 1 previous reference >25% Multiple Reference Frames in H.264/AVC

15. Bernd Girod: Video Coding for Compression and Beyond 15 Mobile & Calendar (CIF, 30 fps) 01234 26 27 28 29 30 31 32 33 34 35 36 37 38 R [Mbit/s] PSNR Y [dB] PBB... with generalized B pictures PBB... with classic B pictures PPP... with 5 previous references PPP... with 1 previous reference ~40% Multiple Reference Frames in H.264/AVC

16. Bernd Girod: Video Coding for Compression and Beyond 16 Outline Video compression – state-of-the-art Beyond compression –Rate-scalable video –Wavelet video coding –Error-resilient video transmission –Unequal error protection –Optimal scheduling for packet networks –Distributed video coding

17. Bernd Girod: Video Coding for Compression and Beyond 17 ?? Internet video streaming Surprising Success of ITU-T Rec. H.263 What H.263 was developed for... Analog videophone... and what is was used for.

18. Bernd Girod: Video Coding for Compression and Beyond 18 Internet Video Streaming How to accommodate heterogeneous bit-rates? How to react to network congestion? How to mitigate late or lost packets? Streaming client DSL dial-up modem Media Server Internet wireless

19. Bernd Girod: Video Coding for Compression and Beyond 19 Fine Granular Scalability (FGS) ~2dB gap H.264 with/without FGS option Foreman sequence (5fps) Base layer 20 kbps Enhancement layer variable bit-rate Efficiency gap

20. Bernd Girod: Video Coding for Compression and Beyond 20 Wavelet Video Coder Temporal Wavelet Transform Spatial Wavelet Transform Spatial Wavelet Transform 7 6 5 4 3 2 1 0 H H LLL LLH LH Original video frames H H H H H H H H H H H H H H H H Embedded Quantization & Entropy Coding Embedded Quantization & Entropy Coding [Taubman & Zakhor, 1994] [Ohm, 1994] [Choi & Woods, 1999] [Hsiang & Woods, VCIP ’99]... and others

21. Bernd Girod: Video Coding for Compression and Beyond 21 Lifting PU Even Frames Synthesis: Odd Frames Low Band High Band PU Even Frames Analysis: Odd Frames Low Band High Band Motion Compensation [Secker & Taubman, 2001] [Popescu & Bottreau, 2001]

22. Bernd Girod: Video Coding for Compression and Beyond 22 MC Wavelet Coding vs. H.264/AVC 2.0 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 36 34 32 30 28 26 24 22 20 38 Luminance PSNR (dB) bit-rate (Mbps) Scalable MC 5/3 Wavelet Non-scalableH.264/AVC Sequence: Mobile CIF H.264/AVC high complexity RD control CABAC PBBPBBP... 5 prev/3 future reference frames data courtesy of M. Flierl [Taubman & Secker, VCIP 2003] courtesy D. Taubman

23. Bernd Girod: Video Coding for Compression and Beyond 23 Wavelet Synthesis with Lossy Motion Vector MC Wavelet Transform MC Wavelet Transform Motion Estimator Motion Estimator Embedded Encoding Embedded Encoding Embedded Encoding Embedded Encoding Decoder Inverse Wavelet Transform Inverse Wavelet Transform Video in Video out [Taubman & Secker, ICIP03] Minimize J=D+ R Minimize J=D+ R

24. Bernd Girod: Video Coding for Compression and Beyond 24 R-D Performance with Lossy Motion Vector Bit - Rate (kbps) Video PSNR (dB) 020040060080010001200 24 26 28 30 32 34 36 3840 Embedded wavelet coefficients Lossless motion Non-embeddedsingle-rate Embedded wavelet coefficients Lossy motion CIF Foreman [Taubman & Secker, VCIP 2003] courtesy D. Taubman

25. Bernd Girod: Video Coding for Compression and Beyond 25 Outline Video compression – state-of-the-art Beyond compression –Rate-scalable video –Wavelet video coding –Error-resilient video transmission –Unequal error protection –Optimal scheduling for packet networks –Distributed video coding

26. Bernd Girod: Video Coding for Compression and Beyond 26 redundancy symbols enhancement layerbase layer Priority Encoding Transmission (PET) information symbols block of packets Reed-Solomon codeword K N-K [Albanese, Blömer, Edmonds, Luby, Sudan, 1996][Davis & Danskin, 1996] [Horn, Stuhlmuller, Link, Girod, 1999][Puri, Ramchandran, 1999] [Mohr, Riskin, Ladner, 2000][Stankovic, Hamzaoui, Xiong, 2002] [Chou, Wang, Padmanabhan, 2003]... and many more... packet network …

27. Bernd Girod: Video Coding for Compression and Beyond 27 Packet Delay Jitter and Loss delay     pdf lead-time loss probability lead-time loss probability loss

28. Bernd Girod: Video Coding for Compression and Beyond 28 Smart Prefetching Idea: Send more important packets earlier to allow for more retransmissions Server Client Internet Request stream Request stream Rate-distortion preamble Rate-distortion preamble Packet Schedule Packet Schedule Video packets Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule [Podolsky, McCanne, Vetterli 2000] [Miao, Ortega 2000] [Chou, Miao 2001]

29. Bernd Girod: Video Coding for Compression and Beyond 29 Rate-Distortion Preamble Each media packet n is labeled by − B n — size [in bits] of data unit n −  d n —distortion reduction if n is decoded − t n — decoding deadline for n PPI I BBBPPPI I BBBP… … …

30. Bernd Girod: Video Coding for Compression and Beyond 30 PB Rate-Distortion Preamble Each media packet n is labeled by − B n — size [in bits] of data unit n −  d n —distortion reduction if n is decoded − t n — decoding deadline for n PPI I BBPPI I BBBP… … … For video:  d n must be made “state-dependent” to accurately capture concealment For video:  d n must be made “state-dependent” to accurately capture concealment

31. Bernd Girod: Video Coding for Compression and Beyond 31 Markov Decision Tree for One Packet... N transmission opportunities before deadline send: 1 ack: 1 0 0 0 send: 1 0 0 ack: 1 0 1 0 1 0 0 1 1 1 0 0 0 0 t current t current +  tt current +2  t Action Observation “Policy“ minimizing J = D + R “Policy“ minimizing J = D + R

32. Bernd Girod: Video Coding for Compression and Beyond 32 R-D Optimized Streaming Performance Foreman 120 frames 10 fps, I-P-P-… H.263+ 2 Layer SNR scalable 20 frame GOP Copy Concealment 20 % loss forward and back Γ-distributed delay –κ = 10 ms –μ = 50 ms –σ = 23 ms Pre-roll 400ms Foreman 120 frames 10 fps, I-P-P-… H.263+ 2 Layer SNR scalable 20 frame GOP Copy Concealment 20 % loss forward and back Γ-distributed delay –κ = 10 ms –μ = 50 ms –σ = 23 ms Pre-roll 400ms PSNR [dB] Bit-Rate [kbps] ~50 %

33. Bernd Girod: Video Coding for Compression and Beyond 33 Naive Coding Questions 1. To achieve graceful degradation in case of channel error for a digitally encoded signal, is an embedded signal representation (aka layers, aka data partitioning) always needed? 2. Can one, in general, send refinement information for an analog (i.e. uncoded) signal transmission over a noisy channel?

34. Bernd Girod: Video Coding for Compression and Beyond 34 Digitally Enhanced Analog Transmission Forward error protection of the signal waveform Information-theoretic bounds [Shamai, Verdu, Zamir,1998] “Systematic lossy source-channel coding” Wyner- Ziv Encoder Wyner- Ziv Encoder Digital Channel Digital Channel Wyner- Ziv Decoder Wyner- Ziv Decoder Side info Analog Channel (uncoded) Analog Channel (uncoded)

35. Bernd Girod: Video Coding for Compression and Beyond 35 Forward Error Protection of Compressed Video Any Old Video Encoder Video Decoder with Error Concealment Error-Prone channel S S’ Wyner-Ziv Decoder A S*S* Wyner-Ziv Encoder A Wyner-Ziv Decoder B S ** Wyner-Ziv Encoder B Graceful degradation without a layered signal representation Analog channel (uncoded) [Aaron, Rane, Girod, ICIP 2003]

36. Bernd Girod: Video Coding for Compression and Beyond 36 Wyner-Ziv MPEG Codec Channel Slepian-Wolf Encoder Wyner-Ziv Encoder ED T -1 Q -1 + MC S*S* MPEG Encoder main S Side information MPEG Encoder coarse T -1 q -1 ED + MC S’ R-S Decoder Reconstructed Frame at Encoder MPEG Encoder coarse R-S Encoder [Rane, Aaron, Girod, VCIP 2004]

37. Bernd Girod: Video Coding for Compression and Beyond 37 Graceful Degradation with Forward Error Protection Main Stream @ 1.092 Mbps FEC (n,k) = (40,36) FEC bitrate = 120 Kbps Total = 1.2 Mbps WZ Stream @ 270 Kbps FEP (n,k) = (52,36) WZ bitrate = 120 Kbps Total = 1.2 Mbps

38. Bernd Girod: Video Coding for Compression and Beyond 38 Visual Comparison of Degradation at Same PSNR With FEC 1 Mbps + 120 kbps (38.32 db) Foreman 50 CIF frames @ symbol error rate = 4 x 10 -4 With FEP 1 Mbps + 120 kbps (38.78 db)

39. Bernd Girod: Video Coding for Compression and Beyond 39 Superior Robustness of FEP With FEC 1 Mbps + 120 kbps (33.03 db) Foreman 50 CIF frames @ symbol error rate = 10 -3 With FEP 1 Mbps + 120 kbps (38.40 db)

40. Bernd Girod: Video Coding for Compression and Beyond 40 Lossy Compression with Side Information Source Encoder Decoder Source Encoder Decoder [Wyner, Ziv, 1976] For mse distortion and Gaussian statistics, rate-distortion functions of the two systems are the same.

41. Bernd Girod: Video Coding for Compression and Beyond 41 Ultra-Low-Complexity Video Coding Interframe Decoder Intraframe Encoder K’ Interpolation/ Extrapolation Key frames K Conventional Intraframe coding Conventional Intraframe decoding X’ Scalar Quantizer Turbo Encoder Buffer WZ frames X Turbo Decoder Request bits Slepian-Wolf Codec Reconstruction Y [Aaron, Zhang, Girod, Asilomar 2002] [Aaron, Rane, Zhang, Girod, DCC 2003]

42. Bernd Girod: Video Coding for Compression and Beyond 42 R-D Performance Ultra-Low-Complexity Video Coder 8 dB 3 dB Sequence: Foreman WZ frames - even frames Key frames - odd frames Side information - motion compensated interpolation of key frames

43. Bernd Girod: Video Coding for Compression and Beyond 43 H263+ Intraframe Coding 330 kbps, 32.9 dB Wyner-Ziv Codec 274 kbps, 39.0 dB Ultra-Low-Complexity Video Coder

44. Bernd Girod: Video Coding for Compression and Beyond 44 H263+ I-B-I-B 276 kbps, 41.8 dB Wyner-Ziv Codec 274 kbps, 39.0 dB Ultra-Low-Complexity Video Coder

45. Bernd Girod: Video Coding for Compression and Beyond 45 Stanford Camera Array Courtesy Marc Levoy, Stanford Computer Graphics Lab

46. Bernd Girod: Video Coding for Compression and Beyond 46 Stanford Camera Array Courtesy Marc Levoy, Stanford Computer Graphics Lab

47. Bernd Girod: Video Coding for Compression and Beyond 47 Light Field Compression Rate: 0.11 bpp PSNR 39.9 dB Rate: 0.11 bpp PSNR 37.4 dB Wyner-Ziv, Pixel-Domain JPEG-2000

48. Bernd Girod: Video Coding for Compression and Beyond 48 Conclusions Video compression is very important... but there is more to video coding than compression Rate-scalable video representations: mc lifting break-through Robust video transmission –Virtual priority mechanisms by packet scheduling –RD gains easily larger than from super-clever compression Distributed video coding: radically different approach –Graceful degradation w/o layers –Ultra-low-complexity coders Ubiquitous J=D+ R

49. Acknowledgments Anne M. Aaron Anne M. Aaron Jacob Chakareski Philip A. Chou J=D+ R Markus Flierl Sang-eun Han Mark Kalman Marc Levoy Yi Liang Shantanu Rane David Rebollo-Monedero Andrew Secker David Taubman Thomas Wiegand Xiaoqing Zhu Rui Zhang

50. Progress is a wonderful thing, if only it would stop... Robert Musil