Video Coding For Compression... and Beyond Bernd Girod Information Systems Laboratory Department of Electrical Engineering Stanford University

Bernd Girod: Video Coding for Compression and Beyond 2 Bit Consumption of US Households Total for 70M households~230 Exabyte/year Television94% Radio1.7% Recorded Music0.4% Newspaper0.0003% Books0.0002% Magazines0.0002% Home video3.3% Video games0.6% Internet0.0003% [Source: UC Berkeley: How much Information] Bit equivalent, assuming state-of-the-art compression, year 2000

Bernd Girod: Video Coding for Compression and Beyond 3 Desirable Compression Ratios DSL ~200 kbps ~ 1,000 : 1 Dial-up modem, wireless link ~ 20 kbps ~ 10,000 : 1 ITU-R Mbps CIF QCIF SDTV broadcasting ~2 Mbps ~ 100 : 1

Bernd Girod: Video Coding for Compression and Beyond 4 Outline Video compression – state-of-the-art Beyond compression –Rate-scalable video –Wavelet video coding –Error-resilient video transmission –Unequal error protection –Optimal scheduling for packet networks –Distributed video coding

Bernd Girod: Video Coding for Compression and Beyond 5 Outline Video compression – state-of-the-art Beyond compression –Rate-scalable video –Wavelet video coding –Error-resilient video transmission –Unequal error protection –Optimal scheduling for packet networks –Distributed video coding

Bernd Girod: Video Coding for Compression and Beyond 6 “It has been customary in the past to transmit successive complete images of the transmitted picture.” [...] “In accordance with this invention, this difficulty is avoided by transmitting only the difference between successive images of the object.”

Bernd Girod: Video Coding for Compression and Beyond 7 Motion-Compensated Hybrid Coding Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in

Bernd Girod: Video Coding for Compression and Beyond 8 Motion-Compensated Hybrid Coding Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in ¼-pixel accuracy

Bernd Girod: Video Coding for Compression and Beyond 9 Motion-Compensated Hybrid Coding Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in Adaptive block sizes...

Bernd Girod: Video Coding for Compression and Beyond 10 Motion-Compensated Hybrid Coding Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in Multiple Past Reference Frames

Bernd Girod: Video Coding for Compression and Beyond 11 Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Motion-Compensated Hybrid Coding Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in Generalized B-Frames

Bernd Girod: Video Coding for Compression and Beyond 12 Rate-Distortion Optimized Coder Control Minimize Lagrangian cost function Strategy: minimize J i for each block i separately, using a common Lagrange multiplier Total distortion Total bit-rate Distortion for block i Rate for block i Lagrangian cost for block i

Bernd Girod: Video Coding for Compression and Beyond 13 Multiple Reference Frames in H.264/AVC Mobile & Calendar (CIF, 30 fps) R [Mbit/s] PSNR Y [dB] PBB... with generalized B pictures PBB... with classic B pictures PPP... with 5 previous references PPP... with 1 previous reference ~15%

Bernd Girod: Video Coding for Compression and Beyond 14 Mobile & Calendar (CIF, 30 fps) R [Mbit/s] PSNR Y [dB] PBB... with generalized B pictures PBB... with classic B pictures PPP... with 5 previous references PPP... with 1 previous reference >25% Multiple Reference Frames in H.264/AVC

Bernd Girod: Video Coding for Compression and Beyond 15 Mobile & Calendar (CIF, 30 fps) R [Mbit/s] PSNR Y [dB] PBB... with generalized B pictures PBB... with classic B pictures PPP... with 5 previous references PPP... with 1 previous reference ~40% Multiple Reference Frames in H.264/AVC

Bernd Girod: Video Coding for Compression and Beyond 16 Outline Video compression – state-of-the-art Beyond compression –Rate-scalable video –Wavelet video coding –Error-resilient video transmission –Unequal error protection –Optimal scheduling for packet networks –Distributed video coding

Bernd Girod: Video Coding for Compression and Beyond 17 ?? Internet video streaming Surprising Success of ITU-T Rec. H.263 What H.263 was developed for... Analog videophone... and what is was used for.

Bernd Girod: Video Coding for Compression and Beyond 18 Internet Video Streaming How to accommodate heterogeneous bit-rates? How to react to network congestion? How to mitigate late or lost packets? Streaming client DSL dial-up modem Media Server Internet wireless

Bernd Girod: Video Coding for Compression and Beyond 19 Fine Granular Scalability (FGS) ~2dB gap H.264 with/without FGS option Foreman sequence (5fps) Base layer 20 kbps Enhancement layer variable bit-rate Efficiency gap

Bernd Girod: Video Coding for Compression and Beyond 20 Wavelet Video Coder Temporal Wavelet Transform Spatial Wavelet Transform Spatial Wavelet Transform H H LLL LLH LH Original video frames H H H H H H H H H H H H H H H H Embedded Quantization & Entropy Coding Embedded Quantization & Entropy Coding [Taubman & Zakhor, 1994] [Ohm, 1994] [Choi & Woods, 1999] [Hsiang & Woods, VCIP ’99]... and others

Bernd Girod: Video Coding for Compression and Beyond 21 Lifting PU Even Frames Synthesis: Odd Frames Low Band High Band PU Even Frames Analysis: Odd Frames Low Band High Band Motion Compensation [Secker & Taubman, 2001] [Popescu & Bottreau, 2001]

Bernd Girod: Video Coding for Compression and Beyond 22 MC Wavelet Coding vs. H.264/AVC Luminance PSNR (dB) bit-rate (Mbps) Scalable MC 5/3 Wavelet Non-scalableH.264/AVC Sequence: Mobile CIF H.264/AVC high complexity RD control CABAC PBBPBBP... 5 prev/3 future reference frames data courtesy of M. Flierl [Taubman & Secker, VCIP 2003] courtesy D. Taubman

Bernd Girod: Video Coding for Compression and Beyond 23 Wavelet Synthesis with Lossy Motion Vector MC Wavelet Transform MC Wavelet Transform Motion Estimator Motion Estimator Embedded Encoding Embedded Encoding Embedded Encoding Embedded Encoding Decoder Inverse Wavelet Transform Inverse Wavelet Transform Video in Video out [Taubman & Secker, ICIP03] Minimize J=D+ R Minimize J=D+ R

Bernd Girod: Video Coding for Compression and Beyond 24 R-D Performance with Lossy Motion Vector Bit - Rate (kbps) Video PSNR (dB) Embedded wavelet coefficients Lossless motion Non-embeddedsingle-rate Embedded wavelet coefficients Lossy motion CIF Foreman [Taubman & Secker, VCIP 2003] courtesy D. Taubman

Bernd Girod: Video Coding for Compression and Beyond 25 Outline Video compression – state-of-the-art Beyond compression –Rate-scalable video –Wavelet video coding –Error-resilient video transmission –Unequal error protection –Optimal scheduling for packet networks –Distributed video coding

Bernd Girod: Video Coding for Compression and Beyond 26 redundancy symbols enhancement layerbase layer Priority Encoding Transmission (PET) information symbols block of packets Reed-Solomon codeword K N-K [Albanese, Blömer, Edmonds, Luby, Sudan, 1996][Davis & Danskin, 1996] [Horn, Stuhlmuller, Link, Girod, 1999][Puri, Ramchandran, 1999] [Mohr, Riskin, Ladner, 2000][Stankovic, Hamzaoui, Xiong, 2002] [Chou, Wang, Padmanabhan, 2003]... and many more... packet network …

Bernd Girod: Video Coding for Compression and Beyond 27 Packet Delay Jitter and Loss delay     pdf lead-time loss probability lead-time loss probability loss

Bernd Girod: Video Coding for Compression and Beyond 28 Smart Prefetching Idea: Send more important packets earlier to allow for more retransmissions Server Client Internet Request stream Request stream Rate-distortion preamble Rate-distortion preamble Packet Schedule Packet Schedule Video packets Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule [Podolsky, McCanne, Vetterli 2000] [Miao, Ortega 2000] [Chou, Miao 2001]

Bernd Girod: Video Coding for Compression and Beyond 29 Rate-Distortion Preamble Each media packet n is labeled by − B n — size [in bits] of data unit n −  d n —distortion reduction if n is decoded − t n — decoding deadline for n PPI I BBBPPPI I BBBP… … …

Bernd Girod: Video Coding for Compression and Beyond 30 PB Rate-Distortion Preamble Each media packet n is labeled by − B n — size [in bits] of data unit n −  d n —distortion reduction if n is decoded − t n — decoding deadline for n PPI I BBPPI I BBBP… … … For video:  d n must be made “state-dependent” to accurately capture concealment For video:  d n must be made “state-dependent” to accurately capture concealment

Bernd Girod: Video Coding for Compression and Beyond 31 Markov Decision Tree for One Packet... N transmission opportunities before deadline send: 1 ack: send: ack: t current t current +  tt current +2  t Action Observation “Policy“ minimizing J = D + R “Policy“ minimizing J = D + R

Bernd Girod: Video Coding for Compression and Beyond 32 R-D Optimized Streaming Performance Foreman 120 frames 10 fps, I-P-P-… H Layer SNR scalable 20 frame GOP Copy Concealment 20 % loss forward and back Γ-distributed delay –κ = 10 ms –μ = 50 ms –σ = 23 ms Pre-roll 400ms Foreman 120 frames 10 fps, I-P-P-… H Layer SNR scalable 20 frame GOP Copy Concealment 20 % loss forward and back Γ-distributed delay –κ = 10 ms –μ = 50 ms –σ = 23 ms Pre-roll 400ms PSNR [dB] Bit-Rate [kbps] ~50 %

Bernd Girod: Video Coding for Compression and Beyond 33 Naive Coding Questions 1. To achieve graceful degradation in case of channel error for a digitally encoded signal, is an embedded signal representation (aka layers, aka data partitioning) always needed? 2. Can one, in general, send refinement information for an analog (i.e. uncoded) signal transmission over a noisy channel?

Bernd Girod: Video Coding for Compression and Beyond 34 Digitally Enhanced Analog Transmission Forward error protection of the signal waveform Information-theoretic bounds [Shamai, Verdu, Zamir,1998] “Systematic lossy source-channel coding” Wyner- Ziv Encoder Wyner- Ziv Encoder Digital Channel Digital Channel Wyner- Ziv Decoder Wyner- Ziv Decoder Side info Analog Channel (uncoded) Analog Channel (uncoded)

Bernd Girod: Video Coding for Compression and Beyond 35 Forward Error Protection of Compressed Video Any Old Video Encoder Video Decoder with Error Concealment Error-Prone channel S S’ Wyner-Ziv Decoder A S*S* Wyner-Ziv Encoder A Wyner-Ziv Decoder B S ** Wyner-Ziv Encoder B Graceful degradation without a layered signal representation Analog channel (uncoded) [Aaron, Rane, Girod, ICIP 2003]

Bernd Girod: Video Coding for Compression and Beyond 36 Wyner-Ziv MPEG Codec Channel Slepian-Wolf Encoder Wyner-Ziv Encoder ED T -1 Q -1 + MC S*S* MPEG Encoder main S Side information MPEG Encoder coarse T -1 q -1 ED + MC S’ R-S Decoder Reconstructed Frame at Encoder MPEG Encoder coarse R-S Encoder [Rane, Aaron, Girod, VCIP 2004]

Bernd Girod: Video Coding for Compression and Beyond 37 Graceful Degradation with Forward Error Protection Main Mbps FEC (n,k) = (40,36) FEC bitrate = 120 Kbps Total = 1.2 Mbps WZ 270 Kbps FEP (n,k) = (52,36) WZ bitrate = 120 Kbps Total = 1.2 Mbps

Bernd Girod: Video Coding for Compression and Beyond 38 Visual Comparison of Degradation at Same PSNR With FEC 1 Mbps kbps (38.32 db) Foreman 50 CIF symbol error rate = 4 x With FEP 1 Mbps kbps (38.78 db)

Bernd Girod: Video Coding for Compression and Beyond 39 Superior Robustness of FEP With FEC 1 Mbps kbps (33.03 db) Foreman 50 CIF symbol error rate = With FEP 1 Mbps kbps (38.40 db)

Bernd Girod: Video Coding for Compression and Beyond 40 Lossy Compression with Side Information Source Encoder Decoder Source Encoder Decoder [Wyner, Ziv, 1976] For mse distortion and Gaussian statistics, rate-distortion functions of the two systems are the same.

Bernd Girod: Video Coding for Compression and Beyond 41 Ultra-Low-Complexity Video Coding Interframe Decoder Intraframe Encoder K’ Interpolation/ Extrapolation Key frames K Conventional Intraframe coding Conventional Intraframe decoding X’ Scalar Quantizer Turbo Encoder Buffer WZ frames X Turbo Decoder Request bits Slepian-Wolf Codec Reconstruction Y [Aaron, Zhang, Girod, Asilomar 2002] [Aaron, Rane, Zhang, Girod, DCC 2003]

Bernd Girod: Video Coding for Compression and Beyond 42 R-D Performance Ultra-Low-Complexity Video Coder 8 dB 3 dB Sequence: Foreman WZ frames - even frames Key frames - odd frames Side information - motion compensated interpolation of key frames

Bernd Girod: Video Coding for Compression and Beyond 43 H263+ Intraframe Coding 330 kbps, 32.9 dB Wyner-Ziv Codec 274 kbps, 39.0 dB Ultra-Low-Complexity Video Coder

Bernd Girod: Video Coding for Compression and Beyond 44 H263+ I-B-I-B 276 kbps, 41.8 dB Wyner-Ziv Codec 274 kbps, 39.0 dB Ultra-Low-Complexity Video Coder

Bernd Girod: Video Coding for Compression and Beyond 45 Stanford Camera Array Courtesy Marc Levoy, Stanford Computer Graphics Lab

Bernd Girod: Video Coding for Compression and Beyond 46 Stanford Camera Array Courtesy Marc Levoy, Stanford Computer Graphics Lab

Bernd Girod: Video Coding for Compression and Beyond 47 Light Field Compression Rate: 0.11 bpp PSNR 39.9 dB Rate: 0.11 bpp PSNR 37.4 dB Wyner-Ziv, Pixel-Domain JPEG-2000

Bernd Girod: Video Coding for Compression and Beyond 48 Conclusions Video compression is very important... but there is more to video coding than compression Rate-scalable video representations: mc lifting break-through Robust video transmission –Virtual priority mechanisms by packet scheduling –RD gains easily larger than from super-clever compression Distributed video coding: radically different approach –Graceful degradation w/o layers –Ultra-low-complexity coders Ubiquitous J=D+ R

Acknowledgments Anne M. Aaron Anne M. Aaron Jacob Chakareski Philip A. Chou J=D+ R Markus Flierl Sang-eun Han Mark Kalman Marc Levoy Yi Liang Shantanu Rane David Rebollo-Monedero Andrew Secker David Taubman Thomas Wiegand Xiaoqing Zhu Rui Zhang

Progress is a wonderful thing, if only it would stop... Robert Musil