Distributed Video Coding VLBV, Sardinia, September 16, 2005 Bernd Girod Information Systems Laboratory Stanford University

B. Girod: Distributed Video Coding 2 Outline  Foundations of distributed coding –Slepian-Wolf Theorem and practical Slepian-Wolf coding –Wyner-Ziv results and practical Wyner-Ziv coding  Low-complexity video encoding –Pixel-domain and transform-domain coding –Hash-based receiver motion estimation –Wyner-Ziv residual coding  Error-resilient video transmission –Systematic lossy joint source-channel coding –Improving the error-resiliency of MPEG (or anything else) by Wyner-Ziv coding

B. Girod: Distributed Video Coding 3 Outline  Foundations of distributed coding –Slepian-Wolf Theorem and practical Slepian-Wolf coding –Wyner-Ziv results and practical Wyner-Ziv coding  Low-complexity video encoding –Pixel-domain and transform-domain coding –Hash-based receiver motion estimation  Error-resilient video transmission –Systematic lossy joint source-channel coding –Improving the error-resiliency of MPEG by Wyner-Ziv coding

B. Girod: Distributed Video Coding 4 Compression of Dependent Sources Source X Source Y Joint Decoder Joint Decoder X Y Joint Encoder Joint Encoder

B. Girod: Distributed Video Coding 5 Distributed Compression of Dependent Sources Source X Source Y Encoder X Encoder Y Joint Decoder Joint Decoder X Y

B. Girod: Distributed Video Coding 6 Slepian Wolf Theorem Independent decoding Achievable rate region for i.i.d sequences

B. Girod: Distributed Video Coding 7 Slepian Wolf Theorem Joint decoding: Vanishing error probability for long sequences Independent decoding: No errors [Slepian, Wolf, 1973] Achievable rate region for i.i.d sequences

B. Girod: Distributed Video Coding 8 Lossless Compression with Receiver Side Information Source Encoder Decoder

B. Girod: Distributed Video Coding 9 Distributed Compression and Channel Coding Idea  Interpret Y as a “noisy” version of X with “channel errors”   Encoder generates “parity bits” P to protect against errors   Decoder concatenates Y and P and performs error-correcting decoding Idea  Interpret Y as a “noisy” version of X with “channel errors”   Encoder generates “parity bits” P to protect against errors   Decoder concatenates Y and P and performs error-correcting decoding Source X|Y Encoder Decoder P

B. Girod: Distributed Video Coding 10 Towards Practical Slepian-Wolf Coding  Convolution coding for data compression [Blizard, 1969, unpublished]  Convolutional source coding [Hellman, 1975]  Coset codes [Pradhan and Ramchandran, 1999]  Trellis codes [Wang and Orchard, 2001]  Turbo codes [Garcia-Frias and Zhao, 2001] [Bajcsy and Mitran, 2001] [Aaron and Girod, 2002]  LDPC codes [Liveris, Xiong, and Georghiades, 2002] ...

B. Girod: Distributed Video Coding 11 Slepian-Wolf Coding Using Turbo Codes Systematic Convolutional Code Interleaver SISO Decoder Decision Parity bits Systematic bits X Y “Correlation channel” [Aaron and Girod, 2002]

B. Girod: Distributed Video Coding 12 Lossy Compression with Side Information Source Encoder Decoder Source Encoder Decoder [Wyner, Ziv, 1976] For mse distortion and Gaussian statistics, rate-distortion functions of the two systems are the same.

B. Girod: Distributed Video Coding 13 Practical Wyner-Ziv Encoder and Decoder Wyner-Ziv Decoder Quantizer Slepian- Wolf Encoder Wyner-Ziv Encoder Slepian- Wolf Decoder Minimum Distortion Reconstruction

B. Girod: Distributed Video Coding 14 Non-Connected Quantization Regions  Example: Non-connected intervals for scalar quantization  Decoder: Minimum mean-squared error reconstruction with side information x x

B. Girod: Distributed Video Coding 15 Outline  Foundations of distributed coding –Slepian-Wolf Theorem and practical Slepian-Wolf coding –Wyner-Ziv results and practical Wyner-Ziv coding  Low-complexity video encoding –Pixel-domain and transform-domain coding –Hash-based receiver motion estimation  Error-resilient video transmission –Systematic lossy joint source-channel coding –Improving the error-resiliency of MPEG by Wyner-Ziv coding

B. Girod: Distributed Video Coding 16 Interframe Video Coding Predictive Interframe Decoder Predictive Interframe Encoder X’ Side Information X

B. Girod: Distributed Video Coding 17 Video Coding with Low Complexity Wyner-Ziv Interframe Decoder Wyner-Ziv Intraframe Encoder X’ Side Information X [Witsenhausen, Wyner, 1978] [Puri, Ramchandran, Allerton 2002] [Aaron, Zhang, Girod, Asilomar 2002]

B. Girod: Distributed Video Coding 19 Low Complexity Encoding and Decoding

B. Girod: Distributed Video Coding 20 Pixel Domain Wyner-Ziv Coder Interframe Decoder Intraframe Encoder Reconstruction X’ Y Video frame X Scalar Quantizer Turbo Encoder Buffer Turbo Decoder Request bits Slepian-Wolf Codec Interpolation Key frames previous next [Aaron, Zhang, Girod, Asilomar 2002] [Aaron, Rane, Zhang, Girod, DCC 2003]

B. Girod: Distributed Video Coding 21 Decoder side information generated by motion- compensated interpolation PSNR 30.3 dB After Wyner-Ziv Decoding 16-level quantization – bpp 11 pixels in error PSNR 36.7 dB Pixel Domain Wyner-Ziv Coder

B. Girod: Distributed Video Coding 22 Pixel Domain Wyner-Ziv Coder Decoder side information generated by motion- compensated interpolation PSNR 24.8 dB After Wyner-Ziv Decoding 16-level quantization – 2.0 bpp 0 pixels in error PSNR 36.5 dB

B. Girod: Distributed Video Coding 23 Stanford Camera Array Courtesy Marc Levoy, Stanford Computer Graphics Lab

B. Girod: Distributed Video Coding 24 … WZ-ENC WZ-DEC WZ-ENC WZ-DEC … Geometry Reconstruction Rendering Wyner-Ziv CamerasConventional Cameras Distributed Compression Distributed Encoding Joint Decoding [Zhu, Aaron, Girod, 2003]

B. Girod: Distributed Video Coding 25 Light Field Compression Rate: 0.11 bpp PSNR 39.9 dB Rate: 0.11 bpp PSNR 37.4 dB Wyner-Ziv, Pixel-Domain JPEG-2000

B. Girod: Distributed Video Coding 26 DCT-Domain Wyner-Ziv Video Encoder For each low frequency coefficient band k level Quantizer DCT Turbo Encoder Extract bit- planes bit-plane 1 bit-plane 2 bit-plane M k … Input video frame Quantizer Entropy Coder Comparison Previous-frame quantized high freq coefficients Wyner-Ziv parity bits High frequency bits Low freq coeffs High freq coeffs

B. Girod: Distributed Video Coding 27 IDCT Wyner-Ziv Video Decoder with Motion Compensation Reconstruction DCT Entropy Decoder and Inverse Quantizer Side information Wyner-Ziv parity bits High frequency bits Turbo Decoder Motion- compensated Extrapolation Previous frame DCT Refined side information Extrapolation Refinement For each low frequency band Decoded frame Reconstructed high frequency coefficients

B. Girod: Distributed Video Coding 28 Rate-Distortion Performance - Salesman Every 8 th frame is a key frame Salesman QCIF sequence at 10fps 100 frames 6 dB 3 dB

B. Girod: Distributed Video Coding 29 Rate-Distortion Performance – Hall Monitor 8 dB 3 dB Every 8 th frame is a key frame Hall Monitor QCIF sequence at 10fps 100 frames

B. Girod: Distributed Video Coding 30 Rate-Distortion Performance – Foreman 2 dB 1.5 dB Every 8 th frame is a key frame Foreman QCIF sequence at 10fps 100 frames

DCT-based Intracoding 149 kbps PSNR Y =30.0 dB Wyner-Ziv DCT codec 152 kbps PSNR Y =35.6 dB GOP=8 Salesman at 10 fps

DCT-based Intracoding 156 kbps PSNR Y =30.2 dB Wyner-Ziv DCT codec 155 kbps PSNR Y =37.1 dB GOP=8 Hall Monitor at 10 fps

DCT-based Intracoding 290 kbps PSNR Y =34.4 dB Wyner-Ziv DCT codec 293 kbps PSNR Y =35.5 dB GOP=8 Foreman at 10 fps

B. Girod: Distributed Video Coding 34 Wyner-Ziv Residual Coding Wyner-Ziv Decoder Wyner-Ziv Encoder Side Information X’ n X’ n-1 Side Information - XnXn X’ n-1 Frame difference XnXn

B. Girod: Distributed Video Coding 35 Rate-Distortion Performance – Foreman Every 8 th frame is a key frame Foreman QCIF sequence at 30fps 100 frames

B. Girod: Distributed Video Coding 36 Rate-Distortion Performance – Foreman Every 8 th frame is a key frame Foreman QCIF sequence at 30fps 100 frames

B. Girod: Distributed Video Coding 37 Outline  Foundations of distributed coding –Slepian-Wolf Theorem and practical Slepian-Wolf coding –Wyner-Ziv results and practical Wyner-Ziv coding  Low-complexity video encoding –Pixel-domain and transform-domain coding –Hash-based receiver motion estimation  Error-resilient video transmission –Systematic lossy joint source-channel coding –Improving the error-resiliency of MPEG by Wyner-Ziv coding

B. Girod: Distributed Video Coding 38 Systematic Lossy Error Protection (SLEP) of Compressed Video Any Old Video Encoder Video Decoder with Error Concealment Error-Prone channel S S’ Wyner-Ziv Decoder A S*S* Wyner-Ziv Encoder A Wyner-Ziv Decoder B S ** Wyner-Ziv Encoder B Graceful degradation without a layered signal representation Analog channel (uncoded) [Aaron, Rane, Girod, ICIP 2003]

B. Girod: Distributed Video Coding 39 MPEG with Systematic Lossy Error Protection Channel Slepian-Wolf Encoder Wyner-Ziv Encoder ED T -1 Q -1 + MC S*S* MPEG Encoder main S Side information MPEG Encoder coarse T -1 q -1 ED + MC S’ R-S Decoder Reconstructed Frame at Encoder MPEG Encoder coarse R-S Encoder [Rane, Aaron, Girod, VCIP 2004] Parity only

B. Girod: Distributed Video Coding 40 Reed-Solomon Coding Across Slices 1 byte in slice filler byte parity byte RS code across slices Transmit parity slices only

B. Girod: Distributed Video Coding 41 Results: CIF Foreman Main Mbps FEC (n,k) = (40,36) FEC bitrate = 120 Kbps Total = 1.2 Mbps WZ 270 Kbps SLEP (n,k) = (52,36) WZ bitrate = 120 Kbps Total = 1.2 Mbps SLEP FEC FEC SLEP

B. Girod: Distributed Video Coding 42 MPEG with Systematic Lossy Error Protection Channel Slepian-Wolf Encoder Wyner-Ziv Encoder ED T -1 Q -1 + MC S*S* MPEG Encoder main S Side information MPEG Encoder coarse T -1 q -1 ED + MC S’ R-S Decoder Reconstructed Frame at Encoder MPEG Encoder coarse R-S Encoder [Rane, Aaron, Girod, VCIP 2004] Parity only

B. Girod: Distributed Video Coding 43 Quantized transformed Prediction error Coarse Quantizer Entropy Coding Q1Q1 Q -1 Quantization parameter (Q) MPEG2 Encoder Conventionally encoded stream Input Video Error-prone Channel Entropy Decoding MPEG2 Decoder T -1 + MC LEGACY BROADCASTING SYSTEM WYNER-ZIV ENCODER WYNER-ZIV DECODER RS Decoder Fallback to coarser version Decoded motion vecs Entropy Decoding Parity only RS Encoder Side Information (motion vectors, mode decisions) SLEP MPEG Codec with Simple Decoder Q1Q1 Entropy Coding Side Info (motion vecs, mode decisions)

B. Girod: Distributed Video Coding 44 Performance at Symbol Error Rate MPEG-2 video: 2 Mbps + FEC 222 Kbps (PSNR dB) MPEG-2 video: 2 Mbps + Wyner-Ziv 222 Kbps (PSNR dB)

B. Girod: Distributed Video Coding 45 Distributed Coding of Video: Why Should We Care?  Chance to reinvent compression from scratch –Entropy coding –Quantization –Signal transforms –Adaptive coding –Rate control –...  Enables new compression applications –Very low complexity encoders –Compression for networks of cameras –Error-resilient transmission of signal waveforms –Digitally enhanced analog transmission –Unequal error protection without layered coding –...

The End Further interest: B. Girod, A. Aaron, S. Rane, D. Rebollo-Monedero, "Distributed Video Coding," Proceedings of the IEEE, Special Issue on Video Coding and Delivery. January