FEC and RDO in SVC Thomas Wiegand 1

Outline Introduction SVC Bit-Stream Raptor Codes Layer-Aware FEC Simulation Results Linear Signal Model Description of the Algorithm Experimental Results 2

Introduction C. Hellge, T. Schierl, and T. Wiegand, “RECEIVER DRIVEN LAYERED MULTICAST WITH LAYER-AWARE FORWARD ERROR CORRECTION,” ICIP C. Hellge, T. Schierl, and T. Wiegand, “MOBILE TV USING SCALABLE VIDEO CODING AND LAYER-AWARE FORWARD ERROR CORRECTION,” ICME C. Hellge, T. Schierl, and T. Wiegand, “Multidimensional Layered Forward Error Correction using Rateless Codes,” ICC M. Winken, H. Schwarz, and T. Wiegand, “JOING RATE- DISTORTION OPTIMIZATION OF TRANSFORM COEFFICIENTS FOR SPATIAL SCALABLE VIDEO CODING USING SVC,” ICIP

SVC Bit-Stream Spatial-temporal-quality cube of SVC 4

RECEIVER DRIVEN LAYERED MULTICAST WITH LAYER-AWARE FORWARD ERROR CORRECTION C. Hellge, T. Schierl, and T. Wiegand ICIP C. Hellge, T. Schierl, and T. Wiegand, “MOBILE TV USING SCALABLE VIDEO CODING AND LAYER-AWARE FORWARD ERROR CORRECTION,” ICME C. Hellge, T. Schierl, and T. Wiegand, “Multidimensional Layered Forward Error Correction using Rateless Codes,” ICC 2008.

SVC Bit-Stream Equal FEC 6

Raptor Codes (1/2) Non-systematic Raptor codes G p  G LT  == 7 precoding processLT coding process SSsPSs ESs

Raptor Codes (2/2) Systematic Raptor codes Construction of pre-code symbols – G LT, G p, and SSs. – G pSys = – Solving 00 k n-1 G p  G LT  == 00 G pSys  = 0 p-1 0 k-1 0 … … p-1 k-1 … …… … … G LT’ G LT’’ k  0 p-1 … p s k 0 k-1 … ? = 8 unknown GpGp I G LT’ s k ks p

Layer-Aware FEC (1/5) Example 1 Example 2 9

Layer-Aware FEC (2/5) Encoding process – Example 3 10

Layer-Aware FEC (3/5) Decoding process – Example 4 11

Layer-Aware FEC (4/5) G LayeredLT (m) = [G* LT0 | G* LT1 | … | G LTm ] 12 G LayeredLT (m)  = PSs 0 PSs 1 PSs 2 … PSs m ESs 0 ESs 1 ESs 2 … ESs m

Layer-Aware FEC (5/5) G pSysLayered (m) 13 0 SS 0 0 SS 1 0 G pSysLayered (m)  = PSs 0 PSs 1 PSs 2 … PSs m 0 ESs 0 0 ESs 1 … 0 ESs m

Simulation Results (1/2) QVGA (BL) and VGA (EL) resolution using SVC over a DVB-H channel. – JSVM 8.8 – GOP size = 16 Size of a transmission block = 186 bytes Mean error burst length = 100 TBs 14

Simulation Results (2/2) 15

Joint Rate-Distortion Optimization of Transform Coefficients For Spatial Scalable Video Coding Using SVC M. Winken, H. Schwarz, and T. Wiegand ICIP

Hybrid Video Decoding s5s6s7s8s5s6s7s8 s 2 s 3 ½ (s 2 +s 3 ) s x u5u6u7u8u5u6u7u8 s1s2s3s4s5s6s7s8s1s2s3s4s5s6s7s8 0000c5c6c7c80000c5c6c7c8 s1s2s3s4000sxs1s2s3s4000sx ½ ½ s1s2s3s4s5s6s7s8s1s2s3s4s5s6s7s ? ? ? ? = ++ =+ Motion compensated values Dequantized residual Decoded pixel values Motion vectors DeQuntized and iDCT parameters Motion compensationiQ and iDCTException

Linear Signal Model (1/6) Linear signal model for K inter frames – s = Ms + Tc + p s: A (K  W  H)  1 vector of decoded signal M: A (K  W  H)  (K  W  H) matrix of motion parameters T: A (K  W  H)  (K  W  H) matrix of inverse quantization and DCT parameters c: A (K  W  H)  1 vector of received transform coefficients p: A (K  W  H)  1 intra signal or motion parameters outside s s11…s1WH…sK1…sKWHs11…s1WH…sK1…sKWH s11…s1WH…sK1…sKWHs11…s1WH…sK1…sKWH c11…c1WH…cK1…cKWHc11…c1WH…cK1…cKWH p11…p1WH…pK1…pKWHp11…p1WH…pK1…pKWH =++ 1KK+1 18 WHWH WHWHWHWH WHWH

Linear Signal Model (2/6) Optimal transform coefficients selection – Decoder receives MVs (M) and quantized transform coefficients (c). – fixed motion parameters (M), quantization parameters (T), and intra predictions (p). Rate and distortion are mainly controlled by c. – c’ = argmin c {D(c) + R(c)} subject to s = Ms + Tc + p D(c) = ||x - s|| 2 2, R(c) = ||c|| 1 19

Linear Signal Model (3/6) Optimal transform coefficients selection – Problem: MVs cannot be determined before the transform coefficients are selected (trade-off) – Solution: 20 s11…s1WHs21…s2WHs31…s3WH…sK1…sKWHs11…s1WHs21…s2WHs31…s3WH…sK1…sKWH c11…c1WHc21…c2WHc31…c3WH…cK1…cKWHc11…c1WHc21…c2WHc31…c3WH…cK1…cKWH p11…p1WHp21…p2WHp31…p3WH…pK1…pKWHp11…p1WHp21…p2WHp31…p3WH…pK1…pKWH =++ s11…s1WHs21…s2WHs31…s3WH…sK1…sKWHs11…s1WHs21…s2WHs31…s3WH…sK1…sKWH fixed initial

Linear Signal Model (4/6) Optimal transform coefficients – Problem size: K  W  H Sliding window approach (Reduce problem size) – s = M s + T c + p 21 window size step size

Linear Signal Model (5/6) Extension for spatial scalability – s 0 = M 0 s 0 + T 0 c 0 + p 0 – s 1 = M 1 s 1 + T 1 c 1 + p 1 + Bs 0 + RT 0 c 0 H.264/AVC MCP & Intra-prediction Hierarchical MCP & Intra-prediction Base layer coding texture motion texture motion Inter-layer prediction Intra Motion Residual Spatial decimation Multiplex Scalable bit-stream H.264/AVC compatible coder H.264/AVC compatible base layer bit-stream 22 Inter-layer motion prediction Inter-layer residual prediction

Linear Signal Model (6/6) Optimal transform coefficients in spatial scalability – c 0 ’ D 0 (c 0 ) + 0 R(c 0 ) c 1 ’ D 1 (c 0,c 1 ) + 1 (R(c 0 )+R(c 1 )) subject to s 0 = M 0 s 0 + T 0 c 0 + p 0 s 1 = M 1 s 1 + T 1 c 1 + p 1 + Bs 0 + RT 0 c 0  c 0 ’ (1-w)  (D 0 (c 0 ) + 0 R(c 0 )) + c 1 ’ w(D 1 (c 0,c 1 ) + 1 (R(c 0 )+R(c 1 ))) where  = (W 1 H 1 )/(W 0 H 0 ) = argmin c0’c1’ 23 = argmin c0’c1’

Description of the Algorithm Determine M 0, T 0, M 1, T 1, B, p 0, R, and p 1 by encoding the first K pictures using SVC reference encoder model. Solve optimization to determine c 0 of the base layer. Based on new c 0, determine B and R again. Solve optimization problem for only the enhancement layer. 24

Experimental Results (1/2) JSVM 9.9 – IPPP – QCIF (base layer) and CIF (enhancement layer) – CABAC – QP difference: 3 – Sliding windows size: 5  5 for base layer and 10  10 for enhancement layer 25

Experimental Results (2/2) 26