1. FEC and RDO in SVC Thomas Wiegand 1

2. Outline Introduction SVC Bit-Stream Raptor Codes Layer-Aware FEC Simulation Results Linear Signal Model Description of the Algorithm Experimental Results 2

3. Introduction C. Hellge, T. Schierl, and T. Wiegand, “RECEIVER DRIVEN LAYERED MULTICAST WITH LAYER-AWARE FORWARD ERROR CORRECTION,” ICIP 2008. C. Hellge, T. Schierl, and T. Wiegand, “MOBILE TV USING SCALABLE VIDEO CODING AND LAYER-AWARE FORWARD ERROR CORRECTION,” ICME 2008. C. Hellge, T. Schierl, and T. Wiegand, “Multidimensional Layered Forward Error Correction using Rateless Codes,” ICC 2008. M. Winken, H. Schwarz, and T. Wiegand, “JOING RATE- DISTORTION OPTIMIZATION OF TRANSFORM COEFFICIENTS FOR SPATIAL SCALABLE VIDEO CODING USING SVC,” ICIP 2008. 3

4. SVC Bit-Stream Spatial-temporal-quality cube of SVC 4 http://vc.cs.nthu.edu.tw/home/paper/codfiles/kcyang/200710100050/Overview_of_the_Scalable_Video_Coding_Extension_of_the_H.264.ppt

5. RECEIVER DRIVEN LAYERED MULTICAST WITH LAYER-AWARE FORWARD ERROR CORRECTION C. Hellge, T. Schierl, and T. Wiegand ICIP 2008 5 C. Hellge, T. Schierl, and T. Wiegand, “MOBILE TV USING SCALABLE VIDEO CODING AND LAYER-AWARE FORWARD ERROR CORRECTION,” ICME 2008. C. Hellge, T. Schierl, and T. Wiegand, “Multidimensional Layered Forward Error Correction using Rateless Codes,” ICC 2008.

6. SVC Bit-Stream Equal FEC 6

7. Raptor Codes (1/2) Non-systematic Raptor codes 0 1 2 3 0 1 2 3 4 5 G p  0 1 2 3 4 5 G LT  0 1 2 3 4 5 6 7 == 7 precoding processLT coding process SSsPSs ESs

8. Raptor Codes (2/2) Systematic Raptor codes Construction of pre-code symbols – G LT, G p, and SSs. – G pSys = – Solving 00 k n-1 G p  G LT  == 00 G pSys  = 0 p-1 0 k-1 0 … … p-1 k-1 … …… … … G LT’ G LT’’ k  0 p-1 … p s k 0 k-1 … ? = 8 unknown GpGp I G LT’ s k ks p

9. Layer-Aware FEC (1/5) Example 1 Example 2 9

10. Layer-Aware FEC (2/5) Encoding process – Example 3 10

11. Layer-Aware FEC (3/5) Decoding process – Example 4 11

12. Layer-Aware FEC (4/5) G LayeredLT (m) = [G* LT0 | G* LT1 | … | G LTm ] 12 G LayeredLT (m)  = PSs 0 PSs 1 PSs 2 … PSs m ESs 0 ESs 1 ESs 2 … ESs m

13. Layer-Aware FEC (5/5) G pSysLayered (m) 13 0 SS 0 0 SS 1 0 G pSysLayered (m)  = PSs 0 PSs 1 PSs 2 … PSs m 0 ESs 0 0 ESs 1 … 0 ESs m

14. Simulation Results (1/2) QVGA (BL) and VGA (EL) resolution using SVC over a DVB-H channel. – JSVM 8.8 – GOP size = 16 Size of a transmission block = 186 bytes Mean error burst length = 100 TBs 14

15. Simulation Results (2/2) 15

16. Joint Rate-Distortion Optimization of Transform Coefficients For Spatial Scalable Video Coding Using SVC M. Winken, H. Schwarz, and T. Wiegand ICIP 2008 16

17. Hybrid Video Decoding 17 12 34 56 78 s5s6s7s8s5s6s7s8 s 2 s 3 ½ (s 2 +s 3 ) s x u5u6u7u8u5u6u7u8 s1s2s3s4s5s6s7s8s1s2s3s4s5s6s7s8 0000c5c6c7c80000c5c6c7c8 s1s2s3s4000sxs1s2s3s4000sx 0 0 0 0 0 1 0 0 0 0 0 0 0 0 ½ ½ 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 s1s2s3s4s5s6s7s8s1s2s3s4s5s6s7s8 0 0 0 0 ? ? ? ? = ++ =+ Motion compensated values Dequantized residual Decoded pixel values Motion vectors DeQuntized and iDCT parameters Motion compensationiQ and iDCTException

18. Linear Signal Model (1/6) Linear signal model for K inter frames – s = Ms + Tc + p s: A (K  W  H)  1 vector of decoded signal M: A (K  W  H)  (K  W  H) matrix of motion parameters T: A (K  W  H)  (K  W  H) matrix of inverse quantization and DCT parameters c: A (K  W  H)  1 vector of received transform coefficients p: A (K  W  H)  1 intra signal or motion parameters outside s s11…s1WH…sK1…sKWHs11…s1WH…sK1…sKWH s11…s1WH…sK1…sKWHs11…s1WH…sK1…sKWH c11…c1WH…cK1…cKWHc11…c1WH…cK1…cKWH p11…p1WH…pK1…pKWHp11…p1WH…pK1…pKWH =++ 1KK+1 18 WHWH WHWHWHWH WHWH

19. Linear Signal Model (2/6) Optimal transform coefficients selection – Decoder receives MVs (M) and quantized transform coefficients (c). – fixed motion parameters (M), quantization parameters (T), and intra predictions (p). Rate and distortion are mainly controlled by c. – c’ = argmin c {D(c) + R(c)} subject to s = Ms + Tc + p D(c) = ||x - s|| 2 2, R(c) = ||c|| 1 19

20. Linear Signal Model (3/6) Optimal transform coefficients selection – Problem: MVs cannot be determined before the transform coefficients are selected (trade-off) – Solution: 20 s11…s1WHs21…s2WHs31…s3WH…sK1…sKWHs11…s1WHs21…s2WHs31…s3WH…sK1…sKWH c11…c1WHc21…c2WHc31…c3WH…cK1…cKWHc11…c1WHc21…c2WHc31…c3WH…cK1…cKWH p11…p1WHp21…p2WHp31…p3WH…pK1…pKWHp11…p1WHp21…p2WHp31…p3WH…pK1…pKWH =++ s11…s1WHs21…s2WHs31…s3WH…sK1…sKWHs11…s1WHs21…s2WHs31…s3WH…sK1…sKWH fixed initial

21. Linear Signal Model (4/6) Optimal transform coefficients – Problem size: K  W  H Sliding window approach (Reduce problem size) – s = M s + T c + p 21 window size step size

22. Linear Signal Model (5/6) Extension for spatial scalability – s 0 = M 0 s 0 + T 0 c 0 + p 0 – s 1 = M 1 s 1 + T 1 c 1 + p 1 + Bs 0 + RT 0 c 0 H.264/AVC MCP & Intra-prediction Hierarchical MCP & Intra-prediction Base layer coding texture motion texture motion Inter-layer prediction Intra Motion Residual Spatial decimation Multiplex Scalable bit-stream H.264/AVC compatible coder H.264/AVC compatible base layer bit-stream 22 Inter-layer motion prediction Inter-layer residual prediction

23. Linear Signal Model (6/6) Optimal transform coefficients in spatial scalability – c 0 ’ D 0 (c 0 ) + 0 R(c 0 ) c 1 ’ D 1 (c 0,c 1 ) + 1 (R(c 0 )+R(c 1 )) subject to s 0 = M 0 s 0 + T 0 c 0 + p 0 s 1 = M 1 s 1 + T 1 c 1 + p 1 + Bs 0 + RT 0 c 0  c 0 ’ (1-w)  (D 0 (c 0 ) + 0 R(c 0 )) + c 1 ’ w(D 1 (c 0,c 1 ) + 1 (R(c 0 )+R(c 1 ))) where  = (W 1 H 1 )/(W 0 H 0 ) = argmin c0’c1’ 23 = argmin c0’c1’

24. Description of the Algorithm Determine M 0, T 0, M 1, T 1, B, p 0, R, and p 1 by encoding the first K pictures using SVC reference encoder model. Solve optimization to determine c 0 of the base layer. Based on new c 0, determine B and R again. Solve optimization problem for only the enhancement layer. 24

25. Experimental Results (1/2) JSVM 9.9 – IPPP – QCIF (base layer) and CIF (enhancement layer) – CABAC – QP difference: 3 – Sliding windows size: 5  5 for base layer and 10  10 for enhancement layer 25

26. Experimental Results (2/2) 26