1. Seamless Switching of Scalable Video Bitstreams for Efficient Streaming Xiaoyan Sun, Feng Wu, Shipeng Li, Wen, Gao, and Ya-Qin Zhang

2. Outline Introduction Seamless Switching Scheme Among Scalable Bitstreams Switching Down Between Two Bitstreams Switching Up Between Two Bitstreams Experimental Results Discussions

3. Introduction Why are scalable bitstreams needed? Switching between non-scalable bitstreams –Drifting error –Key frames –Large storage requirements Multiple non-scalable bitstreams Key frames Non-scalable bitstream1 Non-scalable bitstream2 drifting error Non-scalable bitstream1 Non-scalable bitstream2 key frame

4. Introduction SP-frames scheme (proposed in H.264) –Extra bitstreams are used. –The size of S12 is similar to that of I-frame. –When switching down, S12 may deteriorate the networks. – http://vc.cs.nthu.edu.tw/ezLMS/show.php?id=249 http://vc.cs.nthu.edu.tw/ezLMS/show.php?id=249 P S2 P P S1 P S12 bitstream1 bitstream2 SP frame extra bitstream

5. Introduction Why are multiple scalable bitstreams needed? Single scalable bitstream –FGS –Low coding efficiency in FGS bitstreams. Motion prediction is based on the lowest quality base layer. –Using multiple scalable bitstreams, coding efficiency is higher. Base layer Enhancement layer 1 Enhancement layer 2 Enhancement layer 3 … Base layer Enhancement layer 1 Enhancement layer 2 Enhancement layer 3 … Frame nFrame n+1

6. Seamless Switching Scheme Among Scalable Bitstreams Seamless switching (Definition) –The quality in each scalable bitstream is smooth. –The switching among scalable bitstreams is drifting-free. –Immediately switching from the current scalable bitstream to one operated at lower rates without any delay.

7. Seamless Switching Scheme Among Scalable Bitstreams Scalable Bitstream 1 Scalable Bitstream 2 Scalable Bitstream 1 Scalable Bitstream 3 Bandwidth Time Scalable Bitstream 1 Scalable Bitstream 2 Scalable Bitstream 3

8. Seamless Switching Scheme Among Scalable Bitstreams switching SF switching frame (switching-up point) High-bit-rate scalable bitstream Low-bit-rate scalable bitstream

9. Switching Down Between Two Bitstreams MVs are estimated in SB-H and are applied to both SB-H and SB-L. The quantization information of SB-L is coded in SB-H bitstream. –Overhead bits: 3*MB_number+5 Instead of original frames, base layer frames in SB-H are used to encode base layer frames in SB-L. First QPRange of difference ☆ Switching down at everywhere and with no overhead

10. Switching Down Between Two Bitstreams SF Decoder VLDQ -1 IDCT MC DCTQ L -1 QLQL IDCT ref SB-LrefBase SB-H Base layer SB-L Base layer Video MVs

11. Switching Down Between Two Bitstreams Quality loss of SB-L base layer –The reconstructed frames from SB-H base layer instead of the original video are used as the input of the SB-L base layer. –The same set of MVs for SB-H base layer is used to encode the SB-L base layer.

12. Switching Up Between Two Bitstreams n-1nn+1 Switching point SB-L Base layer SB-H Base layer + = Extra bitstream ☆ Considerable amount of overhead bits

13. Switching Up Between Two Bitstreams SF Encoder IDCT Q s -1 DCT QsQs BPVLC DCTQsQs SB-H Base layer SB-L Base layer Extra bitstream SF frame n-1n+1 n ☆ n ’ is used instead of n to avoid drifting error = S B - L B a s e + E x t r a b i t s t r e a m n’n’ n

14. Switching Up Between Two Bitstreams SF Decoder VLDQ -1 IDCT MC refBase SB-H Base layer SB-L Base layer Video MVs DCTQ s -1 QsQs IDCT BPVLD Extra bitstream

15. Switching Up Between Two Bitstreams The SF frame is simpler than the SP frame. The SF frame gets better quality than the SP frame.

16. Experimental Results SF scheme is applied on MBPFGS –MBPFGS is modified from PFGS Frame base -> Macroblock base – –10fps An I-frame is inserted every 1 ’ s in a non- scalable bitstream An SF frame is inserted every 1 ’ s in a SF bitstream 32kbps 64kbps 80kbps 112kbps http://vc.cs.nthu.edu.tw/home/paper/codfiles/hschen/200305141532/pfgs.PPT MBFGS-H bitstream MBFGS-L bitstream

17. Experimental Results Comparisons of extra bitstream – – Average PSNR loss caused by SF frames < 0.1 dB SequenceSF (Q s =3)Lossless News32959.11147878.1 Foreman34918.33173569.3 Coastguard39824.78158157.7 (Bits)

18. Experimental Results Rate-distortion (static) –Non-scalable bitstream (JVT H.26L codec) Two bitstream with different quality Switching at I-frame –FGS and MPFGS Only one bitstream Non-scalable FGS MPFGS SF Inaccurate MVs and input video

19. Experimental Results PSNR comparisons (dynamic channel) –Foreman –Coastguard 72 kbps 152 kbps 72 kbps 152 kbps

20. Discussions The SF scheme can be extended to the case with multiple scalable bitstreams. –(N-1) extra bitstreams are required. N*(N-1) extra bitstreams are required in SP frames. –High complexity of the decoder in decoding the high-bit- rate scalable bitstream. Three times of the MPEG-4 decoder. Complexity scalability. –Large size of the overhead bits inside the high bit-rate bitstream. The extra bitstream for switching up is large. –if the bitstreams switch up and down once for every second, the extra bitstream will cost 30-40 kbps. –An intelligent server should be able to control switching times.