1. 1 Bi-level Video: Video Communication at Very Low Bit Rates Jiang Li; Gang Chen; Jizheng Xu; Yong Wang; Hanning Zhou; Keman Yu; King To Ng; Heung-Yeung Shum

2. 2 OUTLINE INTRODUCTION (Wireless Video) Scalable Video Coding Introduction (Bi-level Video) Approaches Experimental Result Conclusion

3. 3 INTRODUCTION Qos requirements for delivery of real-time video  Bandwidth: minimum bandwidth requirements (e.g. 28 kb/s)  Delay constraints (e.g. 1 s)  Upper limits on bit error rate (e.g. 1%)

4. 4 INTRODUCTION (cont.) Wireless Channel  Unreliability Noise, Multipath and shadowing makes BER high  Bandwidth Fluctuation Mobile terminal moves between different networks Multipath fading, co-channel interference, noise disturbances…  Heterogeneity Receivers may be different in terms of latency requirements, visual quality requirements, processing capabilities, power limitations, and bandwidth limitations…

5. 5 INTRODUCTION (cont.) Heterogeneity (video conferencing) Sender Receiver link1 link2 Unicast video distribution using multiple point-to- point connections

6. 6 INTRODUCTION (cont.) Sender Receiver Multicast video distribution using point- to-multipoint transmission Lack of flexibility in multicast: Receiver may be different in terms of latency requirements, visual quality requirements, processing capabilities, power limitations, and bandwidth limitations… link1link2

7. 7 INTRODUCTION (cont.) Solutions  Scalable Video Coding  Network-Aware Adaptation of End Systems Knowing the current status of network resources (e.g. bit error condition, available bandwidth) Adapting video streams based on network status  Adaptive Qos Support from Networks Adapting video streams during periods of QoS fluctuations and handoffs

8. 8 Scalable video coding basic Enhancement 1 Enhancement 2

9. 9 Scalable video coding (cont.) An example for multicast basic Enhancement 1 Enhancement 2

10. 10 Scalable video coding (cont.) Scalable video coding mechanisms  SNR Scalability Degraded quality  Spatial Scalability Smaller image size  Temporal Scalability Lower frame rate

11. 11 Introduction for Bi-level Video Problems on Wireless Networks at Vary Low Bit Rates (mobile phone, palm-size PC …)  Low bandwidth (e.g. 9.6k bps)  Weak computational power  Short battery lifetime  Limited display capability (only support for several colors or black/white display)

12. 12 Introduction for Bi-level Video (cont.) Using MPEG1/2/4 or H.261/263  Not smooth  Look like a collection of color blocks (only basic colors are preserved)

13. 13 Introduction for Bi-level Video (cont.) Outline features of scenes are more important than basic colors of blocks  Bi-level image A gray-scale imageBi-level image

14. 14 Approaches We generate a bi-level image sequence from a gray-scale image sequence by thresholding Key problem  Noises and flickers in the bi-level image sequence must be filtered out ( ∵ consuming too many bits)  How to convert a bi-level image sequence and process it so that it is easy to be compressed and still keeps acceptable perception quality

15. 15 Approaches (cont.) step1step2 step3step4

16. 16 Approaches (cont.) Static Region Detection and Duplication  Detecting static regions in the current frame and duplicate their pixels from the previous fame  Filtering noise and flickers out  Bit-rate control

17. 17 Approaches (cont.) Dissimilarity threshold  The threshold of the difference between corresponding pixels in two successive frames  The higher the dissimilarity threshold is, the more pixels are viewed as being similar to corresponding pixels in the previous frame, and the lower bit-rate the generated bit stream is 25 5 Frame j-1Frame j Thresh = 2 Thresh = 6 2 Frame j

18. 18 Approaches (cont.) Calculating the difference between the j & j-1 frames  Laplacian ( 拉普拉辛 ) of a pixel The second derivative of intensity at that pixel L j (x,y) =

19. 19

20. 20 Approaches (cont.) Calculating the difference between the j & j-1 frames  If the Laplacian of a region remains unchanged, the region is most likely static  Sum of absolute differences (SAD) of Laplacian of pixels in a square surrounding the target pixel (x,y)

21. 21 Approaches (cont.) Calculating the difference between the j & j-1 frames L j (x,y) – L j-1 (x,y) SAD j (x,y)

22. 22 Approaches (cont.) Calculating the difference between the j & j-1 frames  If SAD j (x,y) ≤ t d (dissimilarity threshold), the pixel is marked as static Static (white) and motion (black) regions To avoid misidentified static regions static motion goback

23. 23 Approaches (cont.) Adaptive Thresholding  Used to convert a gray-scale image to a bi-level image  Using Ridler’s Iterative Selection method  e.g. sequence {1,3,5,6,7,9,12,13} t 1 = (1+3+5+6+7+9+12+13)/8 = 7 t b = (1+3+5+6)/4 = 3.75t o = (9+12+13)/3 = 11.33 t 2 = (3.75+11.33)/2 = 7.54 t b = (1+3+5+6+7)/4 = 5.5t o = (9+12+13)/3 = 11.33 t 3 = (5.5+11.33)/2 = 8.415 t b = (1+3+5+7)/4 = 5.5t o = (9+12+13)/3 = 11.33 t 4 = (5.5+11.33)/2 = 8.415 ∴ t = 8.415 Take threshold = t + t c goback

24. 24 Approaches (cont.) Adaptive Context-based Arithmetic Encoding  Confidence level The difference between the gray-scale value of the pixel and the threshold Pixels with their gray-scale value near the threshold could be determined as either black or white For those pixels with their absolute values of confidence level less than the half-width of threshold band, the bi- level values of the pixels are assigned according to the indexed probability in the probability tablethreshold band  Coding the whole frame rather than lots of blocks goback

25. 25 Approaches (cont.) Threshold band  For pixels whose gray-scale values are inside the band, their bi-level values could be modified according to the predicted values in the subsequent adaptive context-based arithmetic encoding Dissimilarity Threshold  Used to detect static region Adaptive Thresholding  Used to generate bi-level images

26. 26 Approaches (cont.) Rate Control Buffer size B = I max + 4r/n I max : the maximum number of bits per frame that is allowed to be sent to the buffer r : maximum video bit-rate n : specified frame rate I max +4r/n I p time = t p p I frame rate = n tn-1 p-frames Assign every group rt bits b i bits b p =(rt-bi)/(tn-1) I p p p p p I p p p b p bits p p

27. 27 Approaches (cont.) Rate Control Rate control scale factor f Dissimilarity threshold t d Half-width of threshold band 93.05 8 4 72.54 6 3 52.03 4 2 31.52 2 1 11.01 0 0 15% Overflow, increase f to 9 Increase f by 1 Decrease f by 1

28. 28 Experimental Results ( MPEG4 video clips ) (ordinary clips captured from real scenes using PC digital cameras) (a) Akiyo Little head motion (e) Wang Little head motion (b) Grandma Large head motion (c) Salesman Complex background (d) Chen Large head motion

29. 29 Experimental Results (cont.) Complex background Large head motion

30. 30 Experimental Results (cont.) H.263+ VS. bi-level video

31. 31 Experimental Results (cont.) (a) Akiyo(c) Salesman Complex but stable background H.263+ consumes few bits since the difference of two successive frames is almost negligible

32. 32 Conclusion An I-frame in bi-level video is much smaller than in conventional DCT-based videos  short start-up time for streaming video  Can insert more I-frame Transmission errors quickly be recovered  Provide a smooth perception of motion even in low frame conditions

33. 33 Conclusion (cont.) Clearer shape, smoother motion, shorter initial latency, cheaper computation cost  The number of each pixel is reduced to 1  The coding need not estimate and store motion vectors, thus reduces computational cost and coding bit-rate.  Static region detection reduces flicker effects and therefore improves coding efficiency The coding system works well in handheld PCs, palm-size PCs, mobile phones that possess only a small display screen, very limited computation capability and transmission bandwidth

34. 34 Conclusion (cont.) Microsoft portrait  A very low bit-rate video conferencing software  http://research.microsoft.com/~jiangli/portrait/ http://research.microsoft.com/~jiangli/portrait/