1. Efficient MPEG Compressed Video Analysis Using Macroblock Type Information Soo-Chang Pei, Yu-Zuong Chou IEEE TRANSACTIONS ON MULTIMEDIA, DECEMBER,1999

2. Introduction We develop a novel method for video analysis using the macroblock(MB) type information of MPEG compressed video bitstreams. Only a simple analysis on MB types of frames is needed to achieve efficient scene change, flashlight, and caption detection. The advantages of this novel approach are its direct extraction from the MPEG bitstreams after VLC decoding, very low complexity analysis, frame-based detection accuracy and high sensitivity.

3. Proposed Method SGOP(subgroup of pictures) SGOP 1 are denoted as P f B f B r P r SGOP 2 are denoted as P B f B r i SGOP 3 are denoted as i B f B r P

4. Pattern of MB Types of SGOP ’ s with Abrupt Scene Changes SCPI (Scene change at a P frame or an I frame) Most MB ’ s in the two B frames are forward motion compensated (F mode) A significant number of MB ’ s in P r are Intra-coded (I mode) x

5. Pattern of MB Types of SGOP ’ s with Abrupt Scene Changes(cont.) SCFB (Scene change at front B frame) Most MB ’ s in the two B frames are backward motion compensated (B mode) There are many I mode MB ’ s existing in P r

6. Pattern of MB Types of SGOP ’ s with Abrupt Scene Changes(cont.) SCRB (Scene change at rear B frame) Most MB ’ s in B f will be F mode predicted to P f, and most MB ’ s in the B r will be B mode predicted to P r.

7. We define the MB in a specific position (x,y) of each frame of a SGOP to form a MB group(MBGxy). If the pattern of the MB types in a MBGxy follows the above pattern of SCPI,SCFB or SCRB, we can say there is a changing MB in the position (x,y) laying at P f, B f, or B r.

8. Example of gradual scene changes

9. Pattern of MB Types in SGOP ’ s with Gradual Scene Changes Dissolve scene S 0 to scene S n in the transition sequence, length n+1 If S 0 =0,fade-in. If S n =0,fade-out. The luminance of B frames in SGOP 1 (PBBP) operated by the dissolve effect can be approximately written as

10. Pattern of MB Types in SGOP ’ s with Gradual Scene Changes(cont.) When bidirectional motion compensation is applied to B frames, the compensated frames of three compensation modes(F, B, FB) can be written It is found that the variance of FB compensated error is smaller than that of F or B compensated error. B frames in dissolve video sequence are inclined to be interpolative(FB mode) motion compensated.

12. SFPI(starting frame lies at P r or I frame) A significant number of MS ’ s in P r are I mode because the change of content. MB ’ s in B frames are inclined to be F mode The pattern of MB types of SFPI is very similar to the pattern of SCPI SFFB (starting frame lies at B f ) Most MB ’ s in both B frames are FB mode since the frames of this SGOP form a Complete Dissolve SGOP. SFRB (starting frame lies at B r ) the MB ’ s of B f are inclined to be F mode and those of B r are inclined to be FB mode. x

13. A significant number of MB ’ s of P r in a dissolve video sequence are I mode in both SFFB and SFRB. The patterns of the MB types of the SGOP including ending frames(EFPI,EFFB,EFRB) are similar and symmetric to that including starting frames. There are nine combinations.

14. x

15. Video sequence involving camera motion Zooming operation is slow for the purpose of comfortable watching, and can be easily tracked as a special kind of movement and will not greatly affect the MB types of frames. Slow panning operations would be judged as overall movement of the frame and the MB type information would not show some specific characteristic. The speeding of fast panning exceeds the tracking ability of motion compensation,so a significant number of MB ’ s in the P frame are I mode. The detection process may confuse fast panning operation with dissolve operation because of the similarity.

16. x

17. Example of flashlight in news experiment sequence

18. Pattern of MB Types in SGOP ’ s with Flashlights

19. Experimental Result There are 24 abrupt scene changes, five gradual scene change transitions,f our shots involving camera motion(two fast panning and two zooming sequences) in the test sequence. Each frame of the sequence is in SIF (352*240) format, which yields 330 (22*15) MB ’ s in a frame. The frame structure is IBBPBBPBBPBBPBB (M=3,N=15). Total bit rate is 1.5 Mbps

20. Abrupt scene change detection

21. Locations of changing MB ’ s in the test sequence(frame 1 to 100)

22. Experimental results of abrupt scene change detection ( T mb = 280)

24. Gradual scene change detection

25. Remaining P frames with significant I mode MB ’ s

26. Number of FB mode MB ’ s in B frames

28. Flashlight detection

29. Flashlight detection using news experiment sequence

30. Caption detection

31. The locations of changing MB’s in the caption sequence

32. Abrupt scene change detection Caption detection with eliminating the abrupt scene changes

33. Experiments Using Real Video Sequence of Movies and News

34. Discussion Influence of Bit Rates

36. Taking the forward motion compensated MB for instance, the attribute of MB ’ s change from “ Not coded ” or “ Coded, Larger Quantizer Scale ” to “ Coded, Default ” to obtain better residue data reconstruction instead of changing FB to Intra mode. The change of the bit rates just very slightly affects the occurrences of I,B,F,FB modes is found. The detection algorithm exploiting the MB types is not affected by bit rates.

37. Modification of Proposed Method Corresponding to Different GOP Structure

39. Advantage of Proposed Method The MB type information can be directly extracted from the bit streams of compressed video after VLC decoding. Detecting process is quite simple as compared to current DCT methods that require more operations. Its detection accuracy is on the frame scale, and we can precisely indicate which frame the scene change occurs.

40. Conclusion We have developed a novel video analysis method using MB type information,and satisfactory detection precision and speed is obtained. The method using MB type information benefits from easy data extraction from the bitstream, very simple analysis, frame-based accuracy and high sensitivity to avoid miss detection.