1. Automatic Video Shot Detection from MPEG Bit Stream
Jianping Fan
Department of Computer Science
University of North Carolina at Charlotte
Charlotte, NC 28223

2. Why we need video shots?
Indexing
a. Text Retrieval: Keyword Extraction
Reverse File Indexing
Document Storage

3. Why we need video shots?
Indexing
b. Database Query: Entity Extraction
B-Tree Indexing
Database Storage

4. Why we need video shots?
Indexing
c. Image Retrieval: Object Indexing
Ontology Indexing
Database Storage

5. Why we need video shots?
MPEG Video Sequence
Video shot
frames

6. Why we need video shots?
Story Unit in News Video
Each story unit may consist of multiple connected video shots!

7. ???
Why we need video shots?
Indexing
Shot Indexing
Video Shots in Storage

8. Video shot =?= keyword in video?
???
Shot is used as basic unit for video indexing!
Storage Database
Networks
Query Processing
Server

9. 1. Why we need shot detection?
Potential Search on Video Shots:
Give me some videos which consist
of similar video shots!
b. Give me all the videos which are
related to this video shot!

10. 2. Content Structure of Video Sequence
Video shot
frames
Shot-Based Video Content Interpretation!

11. 3. Coding Structure in MPEG Video
Video sequence
MPEG encoding
GOP ?
shot

12. Interesting Questions
If you are leader of MPEG around 1994, how can you
design new MPEG standards which can be used for
video indexing?

13. New MPEG Standard with Video Shot
Video sequence
Video shot
GOP
GOP
GOP
GOP is Video Shot!
frames

14. 4. What’s mean of video shot?
Why We Have Shots:
(a) Human Editing;
(b) Content Change & Camera Motion
Shot Types:
(a) Scene Cuts;
(b) Fade in & Fade out;
© Dissolves & Wipes

15. 4. What’s mean of video shot?
Scene Cuts:
Sudden change of video content or focus
Shot 1
time

16. 4. What’s mean of video shot?
b. Fade: gradual transition between a screen and a
constant image (fade out).

17. 4. What’s mean of video shot?
c. Dissolve: gradual transition from one screen to
another, the first screen fade & the second one fade out.

18. 4. What’s mean of video shot?
Dissolve

19. 4. What’s mean of video shot?
d. Wipe: a line moves across the screen & new
appears behind the line

20. 4. What’s mean of video shot?
Wipe

21. Conclusion: Why we have video shots?
Video Sequence
Video shot
frames
Big change between two continuous video frames!

22. How can we detect video shots?
Why we can define them as cuts, wipe, ….? Because they are changed between neighboring video frames!!!!
How can we measure the changes or similarities between the neighboring video frames?
How can make decision on whether they are change or not?

23. 5. How to detect video shot?
Major Components for Shot Detection:
Visual Representation of Video Frames: Color, Texture…
Difference Calculation for Neighboring Video Frames
Threshold for Decision Making: How Large is Enough?

24. 5. How to detect video shot?
How to measure statistical property of video frames?
Color Histogram

25. 5. How to detect video shot?
How to measure statistical property of video frames?
Color Histogram

26. 5. How to detect video shot?
How to measure statistical property of video frames?
Texture Histogram

27. 5. How to detect video shot?

28. Color Histogram Difference
5. How to detect video shot?
How we can do this more efficient on MPEG videos?
MPEG videos
a. Easy but not smart
Automatic
Threshold
Determination
MPEG Decoder
Color Histogram Difference
Scene Cut Frame Output

29. Scene Cut

30. 5. How to detect video shot?
Effects on color histogram difference by scene cuts:

31. 5. How to detect video shot?
How to measure the change of video content?
Color Histogram Difference

32. 5. How to detect video shot?
How we can select the threshold automatically?
J. Fan, D.K.Y. Yau, W.G. Aref, A. Rezgui, ``Adaptive motion-
compensated video coding scheme towards content-based
bit rate allocation”, Journal of Electronic Imaging, vol.9,
no.4, pp , 2000.
This algorithm can adapt the thresholds to different videos.
b. J. Fan, et al., ``ClassView: Hierarchical video shot classification
and retrieval”, IEEE Trans. on Multimedia, 2004.
This algorithm can adapt thresholds to different video shots.

33. 5. How to detect video shot?
Relationships among continuous frames can defined as:
scene cuts versus non-scene cuts 1
non-scene cut
scene cut
probability T 1

34. 5. How to detect video shot?
How to obtain T automatically?
Entropy for non-scene cut frames:

35. Interesting Questions
What will happen on MPEG bit stream if the current video
frame is a shot boundary? I B P
How can we use such properties for shot detection?

36. 5. How to detect video shot?
b. Complex but smart approach
If scene cuts happen on I frames in MPEG Video?
MPEG encoding
GOP
How to calculate color histograms of I frames efficiently?
Only decode DC coefficient!!!

37. 5. How to detect video shot?
b. Complex but smart approach
What will happen on MPEG bit stream
if there is a shot boundary?
4Mb/s
1.5Mb/s
high
Picture Quality
low

38. If content change happens on P frames in MPEG video?
Most macroblocks can not find their correspondences!!
Only decode the block coding types in P frames!!!
Reference frame( I or P)
Current P frame

39. If content change happens on B frames in MPEG video?
Only decode the coding types in B frames!!!!
Most macroblocks in B current frame can only be predicted by backward way!!
Future Reference
Previous Reference
Current B frame

40. 6. Shot Detection via Edge Extraction
Edges indicate the content structure of video frames!

41. 6. Shot Detection via Edge Extraction
a. Easy but not smart
MPEG videos
Automatic
Threshold
Determination
MPEG Decoder
Color Edge Detection
Scene Cut Frame Output

42. 6. Shot Detection via Edge Extraction
J. Fan, W.G. Aref, M.-S. Hacid and A.K. Elmagarmid, ``An
automatic isotropic color edge detection technique”,
Pattern Recognition Letters, vol.22, pp , 2001. 1 2 1 1 -1 2 1 1 2 2 -2 1 -1 -1 1 -1 -1 -2 -1 -2 1 -1 -2 -1

43. 6. Shot Detection via Edge Extraction
b. complex but smart
Only decode only part of MPEG videos
DCT
What you find from this figure?

44. 6. Shot Detection via Edge Extraction
1. Calculate the directional edge histogram
Horizontal edges
Vertical edges
Northwest diagonal edges
Northeast diagonal edges
2. Calculate the differences of the directional edge histogram
3. Threshold to obtain the scene cuts

45. 7. Complex Shot Detection
Effects on color histogram different by Fades & Dissolves

46. 7. Complex Shot Detection
Wipe Modeling
Dissolve Modeling

47. 8. Camera Motion Detection
Zoom in

48. 8. Camera Motion Detection
b. Zoom out

49. 8. Camera Motion Detection
c. Left Pan

50. 8. Camera Motion Detection
d. Right Pan

51. Why We Need Keyframe Extraction?

52. Why We Need Keyframe Extraction?

53. Keyframe Extraction What are keyframes? Different from others!
How to measure the significance of video frames?
How to make the decision?

54. 9. Background Modeling for Shot Detection

55. 9. Background Modeling for Shot Detection

56. 9. Background Modeling for Shot Detection

57. 9. Background Modeling for Shot Detection

58. 9. Background Modeling for Shot Detection

59. 9. Background Modeling for Shot Detection

60. 10. Shot-Based Video Content Representation
Statistical Information
Keyframe-Based Information
Color histogram & variance;
(b) edge histogram & variance
© motion histogram & variance;
Color histogram; (b) edge histogram
© motion histogram

61. 11. Shot-Oriented Video Semantics Interpretation

62. Who cares?

63. 12. Who works on this topic?
1. IBM research center
2. Intel
3. Microsoft Redmond & Beijing
4. Kodak at Rochester
5. HP at Polo Alto
6. Google
7. Yahoo!

64. 12. Requirement
If you are asked to work on video shot detection from compressed video stream, what you should know (basic requirement)?
MPEG video coding standard:
how to decode and read MPEG file
b. Knowledge on video shot analysis:
Color, motion, edge, fade, dissolve, …
c. Skills on C++

65. Design a system for automatic shot detection from MPEG video streams
Project Introduction
Design a system for automatic shot detection from MPEG video streams
Interface design: read in MPEG files and display shot boundaries, or even decision process!