1. Performance Characterization of Video-Shot-Change Detection Methods U. Gargi, R. Kasturi, S. Strayer Presented by: Isaac Gerg

2. What is a Shot? “The process of identifying changes in the scene content of a video sequence so that alternate representations may be derived for the purposes of browsing and retrieval.” ~ Quoted directly Shot – A sequence of frames shot from the same camera. Shot-Change examples: cuts, transitions, wipes, etc.

3. Why Do We Care? Indexing – video retrieval Compression (e.g. MPEG) – determining key frames. Removing commercials! (TiVo)

4. Preview 1. Create a method for measuring the performance of a shot-change algorithm. 1. Measure both false detections & missed detections. 2. Measure performance of both cut detection & gradual transition detection. 2. Apply shot-change algorithms to ground truth video sequence. 3. Perform measurements and throughput analysis. 4. Compare the results.

5. Ground Truth Video Sequence 640x480 @ 30 frames/s. ~75 minutes in length M-JPEG format Human volunteers used to establish ground truth. Custom software used to notate shot- change.

6. Defining a Detection Algorithm detection must occur within so many frames of ground truth detection. Mapping Range = R M Cut changes: R M = 3 Gradual Transition: R M =10

7. Detection Performance Measurements Where: MD is Missed Detections; FA is False Alarms.

8. Desirable Characteristics 90%-95% recall with 70%-75% precision. Robust. Automatic thresholds. High throughput. Perform well on both cuts and gradual transitions.

9. Algorithms Evaluated Color Histograms RGB, HSV, YIQ, XYZ, L*a*b, L*u*v, Munsell, Opponent Frame Difference Measurements Bin-to-bin Differences (B2B), Chi-square test, Histogram intersection, Average Color Dimensionality – 1D, 2D, 3D MPEG Algorithms – A, B, C, D, E, F Block Matching Methods – A, B, C

10. Best Methods - Cut Histogram intersection: 1D and 3D methods.

11. Best Methods - Cut MTM colorspace (many flops). LAB appeared as good compromise when considering throughput. Opponent (OPP) almost as good as LAB, but needs only integer computations. [image] Hall, E. L.. Computer Image Processing and Recognition. Academic Press, New York

12. Best Methods - Cut Best recall: MPEG-A, 97% with 6% precision. Uses only I frames. Best precision: MPEG-D, 88% with 79% recall. Uses I, B, & P frames.

13. Worst Methods - Cut Chi-square test histogram difference: Average color of a frame. 2D methods. Indicates luminance is important. YYY colorspace. Indicates color content is important All the block-matching methods.

14. Best Methods - Transition Only MPEG algorithms evaluated. MPEG-D: Uses all frames (I, P, B). Uses multiframe differences to detect gradual transitions. Uses 11 parameters. MPEG-F: Uses color information (Y, Cr, Cb). Uses order statistics to detect gradual transitions. Needs 7 parameters.

15. Worst Methods - Transition MPEG-A was the worst. Only contains I frames. Most performed poorly as they expected a particular transition curve.

16. MPEG - Source Effects Desirable to have a good MPEG method independent of encoder. Authors found dependence on algorithm performance and MPEG encoder used. MPEG does not specify encoding method, only syntax of encoded bitstream. Different error estimates or DCT matrices may be used during encoding. MPEG-F appeared to be the most robust.

17. Conclusions Need accurate model of color. Color & luminance information combined yield best results. MPEG shot detection & gradual transition methods have a long way to go. Encoding too variable. Gradual transitions not detected well by an of the MPEG methods.

18. Questions?