1. Video Dissolve and Wipe Detection via Spatio-Temporal Images of Chromatic Histogram Differences Presentation by Kenton Anderson CMPT 820 March 3 rd, 2005 Mark S. Drew, Ze-Nian Li, and Xiang Zhong International Conference on Image Processing ICIP'00 Vancouver, pp.III 929-932, Sept. 2000

2. Overview Background Background Introduction Introduction Wipe and Cut detection Wipe and Cut detection Dissolve detection Dissolve detection Conclusions Conclusions

3. Background What is a shot? What is a shot? Uninterrupted segment of video time Uninterrupted segment of video time The boundary between two shots is a camera break The boundary between two shots is a camera break Three major types of transitions: Three major types of transitions: Cut (instant) Cut (instant) Wipe(gradual) Wipe(gradual) Dissolve(gradual) Dissolve(gradual)

4. Gradual Transitions Wipe transition Wipe transition Moving boundary line between two shots crossing the screen such that one shot replaces the other Moving boundary line between two shots crossing the screen such that one shot replaces the other Dissolve transition Dissolve transition One shot blends smoothly into a second shot One shot blends smoothly into a second shot

5. Introduction Content-Based Image/Video Search, Retrieval, and Segmentation Why detect video transitions? Why detect video transitions? Segmentation is an important basic step! Segmentation is an important basic step! Before scenes can be searched for content, the location of the scenes have to be determined Before scenes can be searched for content, the location of the scenes have to be determined

6. Introduction This paper presents methods to detect gradual transitions using 2D chromaticity histogram metrics: This paper presents methods to detect gradual transitions using 2D chromaticity histogram metrics: Histogram intersection for wipes Histogram intersection for wipes Color-distance Histogram metric for dissolves Color-distance Histogram metric for dissolves Both generate potential indicators of shot transitions, with good results Both generate potential indicators of shot transitions, with good results

7. Wipe and Cut Detection In a wipe, a boundary line crosses the first shot, revealing the second shot In a wipe, a boundary line crosses the first shot, revealing the second shot

8. Wipe and Cut Detection Ngo et al. “Detection of gradual transitions through temporal slice analysis” Ngo et al. “Detection of gradual transitions through temporal slice analysis” Taking the pixels from the middle column and placing them sideways, stacking them over time Taking the pixels from the middle column and placing them sideways, stacking them over time Y X Column C at time t 1 Column C t Y t1t1

9. Wipe and Cut Detection Detect lines in the resultant spatio-temporal image Detect lines in the resultant spatio-temporal image

10. Wipe and Cut Detection Instead of using pixels, convert to 2D chromaticity coordinates Instead of using pixels, convert to 2D chromaticity coordinates r = R r = R R + G + B g = G g = G R + G + B Recall chromaticity from our class notes Recall chromaticity from our class notesclass notesclass notes

11. Wipe and Cut Detection 2D chromaticity conversion effectively eliminates the shadows 2D chromaticity conversion effectively eliminates the shadows Form a 2D chromaticity histogram for each column Form a 2D chromaticity histogram for each column Using the DC component of the frames Using the DC component of the frames Using Histogram Intersection, compare a frame to its previous frame to detect differences. Using Histogram Intersection, compare a frame to its previous frame to detect differences.Histogram IntersectionHistogram Intersection

12. Wipe and Cut Detection From frame to frame, the histogram intersection value for a column stays nearly the same From frame to frame, the histogram intersection value for a column stays nearly the same When the Wipe Boundary hits that column, the histogram intersection value is near zero When the Wipe Boundary hits that column, the histogram intersection value is near zero t Each element in a row represents a histogram intersection value for each column in the image (Black == 0) X Wipe Cut

13. Wipe and Cut Detection Conclusions Note that this techniques takes into account the entire image frame Note that this techniques takes into account the entire image frame Not just a slice Not just a slice The previous image has no edge enhancement performed on it The previous image has no edge enhancement performed on it Raw data Raw data

14. Dissolve Detection Replaces every pixel with a mixture of the two shots over time, gradually replacing the first by the second Replaces every pixel with a mixture of the two shots over time, gradually replacing the first by the second Each pixel is affected gradually Each pixel is affected gradually

15. Dissolve Detection Frame by frame of a cross dissolve Frame by frame of a cross dissolve Diagram Diagram

16. Dissolve Detection For dissolve detection, 2D Cb-Cr chromaticity space is adopted For dissolve detection, 2D Cb-Cr chromaticity space is adopted Recall from our class notes that YCbCr Colour model is used in JPEG image compression and MPEG video compression Recall from our class notes that YCbCr Colour model is used in JPEG image compression and MPEG video compression YCbCr is closely related to YUV YCbCr is closely related to YUV Y’ is the luma (for gamma corrected signals) Y’ is the luma (for gamma corrected signals) U and V is the chrominance U and V is the chrominance U = B’ – Y’V = R’ – Y‘

17. Dissolve Detection Define transition as; Define transition as; R = A + α(t)(B – A) (1) Where A and B are 2-vectors for video A and video B, in Cb-Cr space Where A and B are 2-vectors for video A and video B, in Cb-Cr space α(t) is a transition function α(t) is a transition function α(t) = Kt, with Kt max ≡ 1 (2)

18. Dissolve Detection Histogram Intersection fails on simple cases for dissolve detection Histogram Intersection fails on simple cases for dissolve detection For example, uniformly-coloured still images For example, uniformly-coloured still images H (K, M) never really drops to zero H (K, M) never really drops to zero To counter this problem, use a histogram- difference metric To counter this problem, use a histogram- difference metric

19. Dissolve Detection Histogram-difference metric: Histogram-difference metric: Hafner et al.’s metric is a weighted distance between colour distributions of two images, generating a histogram distance measure Hafner et al.’s metric is a weighted distance between colour distributions of two images, generating a histogram distance measure Histogram difference D 2 Histogram difference D 2 D 2 = z T Az

20. Dissolve Detection Summary of modifications For the histogram difference D 2 For the histogram difference D 2 Use 2D CbCr chromaticity space (vs 3D color) Use 2D CbCr chromaticity space (vs 3D color) Use only DC components of video Use only DC components of video Analyzes actual pixel values (vs histogram) Analyzes actual pixel values (vs histogram) Use Euclidean Distance metric for difference Use Euclidean Distance metric for difference Derivation Derivation Derivation

21. Dissolve Detection Results of modifications For time t 1 and t 2 (beginning and ending of dissolve transition) For time t 1 and t 2 (beginning and ending of dissolve transition) Temporal differences for each column is Temporal differences for each column is k(t 1 - t 2 ) 2, k is a constant k(t 1 - t 2 ) 2, k is a constant If t 1 – t 2 is constant, D 2 is constant If t 1 – t 2 is constant, D 2 is constant Normalized D 2 is approximately Normalized D 2 is approximately 0 outside a dissolve 0 outside a dissolve 1 during a dissolve 1 during a dissolve

22. Dissolve Detection Results of modifications cont’d For time t 1 and t 2 (beginning and ending of dissolve transition) For time t 1 and t 2 (beginning and ending of dissolve transition) If t 1 is fixed, and t 2 varies, √D 2 is linear If t 1 is fixed, and t 2 varies, √D 2 is linear D 2 has 3 components, each quadratic in time, thus having a linear derivative D 2 has 3 components, each quadratic in time, thus having a linear derivative

23. Dissolve Detection Fully derived expression Fully derived expression for linear transition for linear transition D 2 = 2(1/d 2 max ) K 2 (t 1 -t 2 ) 2 ∑∑(B i -A i ) T (B j -A j ) i j i j

24. Dissolve Detection Process 2 frames as part of a dissolve in Cb-Cr space 2 frames as part of a dissolve in Cb-Cr space t 1 : initial frame t 2 : time-varying frame D2D2 Result (t1 – t2) = ∆t is NOT constant √D2√D2

25. Dissolve Detection Results: Results: √ D 2, differencing frame to initial frame at a 1 frame interval Approx. 1 Approx. 0

26. Dissolve Detection Process 2 frames as part of a dissolve in Cb-Cr space 2 frames as part of a dissolve in Cb-Cr space t 1 : frame A t 2 : frame B D2D2 Result (t1 – t2) = ∆t is constant Derivative D 2 is linear ⌡D2⌡D2

27. Dissolve Detection Results: Results: D 2, differencing for a constant t 1 -t 2D 2, differencing for a constant t 1 -t 2 Boundaries of the transition are evidentBoundaries of the transition are evident Values in Transition periods are relatively constantValues in Transition periods are relatively constant Time t DerivationDerivation of D 2 Derivation

28. Dissolve Dectection Conclusions Use of multiple columns and rows provides a large number of descriptors for gradual transitions Use of multiple columns and rows provides a large number of descriptors for gradual transitions For constant t 1 – t 2 For constant t 1 – t 2 D 2 is constant during transitions, 0 otherwise D 2 is constant during transitions, 0 otherwise For fixed t 1 For fixed t 1 D 2 is 1 during transition, 0 otherwise D 2 is 1 during transition, 0 otherwise In testing, this measure performs best when each video in the dissolve does not change much during the transition In testing, this measure performs best when each video in the dissolve does not change much during the transition

29. Conclusions 2 new measures are presented for detecting cuts, wipes and dissolves 2 new measures are presented for detecting cuts, wipes and dissolves Both use multiple columns (or rows or diagonals) to generate descriptors Both use multiple columns (or rows or diagonals) to generate descriptors Histogram intersection is fast and effective Histogram intersection is fast and effective In Dissolve testing, the measures perform best when each video in the dissolve does not change much during the transition In Dissolve testing, the measures perform best when each video in the dissolve does not change much during the transition

30. Video Dissolve and Wipe Detection The End

31. Histogram Difference Derivation Histogram difference D 2 Histogram difference D 2 D 2 = z T Az (3) A = [a ij ] is a symmetric matrix where a ij denotes similarity between bins i and j A = [a ij ] is a symmetric matrix where a ij denotes similarity between bins i and j A red,orange,blue = A red,orange,blue = 10.90 0.910 001 ROBROB R O B Red and Orange are considered highly similar

32. Histogram Difference Derivation D 2 = z T Az (3) For a ij, For a ij, a ij = (1 – d ij /d max ) (4) d ij defined as a three-dimensional colour difference d ij defined as a three-dimensional colour difference Vector z is a histogram-difference vector (for vectorized histograms) Vector z is a histogram-difference vector (for vectorized histograms) For example, z would be of length 256 if our chromaticity histograms were 16x16 For example, z would be of length 256 if our chromaticity histograms were 16x16

33. Histogram Difference Derivation Instead of 3D colour space, we use 2D CbCr chrominance space Instead of 3D colour space, we use 2D CbCr chrominance space Also, use an Euclidean distance metric Also, use an Euclidean distance metricEuclidean distanceEuclidean distance a ij will no longer be linear under a temporal transition with linear α(t) a ij will no longer be linear under a temporal transition with linear α(t) This modification maintains the linearity: This modification maintains the linearity: a ij = (1 – d 2 ij /d 2 max ) (5)

34. Histogram Difference Derivation Suppose we use only DC components Suppose we use only DC components Each frame will consist of only 1/8 th of the number of rows in an image Each frame will consist of only 1/8 th of the number of rows in an image Recall equation (3) Recall equation (3) D 2 = z T Az (3) z is the difference of 2 histograms, x and y z is the difference of 2 histograms, x and y z = (x – y) x and y are normalized to 0 ≤ x i, y i ≤ 1, ∑x = ∑y = 1 x and y are normalized to 0 ≤ x i, y i ≤ 1, ∑x = ∑y = 1 Then -1 ≤ z i ≤ 1 Then -1 ≤ z i ≤ 1

35. Histogram Difference Derivation To generate an analytic expression: To generate an analytic expression: Assume x and y are infinitely precise, Assume x and y are infinitely precise, z = (1, 1, 1, …, -1, -1, -1) In our video transition context, In our video transition context, This means 1’s entries for current column for the previous frame, and -1’s entries for the current column in the current frame This means 1’s entries for current column for the previous frame, and -1’s entries for the current column in the current frame

36. Histogram Difference Derivation Expanding D 2 = z T Az, given assumptions: Expanding D 2 = z T Az, given assumptions: Where R is the CbCr 2-vector at time t 1 for the i-th row in the current column Where R is the CbCr 2-vector at time t 1 for the i-th row in the current column Differencing between time t 1 and time t 2 Differencing between time t 1 and time t 2

37. Histogram Difference Derivation With a Euclidean distance metric and substituting equation (1): R = A + α(t)(B – A) With a Euclidean distance metric and substituting equation (1): R = A + α(t)(B – A)

38. Histogram Difference Derivation For linear transition (as is usually the case for dissolves), the previous equation can be simplified: For linear transition (as is usually the case for dissolves), the previous equation can be simplified: Since the sum above is simply a constant, then for constant (t 1 – t 2 ), the difference D 2 is constant over the transition! Since the sum above is simply a constant, then for constant (t 1 – t 2 ), the difference D 2 is constant over the transition! Back

40. Histogram Intersection Given a pair of histograms, K and M, each containing n buckets, the intersection is: Given a pair of histograms, K and M, each containing n buckets, the intersection is: n H = ∑ min(K j, M j ) j=1 j=1 The result of the intersection is the number of pixels in M that have corresponding pixels of the same colour in K The result of the intersection is the number of pixels in M that have corresponding pixels of the same colour in K

41. Histogram Intersection Example 1 Two sample histograms for 4-bit greyscale 4x4 images K1K1 M H (K, M) = 6 + 0 + 8 + 2 = 16

42. Histogram Intersection Example 2 Two sample histograms for 4-bit greyscale 4x4 images K2K2 M H (K, M) = 1 + 0 + 1 + 2 = 4

43. Histogram Intersection Normalized between 0 and 1: Normalized between 0 and 1: H(K, M) = Closer to zero, less histogram match Closer to zero, less histogram match ∑min(K j, M j ) ∑M j ∑M j

44. Histogram Intersection High-level representation High-level representation Poor match K M K M Close match K M K M Back

45. Euclidean Distance If u = (x 1, y 1 ) and v = (x 2, y 2 ) are two points on the plane, their Euclidean distance is given by: If u = (x 1, y 1 ) and v = (x 2, y 2 ) are two points on the plane, their Euclidean distance is given by: √(x 1 – x 2 ) 2 + (y 1 – y 2 ) 2 √(x 1 – x 2 ) 2 + (y 1 – y 2 ) 2 Geometrically, it's the length of the segment joining u and v Geometrically, it's the length of the segment joining u and v For d ij, which was previously defined as our 3D colour difference For d ij, which was previously defined as our 3D colour difference Back