E.G.M. PetrakisVideo Processing1  Video is a rich information source  frames (individual images)  links between frames (cuts, fades, dissolves)  changes in color, shapes, motion of both camera and objects  acquisition (shot angles, camera motion)  each type of video has its own characteristics (commercials, news, sports)

E.G.M. PetrakisVideo Processing2 Video Structure  Frame: typically 1/25 or 1/30 seconds  Shot: sequence of similar frames  elementary video units  a single event  Clip / Scene: sequence of shots consecutive in time, space, action  Episode: consecutive scenes  intro, news, reporter, weather

E.G.M. PetrakisVideo Processing3 Del Bimbo 99 Video Structure (cont.d)

E.G.M. PetrakisVideo Processing4 Video Retrieval  A video can be accessed at the  structural level: browsing, retrieval of shots, scenes, episodes  content level: according to camera motion, motion of characters or objects, audio properties, scenes, semantics of color, texture, shape, object properties …

E.G.M. PetrakisVideo Processing5 Video Partitioning  Shot extraction and classification of editing effects due to  camera breaks (cuts): abrupt transitions  gradual transitions: dissolves, wipes, fade- in/out  camera movements: panning, tilting, zoom  Simple methods for camera breaks  More sophisticated methods for gradual transitions and camera movement

E.G.M. PetrakisVideo Processing6 shot1 shot2 Furht. et.al 96 Cut: Frames Between Shots

E.G.M. PetrakisVideo Processing7 Furht. et.al 96 Dissolve: Transition Frames

E.G.M. PetrakisVideo Processing8 Uncompressed Video Partitioning  Detect boundaries of consecutive camera shots  compare adjacent frames  for camera breaks compare color histograms of adjacent frames  for gradual transitions and camera motion histograms are less successful  Other techniques are based on edge detection, motion analysis etc.

E.G.M. PetrakisVideo Processing9 Histograms  Grey level histograms for 3 successive frames  Frames 1 and 2 almost identical  Camera break between 2 and 3  Compute histogram differences

E.G.M. PetrakisVideo Processing10 Color Histograms  H(I,v) : number of pixels in I with intensity v  MxN pixels  Grey-level images: 8 bits/pixel bins in histogram  Color images: 24 bits/pixel bins  Convert color to YUV color space and process intensity only:  I = 0.299R G B

E.G.M. PetrakisVideo Processing11 1. Camera Breaks  Pair-wise pixel comparison (intensities)  Histogram comparison for camera breaks  threshold selection  twin comparison approach  multi-pass approach  Motion vector analysis for camera motion and gradual transitions  Hough transform  Video X-ray

E.G.M. PetrakisVideo Processing12 Pair-Wise Pixel Comparison  Count pixels changed from a frame to the next  A shot boundary is found if more than T b pixels changed  Problem: sensitivity to camera/object motion and noise  many pixels change

E.G.M. PetrakisVideo Processing13 Pair-Wise Block Comparison  Compare blocks instead of pixels  μ i,μ i+1 : mean intensity values in frames  s i,s i+1 : variances  Less sensitive to motion and noise  t: does not change for different video sources

E.G.M. PetrakisVideo Processing14 Pair-Wise Histogram Comparison  Even less sensitive to motion  i: frame count  j: intensity count in H  G=MxN intensities Furht. et.al 96

E.G.M. PetrakisVideo Processing15 gradual transitions camera breaks Furht et.al. 96 Histogram Comparison

E.G.M. PetrakisVideo Processing16 Thresholds  Tolerate variations while ensuring good performance  low thresholds accept many false positives  high thresholds reject true transitions  Threshold: varies from one video source to another  e.g., cartoons exhibit larger frame differences than films

E.G.M. PetrakisVideo Processing17 2. Gradual Transitions  Transitions not as high as in camera breaks  dissolve, fade-in/out, other special effects  high transitions in a neighborhood  lower thresholds do not solve the problem Furht. et.al 96

E.G.M. PetrakisVideo Processing18 Twin-Comparison  Two thresholds  T b for camera break detection  T s < T b for special effects like dissolves, motion  Compare consecutive frames (e.g. histograms)  if difference exceeds T b : camera break  if difference exceeds T s : potential cut  Accumulate differences from that frame until the transition becomes lower than T s  A boundary is detected if the accumulated difference becomes higher than T b

E.G.M. PetrakisVideo Processing19 camera break special effect shot boundary Twin Comparison Example Furht. et.al 96

E.G.M. PetrakisVideo Processing20 Threshold Selection  Based on the distribution of the frame-to- frame histogram changes  Most changes are due to noise ~90%, scene changes ~10%  Scan entire video and compute distribution of changes (e.g., histogram differences)  Assume Gaussian distribution and compute: μ, σ  Compute the two thresholds as  T b =μ+ασ, α=4-6  T s =βμ, β=1.5-2

E.G.M. PetrakisVideo Processing21 Motion Vector Analysis (object motion)  Detect camera breaks using motion vectors (MV)  Compute MVs by block matching  Compute correlation of the same block b i from frame i to frame i+1  Assign a displacement vector Db i to b i  The same for all blocks between frames

E.G.M. PetrakisVideo Processing22 Motion Smoothness  For each frame compute W i  Nominator counts significant motion vectors in frame i  Denominator counts significant transitions in motion vectors  W i  0 indicates camera break

E.G.M. PetrakisVideo Processing23 3. Camera Motion  Detect changes due to camera movement  Camera zoom in/out, tilting/panning  Transitions resemble gradual transitions  More specific techniques  Analysis of motion vectors Furht. et.al 96

E.G.M. PetrakisVideo Processing24 Camera Panning  Most motion vector exhibit same direction  Σ b | θ b - θ m | < Θ p  Θ b is the direction of the vector of block b  θ m is the direction of the entire set of blocks  The variation Θ p should be close to 0 Furht. et.al 96

E.G.M. PetrakisVideo Processing25 Camera Zooming  Assume focus center within the frame and little object motion at the periphery of a frame  Compare v in top/bottom rows, left/right columns  Every column |v top -v bottom | >= max(|v top |,|v bottom |)  Every raw |v left -v right | >= max(|v left |,|v right |)  Zooming: most vectors satisfy these condition

E.G.M. PetrakisVideo Processing26 Compressed Video Partitioning  Frame comparison using  DCT coefficients instead of blocks  MPEG motion vectors  Combination of the above  Same techniques using I frames only (faster) Furht. et.al 96

E.G.M. PetrakisVideo Processing27 Pair-Wise DCT Comparison  Applies to the DCT coefficients of corresponding blocks in consecutive I frames which are f-distance apart  k= coefficients  For each block computes

E.G.M. PetrakisVideo Processing28 Threshold Selection  Same techniques for threshold selection with uncompressed video  Diff l > t : a block has changed  t: does not vary with video sources  Shot boundary: the percentage of blocks that changed exceeds T b  T b varies for different video sources

E.G.M. PetrakisVideo Processing29 Example  Sharp peaks indicate camera breaks  Cannot handle gradual transitions, camera or object motion  Twin comparison using T s, T b thresholds  Applies only to I frames (no DCTs for P, B frames)  Faster but, many false positives D values between successive frames Furht. et.al 96

E.G.M. PetrakisVideo Processing30 Motion Vectors  Motion vectors are associated with P, B frames  The residual error between blocks is DCT encoded  If the error is large DCT encode the original blocks and no motion vectors are stored in this case  Many blocks with no motion vectors indicate camera break  Camera break: motion vectors M < T b ~ 0

E.G.M. PetrakisVideo Processing31 Motion Vectors (cont.d)  Many false positives for static frames (frames with no motion)  Camera breaks: deep and narrow gaps in diagrams with number of vectors  Combine with D(i,i+f): camera break when high D(i,i+f) with M < T b but large gap means no motion (static frames) and not transition  Difficult to detect gradual transitions

E.G.M. PetrakisVideo Processing32 Example static frames shot boundary M < T b Furht. et.al 96

E.G.M. PetrakisVideo Processing33 Hybrid Multiple Pass Approach  First step: DCT comparison on I frames to locate regions of potential interest  low spatial resolution (large f) : very fast  advantage: gradual transitions are likely to be detected because of the large skip factor  disadvantage: many false positives  Second step combining DCTs and motion vectors  smaller skip factor  at the vicinity of candidate boundaries  High processing speed in achieved

E.G.M. PetrakisVideo Processing34 Camera Motion - Object Motion  Detect specific patterns of motion vectors  Similar techniques with uncompressed video  Motion vectors are provided by MPEG P, B frames  Similar results Furht. et.al 96

E.G.M. PetrakisVideo Processing35 Video Browsing  Select a key-frame from each shot  First, middle, last, average frame of shots, I frames for compressed video …  Image retrieval based on key-frames key-frames A.Smeaton,DCU

E.G.M. PetrakisVideo Processing36 Hierarchical Browsing  Problem: large number of key-frames  Solution: organize key-frames hierarchically  video at the top, key-frames for scenes, shots are lower hierarchical video browser A.Smeaton,DCU

E.G.M. PetrakisVideo Processing37 Furht et.al. 96 Hierarchical Browser

E.G.M. PetrakisVideo Processing38 Comments on Video Segmentation  Histograms are sufficient in most cases  Audio could help (silence between shots)  Only one pass through the entire video  Computational cost and delay can be high  A pass at reduced spatial resolution detects potential changes (comparisons every k frames)  Processing at the vicinity of changes to verify the results

E.G.M. PetrakisVideo Processing39 Further Reading  Centre for Digital Video Processing, Dublin University  B. Furht et.al. “Video and Image Processing in Multimedia Systems”, Chapter 12-14, Kluwer, 1996  H.J.Zhang, A. Kankanhalli, S.W.Smoliar, “Automatic Partitioning of Full-Motion Video”, Multimedia Systems, 1(1):10-28, 1993  V. Kobla, D. Doermann, C. Faloutsos “VideoTrails: Representing and Visualizing Structure in Video Sequences” ACM Multimedia 97, Nov. 1997VideoTrails: Representing and Visualizing Structure in Video Sequences  O. Marques, B. Furhrt, “Content Based Image and Video Retrieval”, Kluwer Academic Publishers, 2002  V. Kobla, D. S. Doermann, K-Ip (David) Lin, C. Faloutsos “Compressed domain video indexing techniques using DCT and motion vector information in MPEG video” Proc. of the SPIE Conf. on Storage and Retrieval for Image and Video Databases, Vol. 3022, Feb. 1997

E.G.M. PetrakisVideo Processing40 References  S. Lefevre, J. Holler, N. Vincent, “A review of Real-Time Segmentation of Uncompressed Video Sequences for Content- Based Search and Retrieval”, RFAI publication: Real Time ImagingA review of Real-Time Segmentation of Uncompressed Video Sequences for Content- Based Search and Retrieval  C. Doulaverakis, V. Vagionitis, M. Zervakis and E. Petrakis: "Adaptive Methods for Motion Characterization and Segmentation of MPEG Compressed Frame Sequence", Intern. Conference on Image Analysis and Recognition (ICIAR'2004), Porto, Portugal, Sept./Oct. 2004, Proc. Part I, Springer Verlag (LNCS 3211), pp Adaptive Methods for Motion Characterization and Segmentation of MPEG Compressed Frame Sequence