Presentation is loading. Please wait.

Presentation is loading. Please wait.

M. Wu: ENEE631 Digital Image Processing (Spring'09) Video Content Analysis and Streaming Spring ’09 Instructor: Min Wu Electrical and Computer Engineering.

Similar presentations


Presentation on theme: "M. Wu: ENEE631 Digital Image Processing (Spring'09) Video Content Analysis and Streaming Spring ’09 Instructor: Min Wu Electrical and Computer Engineering."— Presentation transcript:

1 M. Wu: ENEE631 Digital Image Processing (Spring'09) Video Content Analysis and Streaming Spring ’09 Instructor: Min Wu Electrical and Computer Engineering Department, University of Maryland, College Park   bb.eng.umd.edu (select ENEE631 S’09)   ENEE631 Spring’09 Lecture 19 (4/13/2009)

2 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [2] Overview and Logistics Last Time: –General methodologies on motion analysis –Optical flow equations Today: –Wrap up motion analysis –Video content analysis u Basic framework u Temporal segmentation; Compressed domain processing –A quick guide on video communications UMCP ENEE631 Slides (created by M.Wu © 2004)

3 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [3] Review: Optical Flow Equation Orthogonal decomposition of the flow vector v –Projection along “normal direction” ~ v n i.e., along image gradient  f ’s direction –Projection along tangent direction ~ v t i.e., along orthogonal direction to image gradient  f O.F.E.  f f Normal direction Tangent direction From Wang’s Preprint Fig.6.2 UMCP ENEE631 Slides (created by M.Wu © 2001)

4 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [4] Ambiguity in Motion Estimation One equation for two unknowns –Tangent direction of motion vector is undetermined –“Aperture problem” u Aperture ~ small window over which to apply const. intensity assumption u MV can be estimated only if aperture contains 2+ different gradient directions (e.g. corners) –Usually need additional constraints u Spatial smoothness of motion field Indeterminate motion vector over constant region (||  f || = 0) –Reliable motion estimation only for regions with brightness variations (e.g. edges or nonflat textures) From Wang’s Preprint Fig.6.3 UMCP ENEE631 Slides (created by M.Wu © 2001)

5 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [5] General Methodologies for Motion Estimation Two categories: Feature vs. Intensity based estimation Feature based –Step-1 establish correspondences between feature pairs –Step-2 estimate parameters of a chosen motion model by least-square fitting of the correspondences –Good for global/camera motion describable by parametric models u Common models: affine, projective, … (Wang Sec ) u Applications: Image mosaicing, synthesis of multiple-views Intensity based –Apply optical flow equation (or its variation) to local regions –Good for non-simple motion and multiple objects –Applications: video coding, motion prediction and filtering UMCP ENEE631 Slides (created by M.Wu © 2001)

6 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [6] Motion Estimation Criteria Criterion based on displaced frame difference –E.g. in block matching approach Criterion based on optical flow equations Other criteria and considerations –Smoothness constraints –Bayesian criterion UMCP ENEE631 Slides (created by M.Wu © 2001)

7 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [7] Commonly Used Optimization Methods For minimizing the previously defined M.E. error function Exhaustive search –MAD often used for computational simplicity –Guaranteed global optimality at expense of computation complexity –Fast algorithms for sub-optimal solutions Gradient-based search (Appendix B of Wang’s book) –MSE often used for mathematical tractability (differentiable) –Iterative approach u refine an estimate along negative gradient directions of objective func. –Generally converge to local optimal u require good initial estimate –Estimation method of Gradient also affects accuracy & robustness UMCP ENEE631 Slides (created by M.Wu © 2001)

8 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [8] Various Motion Estimation Approaches Pixel-based motion estimation (Wang’s sec.6.3) u Estimate one MV for every pixel u Use relation from Optical Flow Equation to construct M.E. criterion u Add smoothness constraints on motion field to deal with aperture problem and avoid poor estimation of MV Block-matching –Correlation method (Wang’s sec.6.4.5) Deformable block-matching (Wang’s sec.6.5) –Use more block-based motion model than translational model u e.g., affine/bilinear/projective mapping for each block (sec.5.5) u square block in current frame match with non-square block in ref. Mesh-based motion estimation (Wang’s sec.6.6) UMCP ENEE631 Slides (created by M.Wu © 2001)

9 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [9] Example: Pixel-Based Motion Estimation Estimate motion vectors at each pixel –Based on Optical Flow Equation –Add smoothness constraints on motion field to avoid poor M.E. –Gradient based search ~ e.g. steepest gradient descent Motion estimation criterion –Expect LHS of O.F. Equation to be zero –Try to minimize the “residue” of LHS –Smoothness constraints u Add magnitude of spatial gradient of velocity vectors to objective func. UMCP ENEE631 Slides (created by M.Wu © 2001)

10 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [10] Video Content Analysis

11 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [11] Recall: MPEG-7 “Multimedia Content Description Interface” –Not a video coding/compression standard like previous MPEG –Emphasize on how to describe the video content for efficient indexing, search, and retrieval Standardize the description mechanism of content –Descriptor, Description Scheme & Description Definition Languages –Commonly used visual descriptors: Color, Texture, Shape, … Figure from MPEG-7 Document N4031 (March 2001) UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

12 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [12] Introduction to Video Content Analysis Teach computer to “understand” video content –Define features that computer can learn to measure and compare u color (RGB values or other color coordinates) u motion (magnitude and directions) u shape (contours) u texture and patterns –Give example correspondences so that computer can learn u build connections between feature & higher-level semantics/concepts u statistical classification and recognition techniques Video understanding 1.Break a video sequence into chunks, each with consistent content ~ “shot” 2.Group similar shot into scenes that represent certain events 3.Describe connections among scenes via story boards or scene graphs 4.Associate shot/scene with representative feature/semantics for future query UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

13 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [13] Video Understanding (step-1) –Break a video sequence into chunks, each with consistent content ~ “shot” From Yeung-Yeo-Liu: STG (Princeton)

14 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [14] Video Understanding (step-2) –Group similar shot into scenes From Yeung-Yeo-Liu: STG (Princeton)

15 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [15] Video Understanding (step-3) –Describe connections among scenes via story boards or scene graphs From Yeung-Yeo-Liu: STG (Princeton)

16 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [16] Video Temporal Segmentation A first step toward video content understanding –Elect “key frames” to represent each shot for index/retrieval –Sequence of shot duration as a “signature” for a video Two types of transitions –“Cut” ~ abrupt transition – Gradual transition: Fade out and Fade in; Dissolve; Wipe Detecting transitions –Detecting cut is relatively easier u check frame-wise difference –Detecting dissolve and fade by checking linearity u f 0 (1 – t/T) + f 1 * t/T –Detecting wipe ~ more difficult u exploit transition patterns, or linearity of color histogram UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

17 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [17] Detect Dissolve via Linearity in Pixel Changes Dissolve: a linear combination of g and h Detect straight lines in DC frame space –correlation detection on triplets dissolve m n Pixel 1 Pixel 2 Pixel 3 From talks by Joyce-Liu (Princeton)

18 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [18] Examples of Wipes UMCP ENEE408G Slides (created by M.Wu © 2002)

19 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [19] Wipe Detection (1) –Convert the 2-D problem to 1-D by projection u A common strategy in feature extraction and analysis in image processing –Perform horizon, vertical, diagonal projection to detect diverse wipe types UMCP ENEE408G Slides (created by M.Wu © 2002)

20 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [20] Review: Color Histogram Generalize from luminance histogram What is color histogram? –Count the # of pixels with the same color –Plot color-value vs. corresponding pixel# Give idea of the dominate color and color distribution –Ignore the exact spatial location of each color value –Useful in image and video analysis Color histogram can be used to: –Detect gradual shot transition esp. for fancy wipes –Measure content similarity between images / video shots UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

21 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [21] Wipe Detection (2) Diverse and fancy wipes Linear change in color histogram Ref: Joyce & Liu, IEEE Trans. Multimedia, wipe m n Bin 1 Bin 2 Bin 3 From talks by Joyce-Liu (Princeton)

22 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [22] Types of Transitions – [above] Transition types offered by Adobe Premiere – See also transition demos provided by PowerPoint From talks by Joyce-Liu (Princeton) Video transition collection (Dr. Rob Joyce)

23 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [23] Compressed-Domain Processing Does video analysis have to decompress the whole video? Use I & P frames only to reduce computation and enhance robustness in scene change detection … I b b P b b P b b P b b I b b P … Working in compressed domain –Process video by only doing partial decoding (inverse VLC, etc.) without a full decoding (IDCT) to save computation –Low-resolution version provides enough info for transition detection => “DC-image” UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

24 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [24] DC Image –Put DC of each block together –Already contain most information of the video DC Frame Example From Joyce-Liu (Princeton)

25 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [25] Fast Extraction of DC Image From MPEG-1 I frame –Put together DC coeff. from each block (and apply proper scaling) Predictive (P/B) frame –Fast approximation of reference block’s DC –Adding DC of the motion compensation residue u recall DCT is a linear transform See Yeo-Liu’s paper for more derivations on approximations (DC; DC+2AC) C R UMCP ENEE408G Slides (created by M.Wu © 2002)

26 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [26] Compressed-Domain Scene Change Detection Compare nearby frames –Take pixel-wise difference of nearby DC-frames –Or take pixel-wise difference of every N frames to accumulate more changes => useful for detect gradual transitions Observe the pixel-wise difference for different frame pairs cuts, and gradual transitions Figure from Yeo-Liu CSVT’95 paper UMCP ENEE408G Slides (created by M.Wu © 2002)

27 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [27] Scene Change Detection (cont’d) Figure from Yeo-Liu CSVT’95 paper UMCP ENEE408G Slides (created by M.Wu © 2002) –Identify candidate places for gradual transitions –Can further explore the linearity in DC frames => Help differentiate gradual transitions from motions

28 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [28] Summary on Video Temporal Segmentation A first step toward video content understanding Two types of transitions –“Cut” ~ abrupt transition – Gradual transition: Fade out and Fade in; Dissolve; Wipe Detecting transitions: can be done on “DC images” w/o full decompression –Detecting cut is relatively easier ~ check frame-wise difference –Detecting dissolve and fade by checking linearity u f 0 (1 – t/T) + f 1 * t/T –Detecting wipe ~ more difficult u exploit transition patterns, or linearity of color histogram UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

29 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [29] Video Communications

30 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [30] MM + Data Comm. = Effective MM Communications? Multimedia vs. Generic Data –Perceptual no-difference vs. Bit-by-bit accuracy –Unequal importance within multimedia data –High data volume and real-time requirements Need consider the interplay between source coding and transmission and make use of MM specific properties E.g. wireless video need “good” compression algorithm to: –Support scalable video compression rate ( from 10 to several hundred kbps) –Be robust to the transmission errors and channel impairments –Minimize end-to-end delay –Handle missing frames intelligently

31 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [31] Error-Resilient Coding with Localized Synch Marker To reduce error propagation Output sequence Input sequence H.263 encoder MB detection LRM H.263 decoder Error concealment Random noise H.263 with FRM H.263 with LRM (From D. HK PolyUniv. Short Course 6/01)

32 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [32] Issues in Video Communications/Streaming Source coding aspects –Rate-Distortion tradeoff and bit allocation in R-D optimal sense –Scalable coding and Fine Granular Scalability (FGS) –Multiple description coding –Error resilient source coding Channel coding aspects ~ see ENEE626 for general theory –Unequal Error Protection (UEP) channel codes –Embedded modulation for achieving UEP Joint source-channel approaches –Jointly select source and channel coding parameters to optimize end-to-end distortion –Wisely map source codewords to channel symbols –Take advantage of channel’s non-uniform characteristics for UEP Bandwidth resource determination, allocation & adaptation

33 M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec.19 – Video Analysis & Comm [33] Reading References Video temporal segmentation for content analysis –Yeo-Liu CSVT 12/1995 paper (DC-image & scene change detection) –Joyce-Liu TMM 2006 paper (Wipe detection) Video communications –Wang’s video textbook: Chapter 14, 15. –Wood’s book: Chapter 12 UMCP ENEE408G Slides (created by M.Wu © 2002)


Download ppt "M. Wu: ENEE631 Digital Image Processing (Spring'09) Video Content Analysis and Streaming Spring ’09 Instructor: Min Wu Electrical and Computer Engineering."

Similar presentations


Ads by Google