1. Computer Science Engineering Lee Sang Seon

3.  Introduction  Basic notions for temporal video boundaries  Micro-Boundaries  Macro-Boundaries  Mega-Boundaries  Conclusion  Q & A

4.  Brief definition of Temporal Video Boundary technique → Examine the temporal boundary problem at different levels of video content structure analysis  Why we need Temporal Video Boundary technique? Show example

5. Insufficient metadata opening ending

6. Detailed metadata opening ending actor winners awards ending

7.  Video contains three types of modalities (i) Visual (ii) Audio (iii) Textual  Each modality has three levels (i) low-level(ii) mid -level(iii) high-level → levels describe the amount of details described in each modality in terms of granularity and abstraction

8.  For each modality and for each level there if a set of attributes. These can be formalized as vectors:

9.  Adding to this, given a set of vectors → their average value denote the vector

10.  Local method → the difference is computed between consecutive frames  Global method → the difference if computed over a series of frames

11.  Definition  Boundaries associated to the smallest video units for which a given attribute is constant or slowly varying  The attribute can be any feature in the visual, audio, or text domain

13. Data structure that represents the color information of a family of frames. Set of frames that exhibits uniform features = Frame histogram

14.  Histogram difference using L1 metrics  Bin-wise histogram intersection Total number of color bins used Histogram of previous frame Histogram of current frame

16. 1. Contiguous with zero memory → A new frame histogram is compared with previous frame histogram 2. Contiguous with limited memory → A new frame histogram is compared with previous family histogram

17. 3. Non contiguous with unlimited memory → A new frame histogram is compared with all previous family histograms within the same video. 4. Hybrid → First a new frame histogram is compared using the contiguous frames and then generated family histograms are merged using non contiguous case.

19.  Definition  Boundaries between collections of video micro- segments that are clearly identifiable organic parts of an event defining a structural (action) or thematic (story) unit  Video : collection of stories that may or may not be interconnected → Macro-Boundaries detection = Segmenting stories textual cues audio cuesvisual cues

20.  Unimodal segment detection  A video segment exhibits same characteristic over a period of time  Multimodal segment detection  A video segment exhibits a certain characteristic taking into account attributes from different modalities

21. Partition a continuous bitstream of audio data into non-overlapping segments Classification Seven mid-level audio categories Using low-level audio features Audio segmentation & classification Text transcript Extracted from either the closed captions or speech-to- text conversion Segmented and categorized with respect to a predefined topic list Frequency-of-word-occurrence metric is used

22. Pre-merging Steps Uniform segment detection Intra-modal segment clustering Attribute template determination Dominant attribute determination Template application Descent Methods Goal : Create macro- boundaries that are more accurate than the boundaries produced by individual modalities.

23. Text segment Audio segment Video segment

24. Single descent with intersecting union Single descent with intersection Single descent with secondary voting attributes Single descent with conditional union

25.  Definition  Boundaries between collections of macro- segments that exhibit different structural and feature consistency (e.g. different genres)  Example  Commercial detection method

26. Features that can aid in determining the location of the commercial break Triggers Features that can determine the boundaries of the commercial break Verifiers

27. Time interval between detected black frames as triggers Used as verifiers Letterbox change High cut rate(= low cut distance)

28. start

30. Type of boundariesMethodsExample Micro-boundariesFrame & Family histogram comparing and merging Visual scene segmentation Macro-boundariesSingle modality segmentation & Multimodal segmentation Multimodal story segmentation Mega-boundariesTrigger & VerifierCommercial detection

32. &