1 Applications of video-content analysis and retrieval IEEE Multimedia Magazine 2002 JUL-SEP Reporter: 林浩棟

2 Outline Introduction Content-based Video Retrieval Applications Professional and educational applications Consumer domain application Conclusion

3 Introduction This is a survey of technologies and applications for video-content analysis and retrieval.

4 Content-based video retrieval (1/2) Video indexing should be analogous to text document indexing To facilitate fast and accurate content access to video data, we should segment a video document into shots and scenes We should extract keyframes or key sequences as index entries for scenes or stories.

5 Content-based video retrieval (2/2) The core research in content-based video retrieval is developing technologies to automatically parse video, audio, and text to identify meaningful composition structure and to extract and represent content attributes of any video sources.

6 Four processes involved by video- content analysis and indexing Feature extraction Structure analysis Abstraction Indexing

7 Four processes involved by video- content analysis and indexing

8 Feature Extraction The effectiveness of an indexing scheme depends on the effectiveness of attributes in content presentation An effective strategy in video-content analysis is to use attributes extractable from multimedia sources. Much valuable information is also carried in other media components, such as text, audio, and speech that accompany the pictorial component.

9 Four processes involved by video- content analysis and indexing

10 Four processes involved by video- content analysis and indexing Structure analysis The process of extracting temporal structural information of video sequences or programs Organizes video data according to their temporal structures and relations and thus build table of content Stories -> scenes -> shots -> frames

11 Four processes involved by video- content analysis and indexing Shots are a good choice as the basic unit for video-content indexing, and they provide the basis for constructing a video table of content.

12 Four processes involved by video- content analysis and indexing

13 Four processes involved by video- content analysis and indexing Video abstraction Video abstraction is the process of creating a presentation of visual information about a landscape or the structure of video, which should be much shorter than the original video. Example: baseball game Use an MPEG-7-compliant XML description format for the event segment

14 MPEG-7 Providing a standardized description of various multimedia descriptions of user preferences and usage history pertaining to multimedia information.

15 Applications We can broadly classify users into two extremes: Nontechnical consumer Trained, technical, professional corporate users who regularly use the products Professional and educational applications Consumer domain application

16 Professional and educational applications Automated authoring of Web content Searching and browsing large video archives Easy access to educational material Indexing and archiving multimedia presentation

17 Professional and educational applications Automated authoring of Web content Pictorial Transcripts AT&T DVL system

18 Professional and educational applications Searching and browsing large video archives Major news agencies and TV broadcasters own large archives of video that have been accumulated over many years. Intelligent video segmentation and sampling techniques can reduce the visual contents of the video program to a small number of static images.

19 Professional and educational applications Easy access to educational material The availability of large multimedia libraries that we can efficiently search has a strong impact on education.

20 Professional and educational applications Indexing and archiving multimedia presentation example

21 Consumer domain application The widest audience for video-content analysis is consumers Differences between large archives and consumer domain Video overview and access Video content filtering

22 Consumer domain application Video overview and access example

23 Consumer domain application Video content filtering example

24 Conclusion It ’ s important to distinguish between research activities, experiments, and real application that have made, or likely to make, the transition from research labs into the real world The targeted users are the ultimate judges of the technology ’ s usefulness in meeting their need. Users might have a tendency to resist new tools and methods.