Supervised by Prof. LYU, Rung Tsong Michael Department of Computer Science & Engineering The Chinese University of Hong Kong Prepared by: Chan Pik Wah, Pat Ngai Cheuk Han, Table LYU0102 XML for Interoperable Digital Video Library

Outline Project Overview Project Overview Extraction Techniques Extraction Techniques Video Optical Character Recognition (VOCR) Video Optical Character Recognition (VOCR) Scene Change Detection Scene Change Detection Storage Storage XML XML Knowledge Enrichment Knowledge Enrichment Implementation Implementation Tasks in next semester Tasks in next semester

Motivations Rapid increase in the usage of multimedia information Rapid increase in the usage of multimedia information New approach: DIGITAL VIDEO LIBRARY New approach: DIGITAL VIDEO LIBRARY Project Outline

Motivations Little attention paying on video information extraction and storage Little attention paying on video information extraction and storage Scalability of the system in terms of adding new extraction components Scalability of the system in terms of adding new extraction components Lack of a generic framework for presentation and visualization of video information Lack of a generic framework for presentation and visualization of video information Project Outline

Targets Provide an open architecture that can integrate different digital video library functions Provide an open architecture that can integrate different digital video library functions Increase the reusability of the information extracted from videos Increase the reusability of the information extracted from videos Deliver and present the video to multiple computing platform Deliver and present the video to multiple computing platform Project Outline

Ways to achieve Modal concept of the digital video library functions Modal concept of the digital video library functions Collaborating the video information processing module Collaborating the video information processing module Using XML for storage Using XML for storage Universal format Universal format Flexible, scalable Flexible, scalable Present in different ways Present in different ways Easy to search based on particular tags Easy to search based on particular tags Generic framework for presentation and visualization of video information Generic framework for presentation and visualization of video information Project Outline

Overview of our project Project Outline

Achievements Implement two of the video information extraction techniques Implement two of the video information extraction techniques Video Optical Character Detection Video Optical Character Detection Scene Changes Detection Scene Changes Detection Store the extracted information as XML Store the extracted information as XML Build an XML editor in the tool for editing Build an XML editor in the tool for editing Do knowledge enrichment base on the information extracted Do knowledge enrichment base on the information extracted Project Outline

Extraction Techniques Text Detection Camera Motion Face Detection Scene Changes Word Relevance Audio Level Extraction Techniques

Video OCR for Digital News Help to locate topics by extracting the words in the captions Help to locate topics by extracting the words in the captions News captions provide vital search information of the video News captions provide vital search information of the video Video OCR results extracted the keywords on the frames Video OCR results extracted the keywords on the frames The results can be used together with the words extracted from the transcript for indexing The results can be used together with the words extracted from the transcript for indexing Extraction Techniques

Video OCR for Digital News Extraction Techniques

Scene change Detection Technique Detection Technique Effective method for segmenting a video sequence into significant components Effective method for segmenting a video sequence into significant components Extraction Techniques

Existing Method Image difference method Image difference method Histogram Difference Method Histogram Difference Method Histogram Difference Method using DC Coefficient Image Histogram Difference Method using DC Coefficient Image Our Algorithm & Implementation Our Algorithm & Implementation Histogram difference method with dynamic threshold Histogram difference method with dynamic threshold Extraction Techniques

Build and compared the histogram with the pervious scene Build and compared the histogram with the pervious scene Calculate the histogram difference Calculate the histogram difference If (total difference) > threshold If (total difference) > threshold => scene change Use the first frame as key frame Use the first frame as key frame Our Algorithm & Implementation Extraction Techniques

XML Extensible Markup Language Extensible Markup Language W3C W3C Create its own mark-up language for describing the contents Create its own mark-up language for describing the contents Storage

Advantages of using XML Platform and system independent Platform and system independent Create your own tag Create your own tag Adopt Unicode Adopt Unicode Universal format Universal format Scalable Scalable Storage

XML schema Storage

XML Parser A parser is an interface between an XML document and the application program A parser is an interface between an XML document and the application program Document Object Model (DOM) Document Object Model (DOM) Storage

How to present XML Tree model becomes very similar to an XML schema Tree model becomes very similar to an XML schema Represented as nodes that show element/attribute names or the text content and their relative places within the XML Represented as nodes that show element/attribute names or the text content and their relative places within the XML Storage

Content creation in digital video library Collaborating different video information extraction techniques, mainly Collaborating different video information extraction techniques, mainly Knowledge Cross-referencing Knowledge Cross-referencing Knowledge Enrichment Knowledge Enrichment Access to video by content Access to video by content Communicate information trends across time, space Communicate information trends across time, space Provide fast and effective searching Provide fast and effective searching Storage

Knowledge Enrichment Geographic information Geographic information Extract geographic names of countries and cities from text recognized from the video OCR or speech recognition Extract geographic names of countries and cities from text recognized from the video OCR or speech recognition Knowledge from geographic naming database enrich the information Knowledge from geographic naming database enrich the information Allow query or browse for events at a particular location or within some “ distance ” of that location Allow query or browse for events at a particular location or within some “ distance ” of that location Storage

Our Implementation Use a known set of places along with their spatial coordinates and some additional information for knowledge enrichment Use a known set of places along with their spatial coordinates and some additional information for knowledge enrichment Use the XML file as the source material to be processed Use the XML file as the source material to be processed Try to extract names of major cities by processing the text in the source Try to extract names of major cities by processing the text in the source Storage

Geographic naming database An XML file with the following format is used An XML file with the following format is used For each city: For each city: City ID City ID Name of city Name of city Name of country Name of country Longitude Longitude Latitude Latitude Storage

The updated XML file Storage

Knowledge enrichment component in our tool Extract and list out all the cities mentioned in the video Extract and list out all the cities mentioned in the video Allow user to select any of them to look for further information of that city Allow user to select any of them to look for further information of that city Storage

Program Platform Microsoft Visual C++ ® Microsoft Visual C++ ® Object-Oriented Object-Oriented Faster MFC applications Faster MFC applications Composite Controls  ActiveX Composite Controls  ActiveX Microsoft ® DirectShow ® Microsoft ® DirectShow ® Component object Model (COM) Component object Model (COM) High-quality capture and playback of multimedia streams High-quality capture and playback of multimedia streams Implementation

Video Player Implementation

Control filter graph manager filter graph manager Dialog Box create with the class CFormView Dialog Box create with the class CFormView Implementation

Scene Change & VOCD CScrollView CMenu Add the extracted information to XML Implementation

XML Editor TreeView TreeView XML read by parser XML read by parser Tag in XML Tag in XML Node in Tree Node in Tree Implementation

Knowledge Enrichment Dialog Box create as class CFormView Dialog Box create as class CFormView Read the database Read the database Compare with the XML generated Compare with the XML generated Implementation

Problems & Solutions Implementation

Problems & Solutions Multi-modal tool --- Integrate all the Component Multi-modal tool --- Integrate all the Component Docking Window is used Docking Window is used Flexible & Efficient to add different new modals Flexible & Efficient to add different new modals CSizingControlBar CSizingControlBar Implementation

Problems & Solutions Implementation

Demo

Tasks in Next Semester Focus on using XML to do multimedia presentation Focus on using XML to do multimedia presentation Style Sheet (XSLT) not suitable for multimedia document generation Style Sheet (XSLT) not suitable for multimedia document generation New format on multimedia presentation New format on multimedia presentation SMIL SMIL Future

Tasks in Next Semester Time-based multimedia content Time-based multimedia content Capable to synchronize the playback of all multimedia elements Capable to synchronize the playback of all multimedia elements Transform XML generated into SMIL format for presentation Transform XML generated into SMIL format for presentation Design a style sheet suitable for multimedia document generation Design a style sheet suitable for multimedia document generation Future

Q & A