Supervised by Prof. LYU, Rung Tsong Michael Department of Computer Science & Engineering The Chinese University of Hong Kong Prepared by: Chan Pik Wah, Pat Ngai Cheuk Han, Table LYU0102 XML for Interoperable Digital Video Library

Outline Introduction to XVIP Review of Project Extraction Techniques Face Detection Speech Recognition Presentation XSL SMIL Transformation

Motivations Rapid increase in the usage of multimedia information New approach: DIGITAL VIDEO LIBRARY Project Outline

Motivations Little attention paying on video information extraction and storage Scalability of the system in terms of adding new extraction components Lack of a generic framework for presentation and visualization of video information Project Outline

Overview of XVIP Project Outline

Achievements in last semester Project Outline

Achievements in this semester Project Outline

Extraction Techniques Text Detection Camera Motion Face Detection Scene Changes Word Relevance Audio Level Extraction Techniques

Face Detection Object-presence detections are also an important technique. Identify and index features to support image similarity matching. Face detection is a good example Extraction Techniques

Face Detection Name of people appearing in the video How they are interacting with the environment More searchable Extraction Techniques

Face Detection Neural Network-Based Algorithm The basic algorithm used for face detection Extraction Techniques

Face Detection Face Recognition Facial Expression Analysis Enrich the XML Easier for user to search the content of video Extraction Techniques

Speech Recognition  Speech recognition technology can make any spoken data useful for library indexing and retrieval Extraction Techniques

Speech Recognition Engine Extraction Techniques

Speech Recognition ViaVoice Error rate > 50% Extraction Techniques

Presentation of the video XML to SMIL Transformation SMIL XSL Transformation process SMIL

SMIL stands for Synchronized Multimedia Integration Language is currently a W3C Recommendation. It is a markup language that can synchronize and integrate multimedia. It enables authors to specify when and what should be presented. SMIL

Advantages SMIL is text-based Easy to develop with a text editor Generate customized presentations Due to text-based, can generate customized SMIL file for each visitor based on preferences recorded in the visitor's browser SMIL effort is led by the W3C W3C tries to shape a specification that is beneficial to all parties involved. Avoid using container formats. SMIL can stream many media formats, need not to merge clips into a single streaming file. SMIL

Document Structure Quite similar to the structure of HTML document. SMIL has the,, and elements. SMIL is case-sensitive, all the tags should be written in lower case. SMIL is XML-based, so all the tags should be ended properly. A SMIL document example: … Here is the head … … Here is the body … SMIL

Head It normally includes the,, and element as the children of. … …… SMIL

Body It allows user to design when and what to be displayed in the regions defined in the head of the document. The media object elements allow the inclusion of different media objects into a SMIL presentation. Media objects are included by reference (using a URI). Different types of media object(animation, audio, img, video, text or textstream) is reflected in the element name. SMIL

Timing and Synchronization Sequence Example: Parallel Example: … SMIL

Players and Browsers RealPlayer RealNetworks’ RealPlayer is one of the best-equipped Web multimedia players around. The G2 version of RealPlayer started supporting SMIL. SMIL for RealPlayer is the most dominant use of SMIL nowadays. QuickTime GRiNS Internet Explorer Approaching SMIL from the browser side SMIL

XSL Stands for “Extensible Stylesheet Language” XSL is the language defined by the W3C to add formatting information to XML data. Stylesheets allow data to be formatted based on the structure of the data, so one stylesheet can be used with many similar XML documents. XSL

XSL consists of two different standards: Transformation language (XSLT) A common language for transforming one XML document into another Defines how to create a result tree from a source tree. Formatting objects (FO) specifications for how to display document content currently no XSL-FO processing applications in common use. XSL

Working Principles XSL

XSLT Processors Xalan both Xalan C++ (using C++) and Xalan-J (using Java) available Saxon runs on Java MSXML Can be used in VB, VC++, scripts XSL

Transformation Process Input files: XML file generated after doing video information extraction (e.g. speech recognition, scene change detection, VOCD …) Data files in XML format for the additional information (e.g. details of major cities, famous people) Transformation

Transformation Process Output files for SMIL presentation: a.smil file for layout and synchronization of the whole presenetation some.rt files for realtext (e.g. displaying transcript of the video, details of major cities and famous people) Transformation

Design 1 Write the whole transformer with our own code. Details: Use VC++ to read all the input files, get the information and combine them. Create the output the files for the SMIL presentation, simply using fprintf in VC++. Transformation

Design 1 Disadvantage: The layout and design of the SMIL presentation needed to be hard-coded in the VC++ program, make it program-dependent. 1. The layout and design of the presentation is inconvenient to be changed by user. 2. The transformer is hard to extend as whenever a new module is added, a lot modification on the program is needed. Transformation

Design 1 with modification Provide an additional file or interface as a template for user to define the layout or design the SMIL presentation. Disadvantage: The flexibility provided is still limited. Not a standard way to define a template. Transformation

Design 2 Use XSLT processor assisting the XML to SMIL transformation. User can define their own template with XSL. Advantage: Program-independent Extensible Standard templates Transformation

Transformation Process Two Parts: 1. Knowledge Enrichment - Combine the data files of additional information with the original XML file generated after video information extraction 2. Create output files - Use the enriched result to create the output files for the SMIL presentation using XSL. Transformation

Knowledge Enrichment Do combination between the two XML files Since XSLT cannot read 2 input files and it is not target for comparing and combining files. We have to implement this part with our own code. (Combination) City-info. xml / Peop-info. xml Xml after video information extraction Combined- city.xml Transformation

Combined XML This XML file contains the details of the additional information that will be used in the SMIL presentation. 香港 中國南部一個沿海城市 China 紐約 隸屬美國紐約州的城市 America Transformation

Create output files For different realtext files Combine-city.xmlMap-detail.xsl (XSLT) Map-detail.rtXML after video information extraction Transcript.xsl (XSLT) Transcript.rt Transformation

Create output files For the SMIL file It contains layout and content to be displayed in different regions An XSL file defining the presentation layout is required Different regions need different XML data file storing the relative information Different regions have different timing and synchronization order for each child object Challenge Need to combine all the above information together, with proper layout for the presentation and correct displaying order for each child object in different media. Transformation

Create output files Limitation of XSL: It can only read one input data file and one XSL file, and generate one output. It cannot do combination between files. Solution: We use separate XSL files to generate the layout of the whole presentation and the displaying order for the required regions. Then, combining the generated result together with our own code. Transformation

Create output files Solution Combine- city.xml Mapdisplay.xsl (XSLT) Map- display. txt Xml after VIP Layout.xsl (XSLT) Layout. txt (Combination).Smil Combine- peop.xml Peopdisplay. Xsl (XSLT) Peopdisplay. txt …… Transformation

THE END