M4 / September Integrating multimodal descriptions to index large video collections M4 meeting – Munich Nicolas Moënne-Loccoz, Bruno Janvier, Stéphane Marchand-Maillet and Eric Bruno Viper group Computer Vision & Multimedia Laboratory University of Geneva

M4 / September Outline i. Motivations ii. Modelling video documents iii. Indexing video documents iv. Retrieving video documents v. ViCoDE interface demonstration vi. Conclusion

M4 / September Motivations Video Documents End-User Indexing framework Querying Storing  Storing & Querying video documents  Integrate multimodal descriptions  Efficient retrieval algorithms ?

M4 / September Motivations  MPEG-7 multimedia documents model Standard  Weak ontology support  Native XML-Database storage  Document-centred  Inefficient cross documents multidimensionnal indexing

M4 / September  Generic Data Model  Relational DBMS  Raw Data Access  External Indexing & Retrieval Processor Indexing Framework OVAL MaxDB DBMS Indexing Processor Overview Data Model

M4 / September Document Model  Subset of MPEG-7 model MPEG-7 compliance (XML-enabled model)  Temporal Segments Central Data Unit (Data-centred model)  Ontology Capabilities W3C-Ontology Web Language compliant  Feature Space Indexing structures

M4 / September Document Model Description Structure Information DocumentMedia Stream Temporal Segment OntologyAnnotationDescriptorFeature Space Index Composed by Role Uses

M4 / September Document Storage  Videos Documents : TRECVid-2003, MPEG-7, IM2/M4 Corpus ~ 100 hours MPEG-1 Videos  Semantic Descriptions : TRECVid-2003 Annotations, Manual Annotations  Multimodal feature-based descriptions : Shot segmentation Global color & motion histogram Activity descriptors (Wavelet motion, Salient Regions) ASR histogram Audio descriptors

M4 / September Document Index  Document meta-data & structure  DBMS Access Facilities (SQL queries, Query optimization, Indexes)  Document raw-data  Object-based Video Access Library (C++, Java, Matlab) Exact and Random access to media stream raw data Plugins based (MPEG-1, MPEG-2)

M4 / September Document Index  Document descriptions  External Indexing Semantic Description  Use of ontology structure Feature Based Description  Multi-dimensional index structure ( Distance Matrix, Vantage Point Tree … )  Index processor : MATLAB Index file Index Processor Index Data Access Script Query Data

M4 / September Document Retrieval  Multiple feature spaces relevance feedback Set of positive & negative examples : Set of feature space distances : Look for the best linear combination of distances according to the user query: ?

M4 / September Document Retrieval Linear discriminant analysis  Ranked result :  Relevance feedback loop ?

M4 / September  Video Content Description & Exploration  Front-end prototype of the presented framework Java Server Page web content generation OVAL based on the fly generation of the Key-Frame Matlab indexing and retrieval algorithm ViCoDE demo ViCoDE

M4 / September Conclusion  Generic Multimedia Documents Management Framework Generic multimedia document model - MPEG-7 compliant Standard ontology support Generic Multimodal indexing & Retrieval engine  Integration of meeting features  Content based querying of meeting collections ? Questions ?