1. D2 Conceration, Vilamoura, April 16th 2008 1 Video & Image Indexing and Retrieval in the Large Scale V T A L A S Progress Nozha Boujemaa, Scientific Coordinator

2. D2 Conceration, Vilamoura, April 16th 2008 2 Video & Image Indexing and Retrieval in the Large Scale Consortium Instrument: Integrated Project Start-End Date: 1 January 2007 - 31 December 2009 Project co-ordinator: ERCIM Scientific coordinator: INRIA Consortium: 12 partners from 7 countries ERCIM (FR), EADS (FR), CWI (NL), Fraunhofer (GE), INRIA (FR), ROBOTIKER (ES), Univ. of Sunderland (UK), CERTH-ITI (GR), Codeworks (UK), INA (FR), Belga (BE), IRT (GE) + AFP

3. D2 Conceration, Vilamoura, April 16th 2008 3 Video & Image Indexing and Retrieval in the Large Scale Objectives VITALAS is a use-case driven project that aims at providing a pre-industrial prototype system dedicated to intelligent access services to multimedia professional archives that would provide the consumer with new technological functionalities. VITALAS will provide advanced solution for indexing, searching and accessing large scale of non previously (or partly) annotated content. Cross-media (audio/speech, video, image, text) Content enrichment / indexing Interactive retrieval methods (query refinement with log files, RFB, context adaption) Vitalas system to be deployed by INA, Belga and AFP (V1 results)

4. D2 Conceration, Vilamoura, April 16th 2008 4 Video & Image Indexing and Retrieval in the Large Scale First year achievements User issues - WP1: Description of users requirements, Definition of VITALAS Use Cases, Elaboration and availability of test corpora Enabling technologies – WP2: Beyond state-of-the-art mono-media content description techniques (New local visual features [Rebai2007] [Joly2007] ; Well-trained models for word- and syllable-based recognition of German data) New probabilistic similarity search structure, 5 to 50 times faster than SoA techniques on large datasets. Integrated modules as web services +

5. D2 Conceration, Vilamoura, April 16th 2008 5 Video & Image Indexing and Retrieval in the Large Scale First year achievements Cross-Media indexing – WP3: Integration (web services) of partners know-how cross- media indexing tool (text and visual features; combination using SVM) Research activities on cross-media concept vocabularies: Concept dictionaries for cross-media indexing has been derived Cross-Media Retrieval –WP4: Query expressions to represent VITALAS use-cases Search log analysis of Belga image collection to refine query expression Coordination of and participation in INEX multimedia track

6. VITALAS Review, Paris, March 14th 2008 6 Vitalas System - V1 Architecture validation Services interfaces Exchange formats Hosting structure (ESB, Orchestrator) Evaluate & Validate a development and integration methodology of heterogeneous components Origin (Offline part : IRT, UoS, CERTH-ITI, INRIA, EADS Online part : INRIA, INA, EADS, Robotiker) Form (COTS, Open Source) Programming languages (Java, C++, …) Produce a first complete and running application Currently running, and being improved WebLab open source platform available on the web http://weblab-project.org/

7. VITALAS Review, Paris, March 14th 2008 7 Demonstration use case Cross media search : text + image + automatic metadata generation Dataset : 10000 images from Belga Offline part : Indexation Process –Image normalization and metadata extraction : IRT –Text concept extraction : UoS –Image concept extraction : CERTH-ITI –Text and Annotation Indexation : EADS –XML Repository : EADS –Image Indexation : INRIA

8. VITALAS Review, Paris, March 14th 2008 8 Online part : –Search engine GUI : Robotiker –XML Repository : EADS –Text or annotation retrieval : EADS –Image Search : INRIA –Cluster Map Visualisation : INA –Orchestration and Cross-medis Search : EADS Demonstration use case

9. VITALAS Review, Paris, March 14th 2008 9 Exploration Map X Files Subtask technical achievements Interactive exploration from visual similarities Task: exploring images with horses

10. VITALAS Review, Paris, March 14th 2008 10 10 QUERY: Unwanted results.... Just because KIA is part of the title/caption Remember: Belga Use-Case => Public Relations Manager of KIA: … I want to be alerted whenever the KIA logo is used…

11. VITALAS Review, Paris, March 14th 2008 11 11 MISSED RESULT Traditional Jouranlistic annotation no longer meets user requirements But extra-, post-editing of more conceptual metadata is expensive/time-intensive… Remember: Belga Use-Case => Public Relations Manager of KIA: … I want to be alerted whenever the KIA logo is used…

12. VITALAS Review, Paris, March 14th 2008 12 Whats next? Vitalas System V1+ & V2 V1+: (M18) (image and text) Large scale indexing and retrieval Full integration of Map Visualization tools V2: (M24) (audio and Video) Integration of Audio modules (speech to text) Integration of Video Mining and Search tools

13. VITALAS Review, Paris, March 14th 2008 13

14. VITALAS Review, Paris, March 14th 2008 14

15. VITALAS Review, Paris, March 14th 2008 15 Logical architecture SubmitQuery ESB ImageSearch INRIA::Maestro > TextAndAnnotationSearch > EADS::Lucene XMLRepository > EADS::WebLabRepo Robotiker::SearchGUI > Tomcat > EADS::FusedSearch INA:ClusterGUI >

16. VITALAS Review, Paris, March 14th 2008 16 Physical architecture Text / Annotation Search Image Search GUI ESB Fused Search Repository Image Server

17. VITALAS Review, Paris, March 14th 2008 17 Sample Process Fused search

18. D2 Conceration, Vilamoura, April 16th 2008 18 Video & Image Indexing and Retrieval in the Large Scale « Integrated Demo » « EADS, INRIA, ROBOTIKER, INA, UoS, CERTH-ITI, IRT »