1. Representation and Analysis of Multimedia Content: The BOEMIE Proposal
Software & Knowledge Engineering Laboratory
Institute of Informatics & Telecommunications
NCSR “Demokritos”, Athens, Greece
Representation and Analysis of Multimedia Content: The BOEMIE Proposal
D.I. Kosmopoulos, V. Karkaletsis, S. Perantonis, G. Paliouras, C.D. Spyropoulos
LREC Workshop “Crossing media for improved information access”
Genoa, 23 May 2006

2. Contents Multimedia content analysis Existing approaches
BOEMIE proposal
Application scenario
Concluding remarks
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

3. Multimedia Content Analysis - I
Multimedia content grows with increasing rates
Hard to provide semantic indexing of multimedia content
Significant advances in automatic extraction of low-level features from visual content
Little progress in the identification of high- level semantic features
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

4. Multimedia Content Analysis - II
Inadequate the analysis of single modalities
Little progress in the effective combination of semantic features from different modalities.
Significant effort in producing ontologies for semantic webs.
Hard to build and maintain domain- specific multimedia ontologies.
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

5. Existing approaches - I
Combination of modalities may serve as a verification method, a method compensating for inaccuracies, or as an additional information source
Combination methods may be iterated allowing for incremental use of context
Major open issues in combination concern
the efficient utilization of prior knowledge,
the specification of open architecture for the integration of information from multiple sources, and
the use of inference tools
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

6. Existing approaches - II
Most of the extraction approaches are based on machine learning methods
With the advent of promising methodologies in multimedia ontology engineering
knowledge-based approaches are expected to gain in popularity and
be combined with the machine learning methods
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

7. Existing approaches – III
Use of Ontologies to “drive” the information extraction process
providing high-level semantic information that helps disambiguating the labels assigned to MM objects
Major open issues in building and maintaining MM ontologies concern
automatic mapping between low level audio-visual features and high level domain concepts,
automated population and enrichment from unconstrained content,
employing of ontology coordination techniques when multiple ontologies are present
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

8. Existing approaches - IV
Synergy between information extraction and ontology learning through a bootstrapping process
to improve both the conceptual model and the extraction system through iterative refinement
Applied so far in knowledge acquisition from textual content
bootstrapping starts with an information extraction system that uses a domain ontology, or
bootstrapping starts with a seed ontology, usually small
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

9. BOEMIE project
Bootstrapping Ontology Evolution with Multimedia Information Extraction
STRP, IST “Semantic-based Knowledge and Content Systems”
Started: 01/03/2006, Duration: 36 months
Consortium
Inst. of Informatics & Telecommunications, NCSR “Demokritos” (SKEL & CIL), Greece (Coordinator)
Fraunhofer Institute for Media Communication (NetMedia), Germany
Dip. di Informatica e Comunicazione, University of Milano (ISLab), Italy
Inst. of Telematics and Informatics CERTH (IPL), Greece
Hamburg University of Technology (STS), Germany
Tele Atlas, Belgium
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

10. BOEMIE proposal - I
Driven by domain-specific multimedia ontologies, BOEMIE systems will be able to identify high-level semantic features in image, video, audio and text and fuse these features for optimal extraction.
The ontologies will be continuously populated and enriched using the extracted semantic content.
This is a bootstrapping process, since the enriched ontologies will in turn be used to drive the multimedia information extraction system.
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

11. BOEMIE Proposal - II
EVOLVED
ONTOLOGY
INITIAL
POPULATION & ENRICHMENT
COORDINATION
INTERMEDIATE
ONTOLOGY EVOLUTION TOOLKIT
LEARNING TOLS
REASONING ENGINE
MATCHING TOOLS
ONTOLOGY MANAGEMENT TOOL
ONTOLOGY INITIALIZATION AND CONTENT MANAGEMENT TOOL
ONTOLOGY EVOLUTION
EVENTS
DATABASE
MAPS
MAP ANNOTATION INTERFACE
SEMANTICS EXTRACTION RESULTS
OTHER
ONTOLOGIES
SEMANTICS EXTRACTION
MULTIMEDIA CONTENT
SEMANTICS EXTRACTION TOOLKIT
TEXT EXTRACTION TOOLS
AUDIO EXTRACTION TOOLS
INFORMATION FUSION TOOLS
VISUAL EXTRACTION TOOLS
FROM VISUAL CONTENT
FROM NON-VISUAL CONTENT
FROM FUSED CONTENT
Content Collection (crawlers, spiders, etc.)
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

12. BOEMIE proposal - III Semantics extraction
Emphasis to visual content, from images and video, due to its richness and the difficulty of extracting useful information.
Non-visual content, audio/speech and text, will provide supportive evidence, to improve extraction precision.
Fusing information from multiple media sources is needed since
no single modality is powerful enough to encompass all aspects of the content and identify concepts precisely.
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

13. BOEMIE proposal - IV Multimedia Ontologies
development of a unifying representation, a “multimedia semantic model” to integrate:
a multimedia ontology which
describes the structure of multimedia content (content objects, such as a segment in a static image, a time window in audio, a video shot, ...),
describes visual characteristics of content objects in terms of low-level features (colour, shape, texture, motion, …)
a domain ontology which contains knowledge about the selected application domain, and
a geographic ontology which contains additional knowledge about the locations to be used
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

14. BOEMIE proposal – V Ontology evolution involves
ontology population and enrichment, i.e., addition of concepts, relations, properties and instances,
coordination of
homogeneous ontologies e.g. when more than one ontology for the same domain are available, and
heterogeneous ontologies, e.g., updating the links between a modified domain ontology and a multimedia descriptor ontology,
maintenance of semantic consistency
any of the above changes may generate inconsistencies in other parts of the same ontology, in the linked ontologies or in the annotated content base.
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

15. Application scenario - I
Enrichment of digital maps with semantic information
Domain: sport events in a given area (big cities)
Sub-domain initially selected: athletics (running, jumping and throwing events)
Cities will be selected taking into account: number and frequency of sports events, availability of multimedia coverage in English of these events, availability of map and landmark data for the city
BOEMIE will collect multimedia coverage for sport events and strive to extract as much knowledge from the extracted features as possible, using and evolving the corresponding domain ontologies
The identified entities and their properties, will be linked to geographical locations and stored in a content server
The user will be provided with immediate access to the annotated content
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

16. Application scenario - II
Functionality of the BOEMIE prototype
Ontology initialization
In the first phase, a domain ontology for the selected athletics domain will be created.
The concepts of this ontology will be populated using low-level multimedia features.
In the first phase, these features are extracted and linked to the ontology through manual content annotation. Suitable multimedia content for this purpose will be collected and stored in a multimedia repository.
The manual content annotation and ontology initialization will be performed using existing tools.
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

17. Application scenario - III
Functionality of the BOEMIE prototype: semantics extraction
The initial ontology will be used to train the classification and extraction components
Once the training is completed, BOEMIE system will operate automatically to extract meta data and features from MM content
Manual annotation will be used occasionally to train the system for new features
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

18. Application scenario - IV
Functionality of the BOEMIE prototype: semantics extraction, example of a fusion approach
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

19. Application scenario - V
Querying
The prototype will perform reasoning using knowledge from the domain ontology and geographical knowledge to deduce further information and answer user queries.
The user will be able to perform the following queries:
events in a time frame
events of a particular type
events at a certain location
persons related to events
events similar to a given one
events at nearby venues
points of interest near a venue
combinations of the above
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

20. Application scenario - VI
Querying: an example
Find out the location of the venues in which Athlete A has participated in a high jump competition in the city X.
From transcribed radio commentary, the BOEMIE system knows that in 2001, the World Championships in Athletics were held in city X in venue Y. From the geographical data, it knows the exact location of venue Y in city X.
It has further analyzed a video snippet and identified it as a high jump event. From the meta data of the video, the system knows its date of recording in 2001, and in the audio of this snippet, the keywords “X” and A's name were spotted.
Therefore, the system can deduce that A has indeed participated in a high jump competition in city X, namely the World Championships in Athletics 2001.
As a result, the BOEMIE system presents all used multimedia assets as “prove” for its answer and gives the exact location of the venue where the World Championship in Athletics took place.
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

21. Concluding remarks - I
BOEMIE work aims to initiate a discussion on the problem of knowledge acquisition and the synergy of information extraction and ontology evolution
Several open issues:
the role of ontology in fusing information from multiple media
ways to learn the optimal combination of features derived from MM content
how existing ontology languages can be extended to tackle the requirements of MM content analysis
the application of existing ontology learning and inference techniques in the context of MM content
the application of the coordination task in a new context which involves not only homogeneous ontologies, but also heterogeneous ones
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

22. Concluding remarks - II
The main measurable objective of BOEMIE initiative is to improve significantly the performance of existing single-modality approaches in terms of scalability and precision.
Towards that goal, our aim is to
develop a new methodology for extraction and evolution, using a rich multimedia semantic model, and
realize it as an open architecture that will be coupled with the appropriate set of tools.
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006

23. Representation and Analysis of Multimedia Content: The BOEMIE Proposal http://www.boemie.org
D.I. Kosmopoulos, V. Karkaletsis, S. Perantonis, G. Paliouras, C.D. Spyropoulos
THANK YOU !!!
BOEMIE proposal LREC Crossing Media Workshop, Genoa, 23/05/2006