Information Systems & Semantic Web University of Koblenz Landau, Germany Semantic Multimedia Web Ansgar Scherp Basierend auf Folien von Carsten Saathoff, Raphael Troncy und Lynda Hardman

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 2 Was bisher geschah... MMDB als Erweiterung von ORDBMS Information Retrieval als Basis für Queries Feature Extraktion um Inhalt zu beschreiben Feature Transformation um kompaktere Darstellung zu bekommen Fokus auf low-level features Distanzen und Ähnlichkeiten Indizierung von Features für schnellen Zugriff

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 3 Probleme traditioneller MMDB Datenstrukturen und Schemata meist proprietär MMDB typischerweise für eine Applikation aufgesetzt. Spätere Integration mit anderen Applikationen schwer Ad-Hoc Integration eher unmöglich Starker Fokus auf Low-Level Features Semantische Lücke Kein direktes Mapping zwischen Low-Level Features und Semantik des Bildes Retrieval primär über Ähnlichkeit Fast alle Studien zeigen, dass Nutzer dadurch nicht zufriedengestellt werden Nutzer wollen semantisch Anfragen

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 4 Metadaten

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 5 Metadaten (2) Stichworte GPS Information Kamera Daten Datum

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 6 Metadaten Daten über Daten Autor, Creation-Date, Keywords,... Wie repräsentieren? Relationales Schema XML Semantic Web Technologien Metadaten sollten interoperabel sein Web Desktop Intranets Viele Applikation müssen kommunizieren

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 7 Überblick Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM Probleme mit MPEG7 Core Ontology on Multimedia (COMM) Linked Open Data KAT – K-Space Annotation Tool Semi-Automatische Effiziente Annotation Szenarien

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 8 Semantic Web auf einer Folie Employee PostDoc Professor Person rdfs:subClass cooperatesWith rdfs:Range rdfs:Domain Ontology Steffen Staab... rdf:type Meta- data Siegfried Handschuh... Web page URL swrc:cooperatesWith

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 9 Semantic Web for Multimedia Bad Kreuznach 2007 WIAMIS 2008 in Klagenfurt Multimedia Ontology depicts Carsten Saathoff Domain Ontology hasName rdf:type Researcher ResearchMeeting:= >=1 depicts.Researcher ResearchMeeting Zeig mir alle Bilder von Research Meetings!

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 10 Überblick Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM Probleme mit MPEG7 Core Ontology on Multimedia (COMM) Linked Open Data KAT – K-Space Annotation Tool Semi-Automatische Effiziente Annotation Szenarien

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 11 Semantische Lücke 0010EE -> bläulich 0033FE -> bläulich Visuell ähnlich! Aber semantisch unterschiedlich!

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 12 Semantische Lücke Visuell ähnlich, semantisch ähnlich, aber... Italien USA

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 13 Ebenen der Semantik Generische Objekte Generische Szene Spezifische Objekte Spezifische Szene Abstrakte Objekte Abstrakte Szene Eine Person Personen unterhalten sich Churchill Churchill, Roosevelt, Stalin sitzen zusammen Churchill, Premierminister, GB,... Big Three, Yalta Konferenz WWII,... Wissen

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 14 Überblick Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM Probleme mit MPEG7 Core Ontology on Multimedia (COMM) Linked Open Data KAT – K-Space Annotation Tool Semi-Automatische Effiziente Annotation Szenarien

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 15 Overview of Canonical Processes

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 16 Example 1: CeWe Color PhotoBook Application for authoring digital photo books Automatic selection, sorting and ordering of photos Context analysis methods: timestamp, annotation, etc. Content analysis methods: color histograms, edge detection, etc. Customized layout and background Print by the European leader photo finisher company

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 17 CeWe Color PhotoBook Processes My winter ski holidays with my friends

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 18 CeWe Color PhotoBook Processes

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 19 CeWe Color PhotoBook Processes

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 20 CeWe Color PhotoBook Processes

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 21 CeWe Color PhotoBook Processes

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 22 CeWe Color PhotoBook Processes

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 23 Example 2: Vox Populi Video Sequences Generation Stefano Bocconi, Frank Nack Interview with America video footage with interviews and background material about the opinion of American people after Example question: What do you think of the war in Afghanistan? I am never a fan of military action, in the big picture I dont think it is ever a good thing, but I think there are circumstances in which I certainly cant think of a more effective way to counter this sort of thing…

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 24 Vox Populi Premeditate Process Analogous to the pre-production process in the film industry Static versus dynamic video artifact Output Script, planning of the videos to be captured Questions to the interviewee prepared Profiles of the people interviewed: education, age, gender, race Locations where the interviews take place

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 25 Vox Populi Annotations Contextual Interviewee (social), locations Descriptive Question asked and transcription of the answers Filmic continuity, examples: gaze direction of speaker (left, centre, right) framing (close-up, medium shot, long shot) Rhetorical Rhetorical Statement Argumentation model: Toulmin model

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 26 Vox Populi Statement Annotations Statement formally annotated: E.g. war best solution A thesaurus containing: Terms on the topics discussed (155) Relations between terms: similar (72), opposite (108), generalization (10), specialization (10) E.g. war opposite diplomacy

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 27 Toulmin Model ClaimData Qualifier WarrantBackingConditionConcession 57 Claims, 16 Data, 4 Concessions, 3 Warrants, 1 Condition

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 28 Vox Populi Query Interface

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 29 Vox Populi Organize Process Using the thesaurus, create a graph of related statements nodes are the statements (corresponding to video segments)war best solution,diplomacy best solution,war not solution edges are either support or contradict support contradict war best solution war not solution diplomacy best solution

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 30 Result of Vox Populi Query I am not a fan of military actions War has never solved anything I cannot think of a more effective solution Two billions dollar bombs on tents

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 31 Vox Populi Processes

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 32 Canonical Processes 101 Canonical: reduced to the simplest and most significant form possible without loss of generality Formalization of each process in UML diagrams Process Process artifacts Process actors External world artifacts

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 33 Premeditate Process where initial ideas about media production are established Design a photo book of my last holidays for my family Create argument-based sequences of videos of interviews after September 11

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 34 Create Media Asset Process where media assets are captured, generated or transformed

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 35 Annotate Process where annotation is created

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 36 Semantic Annotate The annotation uses some controlled vocabularies Subject matter annotations of your photos Rhetorical annotations in Vox Populi

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 37 Package Process where process artifacts are logically and physically packed

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 38 Query Process where a user retrieves a set of process artifacts based on a given query

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 39 Construct Message Process where an author specifies the message they wish to convey Our holiday was sporty, great weather and fun Create clash about whether war is a good thing

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 40 Organize Process where process artifacts are organized according to the message Organize a number of 2-page layouts in photobook Use semantic graph to select related video clips to form linear presentation of parts of argument structure

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 41 Publish Process where final content and user interface is created

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 42 Distribute Process where final interaction between end-users and produced media occurs

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 43 Canonical Processes Possible Flow

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 44 Sum Up Community agreement, not yet another model Large proportion of the functionality provided by multimedia applications can be described in terms of this model Initial step towards the definition of open web-based data structures for describing and sharing semantically annotated media assets

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 45 Überblick Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM Probleme mit MPEG7 Core Ontology on Multimedia (COMM) Linked Open Data KAT – K-Space Annotation Tool Semi-Automatische Effiziente Annotation Szenarien

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 46 MPEG7 ISO Standard der MPEG Community Einheitliches Format zu Speicherung von Multimedia Metadaten Struktur Features Semantik Extrem (!) umfangreich Darauf basierend MPEG21 mit Fokus auf DRM etc. Hat im Gegensatz zu MPEG1-4 nichts mit Kodierung zu tun

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 47 MPEG7 (2) Basiert auf XML MPEG7 Beschreibung ist eine Hierarchie Deskriptoren beschreiben Eigenschaften von Multimedia Daten Struktur Video -> Shots -> Frames Bilder -> Segmente Semantik Low-Level Features Um Flexibilität zu gewahren, können Deskriptoren sehr vielseitig kombiniert werden.

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 48 Big Three

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 49 Issues Churchill Roosevelt Stalin... How do you formulate a query to get images showing Churchill et al.? First Shot (Xpath): //StillRegion[.//Keyword=Churchill or.//Keyword=Roosevelt or.//Keyword=Stalin] Winston Churchill Recognizer Franklin Roosevelt Recognizer Josef Stalin Recognizer

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 50 Probleme mit MPEG-7? Annotationen sind nicht interoperabel! Mehrdeutigkeiten Mehrere Möglichkeiten um semantisch identische Annotationen zu beschreiben Deskriptoren können auf viele Arten kombiniert werden Komplexe Anfragen müssen alle Alternativen beachten

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 51 Capabilities and Maturity Levels Integration Automation Former SituationCurrent SituationFuture / Desired Situation no standard, no vocabulary manual 1:1 agreement on format and semantics tight coupling of data and applications standard vocabulary manual 1:1 agreement on mpeg-7 vocabulary tight coupling of data and applications standard vocabulary pre-defined meaning ad-hoc coupling of data and applications CORE ONTOLOGY Nächster Teil der VL Formerly MPEG-7 COMM

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 52 Ontology Stack Core Ontologies Domain Ontologies Foundational Ontologies Span across multiple fields, each covering multiple domains Modelling of the most abstract concepts like event, object,... Core Ontologies Situated in one field, but spans multiple domains Can base on foundational ontologies Examples fields: events, annotation, communication,... Domain Ontology For a specific domain, e.g., fishery, human body, etc.

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 53 MPEG-7 COMM Requirements on a high quality MM Ontology Challenge Building Block Legend

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 54 Text Descriptor Music Manager Compound Document Requirements for COMM Reusability Design a core ontology for any multimedia related application MPEG-7-Compliance Support most important description tools Extensibility Enable inclusion of further description tools (even those that are not part of MPEG-7!) media types Separation of Concerns Clear separation of domain knowledge and knowledge about structure Modularity Enable customization of multimedia ontology High degree of axiomatization Ensure interoperability through machine accessible semantics Churchill Recognizer Josef Stalin Recognizer Face Detector Photo Manager Authoring Tool Semantic Annotation decomposition visual descriptors audio descriptors

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 55 Is MPEG-7 a good Basis for a high Quality Ontology? Shortcomings of badly modelled ontologies [Oberle et al., 2006]: 1)Conceptual ambiguity Difficulties in understanding the meaning of concepts and their relations 2)Poor axiomatization Axiomatization of well defined concepts is missing 3)Loose Design Presence of modelling artefacts (concepts without ontological meaning) Shortcomings mainly hinder Extensibility Interoperability Especially 1) and 2) are major shortcomings of MPEG-7 1-to-1 translations from MPEG-7 to OWL/RDFS (e.g. [Hunter, 2003a]) will not result in high quality ontologies! Churchill Roosevelt Stalin

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 56 MPEG-7 COMM Requirements on a high quality MM Ontology Quality of Ontologies Challenge Building Block Legend

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 57 How to Design a High Quality Multimedia Ontology? Approach from [Oberle, 2005], [Oberle et al., 2006]: Use a well designed foundational ontology as a modelling basis to avoid shortcomings Foundational ontologies provide Formal precision Domain independence Broad scope Building upon foundational ontologies prevents easy inclusion of modeling artefacts reduces conceptual ambiguity inherit rich axiomatization

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 58 Methodology MPEG-7 COMM Requirements: High Quality MM Ontology Quality of Ontologies Quality Measures for Ontologies Reference Ontologie MPEG-7 Compliance Challenge Building Block Legend

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 59 Methodology for Design Pattern Definition Identification of most important MPEG-7 functionalities [Arndt et al., 2007] : Decomposition of multimedia content into segments Annotation of segments with meta data (e.g. visual descriptor, media information, creation & production, …) General: Identify repetitive structures and describe them at an abstract level Describe digital data by digital data at an arbitrary level of granularity Additional patterns are needed for: Complex data types of MPEG-7 Semantic annotation by using domain ontologies Interface between reusable multimedia core and domain specific knowledge

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 60 DOLCE Design Patterns: OIO Foundational ontology DOLCE+DnS Ultralight Aims at capturing the most essential aspects in the world Defines disjunctive upper classes Event, Object, Quality and Abstract Follows a pattern-oriented approach for ontology design 2 design patterns (extensions) that are especially important for MPEG-7: Ontology of Information objects (OIO): Formalization of information exchange Information object represents pure abstract information (message) Relevance for multimedia ontology: MPEG-7 describes digital data (multimedia information objects) with digital data (annotation) Digital data entities are information objects

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 61 DOLCE Design Patterns: D&S Descriptions & Situations (D&S): Formalization of Context Relevance for multimedia ontology: Meaning of digital data depends on context Digital data entities are connected through computational situations (e.g. input and output data of an algorithm) Algorithms are descriptions Annotations and decompositions are situations that satisfy the rules of an algorithm / method

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 62 Methodology MPEG-7 COMM Requirements: High Quality MM Ontology Quality of Ontologies Quality Measures for Ontologies Reference Ontologie Identification of repetitive structures MPEG-7 Compliance Pattern definition through Specialization Challenge Building Block Legend Repr. of Context Repr. of Information

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 63 Ontology of Information Objects (OIO)

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 64 Example Information Object Graz Tourist Guide Information Realization Booklet Information Encoding: English German About: Places, Buildings (e.g. Clock Tower) Agent: 1. iMedia Visitor / 2. Tourist Officer / 3. Graphics Designer Expresses: 1.Walking Path through Graz 2.Small-Size Tourist Guide 3.Arrangement of Illustrations

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 65 Descriptions & Situations (D&S) Distinction between: DOLCE ground entities (regions, endurants, perdurants) Descriptive entities (parameters, roles, courses) Descriptions Formalize context Define descriptive concepts Situations Are explained by descriptions Are settings for ground entities

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 66 Putting it Together: Decomposition Pattern

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 67 Example Image1 playsRole SegmentationInput Segment2 playsRole SegmOutp Segment4 playsRole SegmOutp Segment1 playsRole SegmOutp Segment3 playsRole SegmOutp Segment1 playsRole SegmInput Via its role in a computational task the different parts may be arbitrarily nested and related to different computing algorithms Querying for all subparts takes place along a well-defined pattern

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 68 Modular Architecture Multimedia ontology consists of Core module that contains the design patterns Modules that specialize the core module for different media types Modules that contain media independent MPEG-7 description tools such as media information or creation & production Data type module that formalizes MPEG-7 data types e.g. matrices, vectors, unsigned- int-5, float-vector, probability- vector, …

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 69 Does the Multimedia Ontology fulfil the Requirements? Reusability MPEG-7-Compliance Design patterns enable the representation of description tools Extensibility Design patterns are media independent possibility to include further media types arbitrary descriptors Extensions of multimedia ontology will not affect legacy annotations due to DOLCE+D&S+OIO Separation of Concerns Clear separation between domain specific and multimedia related knowledge Modularity Modular architecture allows customization High degree of axiomatization Design patterns come with generic axiomatization that is refined in derived ontology modules Churchill Recognizer Josef Stalin Recognizer Photo Manager COMM One such extension has already been done for Text Annotation. Another one for compund document annotation is currently developed! Content & Media Annotation Pattern Semantic Annotation Pattern See slide before this slide! OWL-DL version available for download.

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 70 Benefits of a DOLCE-aligned Multimedia Ontology Usage of DOLCE enforces clean design Constraints prohibit arbitrary placement of MPEG-7 concepts into DOLCE Similar concepts will be placed on similar locations of the taxonomy Things that are different, have to be separated (e.g. data and the perceivable content that is carried) Extensibility due to underlying general taxonomy of DOLCE Possibility to describe multimedia domain at an arbitrary level of detail (e.g. segments have pixels as atomic parts) Rigorous application of the D&S and OIO patterns allows description of digital data in different contexts (e.g. data acting as input or output for an algorithm)

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 71 Benefits compared to MPEG-7 Linkage with domain ontologies allows meaningful semantic annotation of multimedia content Semantic part can be entirely replaced with a domain ontology Clear separation between domain ontologies and multimedia core ontology through semantic annotation pattern Easier queries Annotation pattern guarantees equal representation of all annotations Complex data type pattern guarantees uniform access to nested data No complex XML-structures to parse Multimedia ontology only uses restricted inventory of DOLCE predicates Higher interoperability through machine accessible semantics and underlying DOLCE axiomatization

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 72 Yalta_Conference World War II Yalta... History Ontology Creating a Multimedia Presentation Revisited SR1 SR2 SR3 Winston Churchill Recognizer Franklin D. Roosevelt Recognizer Josef Stalin Recognizer Churchill Roosevelt Stalin Sparql: select ?image where { ?image plays AnnotatedDataRole. ?x plays SemanticLabelRole. ?x rdf:type pol:President } Photo Manager Authoring Tool

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 73 Überblick Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM Probleme mit MPEG7 Core Ontology on Multimedia (COMM) Linked Open Data KAT – K-Space Annotation Tool Semi-Automatische Effiziente Annotation Szenarien

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 74 A Giant Graph Open to the World Annotate the content (interpretation) Elephant, Ganesh, Thailande, Holidays, Chiang Mai Link to knowledge on the Web :img foaf:depicts dbpedia:Ganesh dbpedia:Ganesh rdfs:label "Vinayaka" dbpedia:Ganesh skos:altlabel "Ganapati" dbpedia:Ganesh rdf:type wn:synset-Deities-noun-1 dbpedia:Ganesh owl:sameas wn:synset-Ganesh-noun-1 An image of the Elephant Ganesh Raphaël Troncy

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 75 Linking Open Data Project Expose open datasets in RDF Set RDF links among the data items for different datasets Over 2 billion triples, 3 millions links (March 2008)

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 76 Linked Open Data March 2009

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 77 Warum ist LOD wichtig? RDFa wird von der Google Suchmaschine verarbeitet Content provider bieten nun RDFa an Erhöht Click-trough-rate auf Webseiten (Werbeanzeigen) Erhöht Ranking der Webseiten in Google SIOC Ontology zur Verlinkung von Online Communities wird genutzt von Yahoo! Wird durch SearchMonkey eingesammelt Tools um RDFa zu publishen Usw.

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 78 DBpedia DBpedia is a community effort to: extract structured "infobox" information from Wikipedia interlink DBpedia with other datasets on the Web

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 79 DBpedia

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 80 Automatic Links Among Open Datasets Processors can switch automatically from one to the other …

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 81 Take Home Message Reuse what is there Of course, one could create RDF data manually … … but that is unrealistic on a large scale Goal is to generate RDF data automatically when possible and "fill in" by hand only when necessary service to get RDF from flickr images service to get RDF from XMP bin/blosxom.cgi/WorkRelated/SemanticWeb/xmpextract.ht ml bin/blosxom.cgi/WorkRelated/SemanticWeb/xmpextract.ht ml Expose what you make

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 82 Überblick Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM Probleme mit MPEG7 Core Ontology on Multimedia (COMM) Linked Open Data KAT – K-Space Annotation Tool Semi-Automatische Effiziente Annotation Szenarien

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 83 KAT: K-Space Annotation Tool Goal Efficient annotation of multimedia content Means to create semantically rich annotations KAT provides framework for Executing analysis plugins Providing visualisation plugins Displaying/annotating content Browsing Interfaces with Core Ontology for Multimedia (COMM) Provides the common model Role based messaging to leverage reuse of components

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 84 Efficient Annotation Reduce time required by user for annotating content Integration of Automatic analysis methods Region labeling, object detection Key Frame Extraction, Shot Boundary Detection Automatic Organisation Clustering Inferencing Based on formal domain ontologies Semi-automation

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 85 Semantically Rich Annotations Relational Annotation Express how depicted entities are related Example: Soccer Game Who is tackling whom? Why was the penalty given? Ontologies provide means to express relations KAT aims at providing the means to efficiently create them Event-Based annotation Events are prominent in multimedia Create and manage events Relate events and media Allow for event-based retrieval and exploration

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 86 Architecture KAT-Core Plugin GUI Plugin View register COMM Repository... store display

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 87 Szenarien Effiziente Annotation von persönlichen Bildern Semi-Automatische, Semantische Annotation von Sport- Ereignissen Browsing von Bildsammlungen im Web

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 88 Persönliche Bildsammlungen Heutzutage typischerweise in Ordnern auf Festplatte Wenig Annotationen weil zu aufwendig Annotationen enthalten Viel Hintergrund Wissen Gefühle und persönliche Momente Frage: Wie kann ein Nutzer hier unterstützt werden? Automatische Annotation liefert nur einfache Semantik Daher: Clustering um Ereignisse zu finden User kann dann ganze Ereignisse annotieren Verwendung von NLP (Textanalyse) um semantische Annotation zu erzeugen

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 89 Sportereignisse Kombination verschiedener Algorithmen Highlight Detection: Goals, Corner-Shot,... Features: Motion, Geräusche, sichtbare Konzepte Analyse von Minute-by-Minute Reports Liefert andere Ereignisse Ergebnisse oft nur global Manuelles Refinement Zuweisen von Namen zu Spielern im Video/Bild Zuweisen von Rollen: Wer hat das Faul begangen, wer war Opfer Verknüpfen von Ereignissen Ziel: möglichst vollständige Annotationen effizient erstellen

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 90 Browsen von Bildkollektionen Flickr als COMM Maping von Tags in Wordnet Wordnet: linguistische Ontologie Mapping von Geoinformationen nach Geonames Ontologie von geographischen Informationen (Länder, Orte,...) Mapping nach dbpedia Wikipedia als maschienenlesbare Version Tags bekommen Kontext Anzeige werwandter Bilder, komplexe Queries Ergänzen der Annotationen

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide KAT as basis for der User Interface Objectives Explore and visualize semantic Web 2.0 data in real-time Acquainting oneself about an area of interest Semantic data 1 billion triples from DBpedia, GeoNames, WordNet, FOAF files and Flickr Very large, mixed-quality, semantically heterogeneous Winner of Billion Triples Track, Semantic Web Conference 2008, Karlsruhe [ISWC2008]

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide SemaPlorer – Web 2.0 Content Brower Search for locations, persons and tags Active facets like tags, location Map showing locations, sights, pictures Geo-referenced image from Flickr Information on locations, persons, tags SemaPlorer Live!

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 93 Werbeblock HiWi-Jobs für SemaPlorer++ Interesse an Arbeit in einer Gruppe Tätigkeit die Kenntnisse aus dem Studium (und darüber hinaus) praktischen anwenden lässt Spaß am Tütfteln Entwicklung mit Java Mail mit Beschreibung an Erfahrungen / Lebenslauf an

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 94 Werbeblock ImageAtlas II Projektpraktikum Entwicklung einer Plattform zum Taggen und zur Diskussion von Bildern der Kunstgeschichte und Bildwissenschaft Zusammen mit dem Institut für Kunstwissenschaft Anmeldung: Jetzt über KLIPS -> Lehre -> WS09/10 Fragen? Mail an

ISWeb - Information Systems & Semantic Web Carsten Saathoff MMDB Slide 95 Werbeblock Diplomarbeiten Beispiele für Themen Kontextsensitive Visualisierung von Events und Objekten auf der Karte Repräsentation von dynamischen organisationalen Prozessen am Beispiel des Notfallmanagements TripleRanked Faceted Browsing Interface of Linked Open Data Auch online unter Und weitere Themen …