1. Information Systems & Semantic Web University of Koblenz Landau, Germany Semantic Multimedia Web Ansgar Scherp Basierend auf Folien von Carsten Saathoff, Raphael Troncy und Lynda Hardman

2. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 2 Was bisher geschah... MMDB als Erweiterung von ORDBMS Information Retrieval als Basis für Queries Feature Extraktion um Inhalt zu beschreiben Feature Transformation um kompaktere Darstellung zu bekommen Fokus auf low-level features Distanzen und Ähnlichkeiten Indizierung von Features für schnellen Zugriff

3. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 3 Probleme traditioneller MMDB Datenstrukturen und Schemata meist proprietär MMDB typischerweise für eine Applikation aufgesetzt. Spätere Integration mit anderen Applikationen schwer Ad-Hoc Integration eher unmöglich Starker Fokus auf Low-Level Features Semantische Lücke Kein direktes Mapping zwischen Low-Level Features und Semantik des Bildes Retrieval primär über Ähnlichkeit Fast alle Studien zeigen, dass Nutzer dadurch nicht zufriedengestellt werden Nutzer wollen semantisch Anfragen

4. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 4 Metadaten

5. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 5 Metadaten (2) Stichworte GPS Information Kamera Daten Datum

6. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 6 Metadaten Daten über Daten Autor, Creation-Date, Keywords,... Wie repräsentieren? Relationales Schema XML Semantic Web Technologien Metadaten sollten interoperabel sein Web Desktop Intranets Viele Applikation müssen kommunizieren

7. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 7 Überblick Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM Probleme mit MPEG7 Core Ontology on Multimedia (COMM) Linked Open Data KAT – K-Space Annotation Tool Semi-Automatische Effiziente Annotation Szenarien

8. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 8 Semantic Web auf einer Folie Employee PostDoc Professor Person rdfs:subClass cooperatesWith rdfs:Range rdfs:Domain Ontology Steffen Staab... http://www.uni-koblenz.de/~staab rdf:type Meta- data Siegfried Handschuh... Web page http://www.deri.ie/~sha URL swrc:cooperatesWith

9. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 9 Semantic Web for Multimedia IsWeb @ Bad Kreuznach 2007 WIAMIS 2008 in Klagenfurt Multimedia Ontology http://kodemaniak.de/foaf.rdf depicts Carsten Saathoff Domain Ontology hasName rdf:type Researcher ResearchMeeting:= >=1 depicts.Researcher ResearchMeeting Zeig mir alle Bilder von Research Meetings! http://isweb.uni-koblenz.de/ http://wiamis2008.org

10. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 10 Überblick Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM Probleme mit MPEG7 Core Ontology on Multimedia (COMM) Linked Open Data KAT – K-Space Annotation Tool Semi-Automatische Effiziente Annotation Szenarien

11. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 11 Semantische Lücke 0010EE -> bläulich 0033FE -> bläulich Visuell ähnlich! Aber semantisch unterschiedlich!

12. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 12 Semantische Lücke Visuell ähnlich, semantisch ähnlich, aber... Italien USA

13. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 13 Ebenen der Semantik Generische Objekte Generische Szene Spezifische Objekte Spezifische Szene Abstrakte Objekte Abstrakte Szene Eine Person Personen unterhalten sich Churchill Churchill, Roosevelt, Stalin sitzen zusammen Churchill, Premierminister, GB,... Big Three, Yalta Konferenz WWII,... Wissen

14. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 14 Überblick Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM Probleme mit MPEG7 Core Ontology on Multimedia (COMM) Linked Open Data KAT – K-Space Annotation Tool Semi-Automatische Effiziente Annotation Szenarien

15. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 15 Overview of Canonical Processes

16. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 16 Example 1: CeWe Color PhotoBook Application for authoring digital photo books Automatic selection, sorting and ordering of photos Context analysis methods: timestamp, annotation, etc. Content analysis methods: color histograms, edge detection, etc. Customized layout and background Print by the European leader photo finisher company http://www.cewe-photobook.com

17. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 17 CeWe Color PhotoBook Processes My winter ski holidays with my friends

18. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 18 CeWe Color PhotoBook Processes

19. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 19 CeWe Color PhotoBook Processes

20. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 20 CeWe Color PhotoBook Processes

21. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 21 CeWe Color PhotoBook Processes

22. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 22 CeWe Color PhotoBook Processes

23. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 23 Example 2: Vox Populi Video Sequences Generation Stefano Bocconi, Frank Nack Interview with America video footage with interviews and background material about the opinion of American people after 9-11 http://www.interviewwithamerica.com http://www.interviewwithamerica.com Example question: What do you think of the war in Afghanistan? I am never a fan of military action, in the big picture I dont think it is ever a good thing, but I think there are circumstances in which I certainly cant think of a more effective way to counter this sort of thing…

24. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 24 Vox Populi Premeditate Process Analogous to the pre-production process in the film industry Static versus dynamic video artifact Output Script, planning of the videos to be captured Questions to the interviewee prepared Profiles of the people interviewed: education, age, gender, race Locations where the interviews take place

25. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 25 Vox Populi Annotations Contextual Interviewee (social), locations Descriptive Question asked and transcription of the answers Filmic continuity, examples: gaze direction of speaker (left, centre, right) framing (close-up, medium shot, long shot) Rhetorical Rhetorical Statement Argumentation model: Toulmin model

26. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 26 Vox Populi Statement Annotations Statement formally annotated: E.g. war best solution A thesaurus containing: Terms on the topics discussed (155) Relations between terms: similar (72), opposite (108), generalization (10), specialization (10) E.g. war opposite diplomacy

27. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 27 Toulmin Model ClaimData Qualifier WarrantBackingConditionConcession 57 Claims, 16 Data, 4 Concessions, 3 Warrants, 1 Condition

28. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 28 Vox Populi Query Interface

29. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 29 Vox Populi Organize Process Using the thesaurus, create a graph of related statements nodes are the statements (corresponding to video segments)war best solution,diplomacy best solution,war not solution edges are either support or contradict support contradict war best solution war not solution diplomacy best solution

30. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 30 Result of Vox Populi Query I am not a fan of military actions War has never solved anything I cannot think of a more effective solution Two billions dollar bombs on tents

31. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 31 Vox Populi Processes

32. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 32 Canonical Processes 101 Canonical: reduced to the simplest and most significant form possible without loss of generality Formalization of each process in UML diagrams Process Process artifacts Process actors External world artifacts

33. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 33 Premeditate Process where initial ideas about media production are established Design a photo book of my last holidays for my family Create argument-based sequences of videos of interviews after September 11

34. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 34 Create Media Asset Process where media assets are captured, generated or transformed

35. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 35 Annotate Process where annotation is created

36. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 36 Semantic Annotate The annotation uses some controlled vocabularies Subject matter annotations of your photos Rhetorical annotations in Vox Populi

37. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 37 Package Process where process artifacts are logically and physically packed

38. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 38 Query Process where a user retrieves a set of process artifacts based on a given query

39. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 39 Construct Message Process where an author specifies the message they wish to convey Our holiday was sporty, great weather and fun Create clash about whether war is a good thing

40. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 40 Organize Process where process artifacts are organized according to the message Organize a number of 2-page layouts in photobook Use semantic graph to select related video clips to form linear presentation of parts of argument structure

41. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 41 Publish Process where final content and user interface is created

42. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 42 Distribute Process where final interaction between end-users and produced media occurs

43. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 43 Canonical Processes Possible Flow

44. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 44 Sum Up Community agreement, not yet another model Large proportion of the functionality provided by multimedia applications can be described in terms of this model Initial step towards the definition of open web-based data structures for describing and sharing semantically annotated media assets

45. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 45 Überblick Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM Probleme mit MPEG7 Core Ontology on Multimedia (COMM) Linked Open Data KAT – K-Space Annotation Tool Semi-Automatische Effiziente Annotation Szenarien

46. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 46 MPEG7 ISO Standard der MPEG Community Einheitliches Format zu Speicherung von Multimedia Metadaten Struktur Features Semantik Extrem (!) umfangreich Darauf basierend MPEG21 mit Fokus auf DRM etc. Hat im Gegensatz zu MPEG1-4 nichts mit Kodierung zu tun

47. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 47 MPEG7 (2) Basiert auf XML MPEG7 Beschreibung ist eine Hierarchie Deskriptoren beschreiben Eigenschaften von Multimedia Daten Struktur Video -> Shots -> Frames Bilder -> Segmente Semantik Low-Level Features Um Flexibilität zu gewahren, können Deskriptoren sehr vielseitig kombiniert werden.

48. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 48 Big Three

49. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 49 Issues Churchill Roosevelt Stalin... How do you formulate a query to get images showing Churchill et al.? First Shot (Xpath): //StillRegion[.//Keyword=Churchill or.//Keyword=Roosevelt or.//Keyword=Stalin] Winston Churchill Recognizer Franklin Roosevelt Recognizer Josef Stalin Recognizer

50. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 50 Probleme mit MPEG-7? Annotationen sind nicht interoperabel! Mehrdeutigkeiten Mehrere Möglichkeiten um semantisch identische Annotationen zu beschreiben Deskriptoren können auf viele Arten kombiniert werden Komplexe Anfragen müssen alle Alternativen beachten

51. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 51 Capabilities and Maturity Levels Integration Automation Former SituationCurrent SituationFuture / Desired Situation no standard, no vocabulary manual 1:1 agreement on format and semantics tight coupling of data and applications standard vocabulary manual 1:1 agreement on mpeg-7 vocabulary tight coupling of data and applications standard vocabulary pre-defined meaning ad-hoc coupling of data and applications CORE ONTOLOGY Nächster Teil der VL Formerly MPEG-7 COMM

52. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 52 Ontology Stack Core Ontologies Domain Ontologies Foundational Ontologies Span across multiple fields, each covering multiple domains Modelling of the most abstract concepts like event, object,... Core Ontologies Situated in one field, but spans multiple domains Can base on foundational ontologies Examples fields: events, annotation, communication,... Domain Ontology For a specific domain, e.g., fishery, human body, etc.

53. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 53 MPEG-7 COMM Requirements on a high quality MM Ontology Challenge Building Block Legend

54. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 54 Text Descriptor Music Manager Compound Document Requirements for COMM Reusability Design a core ontology for any multimedia related application MPEG-7-Compliance Support most important description tools Extensibility Enable inclusion of further description tools (even those that are not part of MPEG-7!) media types Separation of Concerns Clear separation of domain knowledge and knowledge about structure Modularity Enable customization of multimedia ontology High degree of axiomatization Ensure interoperability through machine accessible semantics Churchill Recognizer Josef Stalin Recognizer Face Detector Photo Manager Authoring Tool Semantic Annotation decomposition visual descriptors audio descriptors.........

55. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 55 Is MPEG-7 a good Basis for a high Quality Ontology? Shortcomings of badly modelled ontologies [Oberle et al., 2006]: 1)Conceptual ambiguity Difficulties in understanding the meaning of concepts and their relations 2)Poor axiomatization Axiomatization of well defined concepts is missing 3)Loose Design Presence of modelling artefacts (concepts without ontological meaning) Shortcomings mainly hinder Extensibility Interoperability Especially 1) and 2) are major shortcomings of MPEG-7 1-to-1 translations from MPEG-7 to OWL/RDFS (e.g. [Hunter, 2003a]) will not result in high quality ontologies! Churchill Roosevelt Stalin

56. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 56 MPEG-7 COMM Requirements on a high quality MM Ontology Quality of Ontologies Challenge Building Block Legend

57. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 57 How to Design a High Quality Multimedia Ontology? Approach from [Oberle, 2005], [Oberle et al., 2006]: Use a well designed foundational ontology as a modelling basis to avoid shortcomings Foundational ontologies provide Formal precision Domain independence Broad scope Building upon foundational ontologies prevents easy inclusion of modeling artefacts reduces conceptual ambiguity inherit rich axiomatization

58. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 58 Methodology MPEG-7 COMM Requirements: High Quality MM Ontology Quality of Ontologies Quality Measures for Ontologies Reference Ontologie MPEG-7 Compliance Challenge Building Block Legend

59. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 59 Methodology for Design Pattern Definition Identification of most important MPEG-7 functionalities [Arndt et al., 2007] : Decomposition of multimedia content into segments Annotation of segments with meta data (e.g. visual descriptor, media information, creation & production, …) General: Identify repetitive structures and describe them at an abstract level Describe digital data by digital data at an arbitrary level of granularity Additional patterns are needed for: Complex data types of MPEG-7 Semantic annotation by using domain ontologies Interface between reusable multimedia core and domain specific knowledge

60. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 60 DOLCE Design Patterns: OIO Foundational ontology DOLCE+DnS Ultralight Aims at capturing the most essential aspects in the world Defines disjunctive upper classes Event, Object, Quality and Abstract Follows a pattern-oriented approach for ontology design 2 design patterns (extensions) that are especially important for MPEG-7: Ontology of Information objects (OIO): Formalization of information exchange Information object represents pure abstract information (message) Relevance for multimedia ontology: MPEG-7 describes digital data (multimedia information objects) with digital data (annotation) Digital data entities are information objects

61. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 61 DOLCE Design Patterns: D&S Descriptions & Situations (D&S): Formalization of Context Relevance for multimedia ontology: Meaning of digital data depends on context Digital data entities are connected through computational situations (e.g. input and output data of an algorithm) Algorithms are descriptions Annotations and decompositions are situations that satisfy the rules of an algorithm / method

62. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 62 Methodology MPEG-7 COMM Requirements: High Quality MM Ontology Quality of Ontologies Quality Measures for Ontologies Reference Ontologie Identification of repetitive structures MPEG-7 Compliance Pattern definition through Specialization Challenge Building Block Legend Repr. of Context Repr. of Information

63. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 63 Ontology of Information Objects (OIO)

64. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 64 Example Information Object Graz Tourist Guide Information Realization http://cms.graztourismus.at/cms/ziel/42425/EN/ Booklet Information Encoding: English German About: Places, Buildings (e.g. Clock Tower) Agent: 1. iMedia Visitor / 2. Tourist Officer / 3. Graphics Designer Expresses: 1.Walking Path through Graz 2.Small-Size Tourist Guide 3.Arrangement of Illustrations

65. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 65 Descriptions & Situations (D&S) Distinction between: DOLCE ground entities (regions, endurants, perdurants) Descriptive entities (parameters, roles, courses) Descriptions Formalize context Define descriptive concepts Situations Are explained by descriptions Are settings for ground entities

66. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 66 Putting it Together: Decomposition Pattern

67. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 67 Example Image1 playsRole SegmentationInput Segment2 playsRole SegmOutp Segment4 playsRole SegmOutp Segment1 playsRole SegmOutp Segment3 playsRole SegmOutp Segment1 playsRole SegmInput Via its role in a computational task the different parts may be arbitrarily nested and related to different computing algorithms Querying for all subparts takes place along a well-defined pattern

68. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 68 Modular Architecture Multimedia ontology consists of Core module that contains the design patterns Modules that specialize the core module for different media types Modules that contain media independent MPEG-7 description tools such as media information or creation & production Data type module that formalizes MPEG-7 data types e.g. matrices, vectors, unsigned- int-5, float-vector, probability- vector, …

69. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 69 Does the Multimedia Ontology fulfil the Requirements? Reusability MPEG-7-Compliance Design patterns enable the representation of description tools Extensibility Design patterns are media independent possibility to include further media types arbitrary descriptors Extensions of multimedia ontology will not affect legacy annotations due to DOLCE+D&S+OIO Separation of Concerns Clear separation between domain specific and multimedia related knowledge Modularity Modular architecture allows customization High degree of axiomatization Design patterns come with generic axiomatization that is refined in derived ontology modules Churchill Recognizer Josef Stalin Recognizer Photo Manager COMM One such extension has already been done for Text Annotation. Another one for compund document annotation is currently developed! Content & Media Annotation Pattern Semantic Annotation Pattern See slide before this slide! OWL-DL version available for download.

70. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 70 Benefits of a DOLCE-aligned Multimedia Ontology Usage of DOLCE enforces clean design Constraints prohibit arbitrary placement of MPEG-7 concepts into DOLCE Similar concepts will be placed on similar locations of the taxonomy Things that are different, have to be separated (e.g. data and the perceivable content that is carried) Extensibility due to underlying general taxonomy of DOLCE Possibility to describe multimedia domain at an arbitrary level of detail (e.g. segments have pixels as atomic parts) Rigorous application of the D&S and OIO patterns allows description of digital data in different contexts (e.g. data acting as input or output for an algorithm)

71. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 71 Benefits compared to MPEG-7 Linkage with domain ontologies allows meaningful semantic annotation of multimedia content Semantic part can be entirely replaced with a domain ontology Clear separation between domain ontologies and multimedia core ontology through semantic annotation pattern Easier queries Annotation pattern guarantees equal representation of all annotations Complex data type pattern guarantees uniform access to nested data No complex XML-structures to parse Multimedia ontology only uses restricted inventory of DOLCE predicates Higher interoperability through machine accessible semantics and underlying DOLCE axiomatization

72. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 72 http://en.wikipedia.org/wiki/ Yalta_Conference World War II Yalta... History Ontology Creating a Multimedia Presentation Revisited SR1 SR2 SR3 Winston Churchill Recognizer Franklin D. Roosevelt Recognizer Josef Stalin Recognizer Churchill Roosevelt Stalin Sparql: select ?image where { ?image plays AnnotatedDataRole. ?x plays SemanticLabelRole. ?x rdf:type pol:President } Photo Manager Authoring Tool

73. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 73 Überblick Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM Probleme mit MPEG7 Core Ontology on Multimedia (COMM) Linked Open Data KAT – K-Space Annotation Tool Semi-Automatische Effiziente Annotation Szenarien

74. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 74 A Giant Graph Open to the World Annotate the content (interpretation) Elephant, Ganesh, Thailande, Holidays, Chiang Mai Link to knowledge on the Web :img foaf:depicts dbpedia:Ganesh dbpedia:Ganesh rdfs:label "Vinayaka" dbpedia:Ganesh skos:altlabel "Ganapati" dbpedia:Ganesh rdf:type wn:synset-Deities-noun-1 dbpedia:Ganesh owl:sameas wn:synset-Ganesh-noun-1 An image of the Elephant Ganesh Raphaël Troncy

75. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 75 Linking Open Data Project Expose open datasets in RDF Set RDF links among the data items for different datasets Over 2 billion triples, 3 millions links (March 2008) http://richard.cyganiak.de/2007/10/lod/

76. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 76 Linked Open Data March 2009

77. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 77 Warum ist LOD wichtig? RDFa wird von der Google Suchmaschine verarbeitet Content provider bieten nun RDFa an Erhöht Click-trough-rate auf Webseiten (Werbeanzeigen) Erhöht Ranking der Webseiten in Google SIOC Ontology zur Verlinkung von Online Communities wird genutzt von Yahoo! Wird durch SearchMonkey eingesammelt Tools um RDFa zu publishen Usw.

78. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 78 DBpedia DBpedia is a community effort to: extract structured "infobox" information from Wikipedia interlink DBpedia with other datasets on the Web

79. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 79 DBpedia

80. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 80 Automatic Links Among Open Datasets Processors can switch automatically from one to the other …

81. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 81 Take Home Message Reuse what is there Of course, one could create RDF data manually … … but that is unrealistic on a large scale Goal is to generate RDF data automatically when possible and "fill in" by hand only when necessary service to get RDF from flickr images http://www.kanzaki.com/works/2005/imgdsc/flickr2rdf http://www.kanzaki.com/works/2005/imgdsc/flickr2rdf service to get RDF from XMP http://www.ivan-herman.net/cgi- bin/blosxom.cgi/WorkRelated/SemanticWeb/xmpextract.ht ml http://www.ivan-herman.net/cgi- bin/blosxom.cgi/WorkRelated/SemanticWeb/xmpextract.ht ml Expose what you make

82. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 82 Überblick Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM Probleme mit MPEG7 Core Ontology on Multimedia (COMM) Linked Open Data KAT – K-Space Annotation Tool Semi-Automatische Effiziente Annotation Szenarien

83. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 83 KAT: K-Space Annotation Tool Goal Efficient annotation of multimedia content Means to create semantically rich annotations KAT provides framework for Executing analysis plugins Providing visualisation plugins Displaying/annotating content Browsing Interfaces with Core Ontology for Multimedia (COMM) Provides the common model Role based messaging to leverage reuse of components

84. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 84 Efficient Annotation Reduce time required by user for annotating content Integration of Automatic analysis methods Region labeling, object detection Key Frame Extraction, Shot Boundary Detection Automatic Organisation Clustering Inferencing Based on formal domain ontologies Semi-automation

85. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 85 Semantically Rich Annotations Relational Annotation Express how depicted entities are related Example: Soccer Game Who is tackling whom? Why was the penalty given? Ontologies provide means to express relations KAT aims at providing the means to efficiently create them Event-Based annotation Events are prominent in multimedia Create and manage events Relate events and media Allow for event-based retrieval and exploration

86. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 86 Architecture KAT-Core Plugin GUI Plugin View register COMM Repository... store display

87. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 87 Szenarien Effiziente Annotation von persönlichen Bildern Semi-Automatische, Semantische Annotation von Sport- Ereignissen Browsing von Bildsammlungen im Web

88. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 88 Persönliche Bildsammlungen Heutzutage typischerweise in Ordnern auf Festplatte Wenig Annotationen weil zu aufwendig Annotationen enthalten Viel Hintergrund Wissen Gefühle und persönliche Momente Frage: Wie kann ein Nutzer hier unterstützt werden? Automatische Annotation liefert nur einfache Semantik Daher: Clustering um Ereignisse zu finden User kann dann ganze Ereignisse annotieren Verwendung von NLP (Textanalyse) um semantische Annotation zu erzeugen

89. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 89 Sportereignisse Kombination verschiedener Algorithmen Highlight Detection: Goals, Corner-Shot,... Features: Motion, Geräusche, sichtbare Konzepte Analyse von Minute-by-Minute Reports Liefert andere Ereignisse Ergebnisse oft nur global Manuelles Refinement Zuweisen von Namen zu Spielern im Video/Bild Zuweisen von Rollen: Wer hat das Faul begangen, wer war Opfer Verknüpfen von Ereignissen Ziel: möglichst vollständige Annotationen effizient erstellen

90. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 90 Browsen von Bildkollektionen Flickr als COMM Maping von Tags in Wordnet Wordnet: linguistische Ontologie Mapping von Geoinformationen nach Geonames Ontologie von geographischen Informationen (Länder, Orte,...) Mapping nach dbpedia Wikipedia als maschienenlesbare Version Tags bekommen Kontext Anzeige werwandter Bilder, komplexe Queries Ergänzen der Annotationen

91. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 91 91 KAT as basis for der User Interface Objectives Explore and visualize semantic Web 2.0 data in real-time Acquainting oneself about an area of interest Semantic data 1 billion triples from DBpedia, GeoNames, WordNet, FOAF files and Flickr Very large, mixed-quality, semantically heterogeneous Winner of Billion Triples Track, Semantic Web Conference 2008, Karlsruhe [ISWC2008]

92. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 92 92 SemaPlorer – Web 2.0 Content Brower Search for locations, persons and tags Active facets like tags, location Map showing locations, sights, pictures Geo-referenced image from Flickr Information on locations, persons, tags SemaPlorer Live! http://btc.isweb.uni-koblenz.de/

93. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 93 Werbeblock HiWi-Jobs für SemaPlorer++ Interesse an Arbeit in einer Gruppe Tätigkeit die Kenntnisse aus dem Studium (und darüber hinaus) praktischen anwenden lässt Spaß am Tütfteln Entwicklung mit Java Mail mit Beschreibung an Erfahrungen / Lebenslauf an scherp@uni-koblenz.de

94. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 94 Werbeblock ImageAtlas II Projektpraktikum Entwicklung einer Plattform zum Taggen und zur Diskussion von Bildern der Kunstgeschichte und Bildwissenschaft Zusammen mit dem Institut für Kunstwissenschaft Anmeldung: Jetzt über KLIPS http://isweb.uni-koblenz.de/ -> Lehre -> WS09/10 Fragen? Mail an scherp@uni-koblenz.de

95. ISWeb - Information Systems & Semantic Web Carsten Saathoff saathoff@uni-koblenz.de MMDB 02.06.08 Slide 95 Werbeblock Diplomarbeiten Beispiele für Themen Kontextsensitive Visualisierung von Events und Objekten auf der Karte Repräsentation von dynamischen organisationalen Prozessen am Beispiel des Notfallmanagements TripleRanked Faceted Browsing Interface of Linked Open Data Auch online unter http://isweb.uni-koblenz.de/interactiveweb Und weitere Themen …