1. Rhizomik Semantic Integration and Retrieval of Multimedia Metadata Roberto García and Universitat de Lleida, Lleida, Spain Òscar Celma Universitat Pompeu Fabra, Barcelona, Spain

2. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Contents Introduction Ontological framework Integration framework Integration & Retrieval Scenario Conclusions Future Work

3. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Contents Introduction Ontological framework Integration framework Integration & Retrieval Scenario Conclusions Future Work

4. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Introduction Digital Media requires Multimedia Metadata. Main requirements: Great amounts of metadata Low-level to user-friendly metadata Interoperability Proposal: semantic metadata and an ontological framework for integration.

5. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Contents Introduction Ontological framework Integration framework Integration & Retrieval Scenario Conclusions Future Work

6. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Ontological Framework Multimedia ontology. Profit from non-semantic initiative MPEG-7. XML Semantic Web: XSD2OWL: schema to ontology. XML2RDF: XML instance data to RDF instances. Complete MPEG-7 OWL Ontology

7. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik MPEG-7: Multimedia Content Description Standard content-based description for audiovisual information content searching, filtering and identification Descriptors: low-level features structure semantic Ontological Framework models collections creation...

8. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Ontological Framework Other semantic multimedia metadata initiatives: Hunter02: not complete, RDF+DAML. Tsinaraki04: not complete, semantic part of MDS. Troncy03: not complete, from an ontology to MPEG-7. Direct benefits: XQuery vs. Semantic Query, example next…

9. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Ontological Framework One XQuery for each type vs. One semantic query for supertype. SegmentType hierarchy.

10. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Ontological Framework ReDeFer XSD2OWL Mappings: owl:intersectionOf owl:unionOf sequence choice owl:maxCardinality owl:minCardinality @maxOccurs @minOccurs rdfs:subClassOfextension@base|restriction@base owl:RestrictioncomplexType//element owl:ClasscomplexType|group|attributeGroup rdfs:rangeelement@type rdfs:subPropertyOfelement@substitutionGroup rdf:Property owl:DatatypeProperty owl:ObjectProperty element|attribute OWLXML Schema

11. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Contents Introduction Ontological framework Integration framework Integration & Retrieval Scenario Conclusions Future Work

12. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Integration Framework MPEG-7 OWL Ontology as integration base for: Semantic metadata: align ontologies with MPEG-7 ontology. MusicBrainz (http://musicbrainz.org) Simac (http://www.semanticaudio.org) Music Vocabulary (http://www.kanzaki.com/ns/music) Etc. XML2RDF: RDF from MPEG-7 XML instance metadata.

13. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Integration Framework. Architecture

14. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Integration Framework MusicBrainz to MPEG-7 OWL mappings: musicbrainz:duration mpeg7:MediaDuration musicbrainz:sortName mpeg7:Name dc:title mpeg7:Title dc:author mpeg7:Creator musicbrainz:Track mpeg7:AudioSegmentType musicbrainz:Album mpeg7:CollectionType musicbrainz:Artist mpeg7:CreatorType

15. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Integration Framework Simac to MPEG-7 OWL mappings: simac:meter mpeg7:Meter simac:tempo mpeg7:Beat simac:key mpeg7:Key simac:mode mpeg7:Scale simac:Descriptor mpeg7:AudioDSType simac:duration mpeg7:MediaDuration simac:title mpeg7:Title simac:Track mpeg7:AudioSegmentType simac:name mpeg7:GivenName simac:Artist mpeg7:CreatorType

16. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Integration Framework Music Vocabulary ontology to MPEG-7 OWL mappings: music:meter mpeg7:Meter music:key mpeg7:Key music:Artist mpeg7:CreatorType music:sections mpeg7:AudioSegment music:Music_Unit mpeg7:AudioSegmentType

17. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Integration Framework. Architecture

18. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Integration Framework XML2RDF: XML tree RDF graph. Deduce blank node types from XSD2OWL ontologies restrictions.

19. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Integration Framework XML2RDF example

20. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Contents Introduction Ontological framework Integration framework Integration & Retrieval Scenario Conclusions Future Work

21. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Integration & Retrieval Scenario Low-level descriptors from audio track signal: Track editorial metadata: a simac:Track; simac:tempo 122; musicbrainz:trmid e3c41bc1-4fdc-4ccd-a471-243a0596518f. a musicbrainz:Track; dc:title Blowin in the wind; dc:author [musicbrainz:sortName Bob Dylan]; musicbrainz:trmid e3c41bc1-4fdc-4ccd-a471-243a0596518f.

22. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Integration & Retrieval Scenario Generic MPEG-7 based integration rules. Example: audio track with the same fingerprint are the same individual, their metadata is merged. mpeg7:AudioType(track1) mpeg7:AudioType(track2) musicbrainz:trmid(track1, trm1) musicbrainz:trmid(track2, trm2) (trm1 = trm2) owl:sameIndividualAs(track1, track2)

23. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Integration Framework

24. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Integration & Retrieval Scenario Transparent retrieval, MPEG-7 based queries. RDQL example: SELECT ?title WHERE (?track,,mpg7:AudioSegmentType), (?track,,?title), (?track,,?author), (?author,,Bob Dylan), (?track,,?tempo) AND (?tempo >= 120) USING mpeg7 FOR

25. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Contents Introduction Ontological framework Integration framework Integration & Retrieval Scenario Conclusions Future Work

26. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Integration Framework

27. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Conclusions MPEG-7 OWL ontology, an upper-level multimedia ontology for: Semantic metadata integration align ontologies to MPEG-7 OWL. XML metadata integration XML2RDF. Semantic queries for retrieval generic queries based on MPEG-7.

28. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Contents Introduction Ontological framework Integration framework Integration & Retrieval Scenario Conclusions Future Work

29. Semantic Integration and Retrieval of Multimedia Metadata Rhizomik Future Work Semantics-based recommendation systems http://foafing-the-music.iua.upf.edu http://rhizomik.net Infer user-friendly metadata from low-level descriptors. mpeg7:AudioType(track) mpeg7:Beat(track, t) (t > 120) mpeg7:Loudness(track, l) (l > 0.9) genres:Rock(track) danceability(track, high)

30. Rhizomik Thank you for your attention Contact: roberto@rhizomik.net ocelma@iua.upf.edu