A Stepwise Modeling Approach for Individual Media Semantics Annett Mitschick, Klaus Meißner TU Dresden, Department of Computer Science, Multimedia Technology Group „Applications of Semantic Technologies“ (AST' 06)

AST Outline Introduction Approach: -Semantics for Personal Media Management -Information Instantiation and Integration -Implemented System Architecture Conclusion and Future Work

AST Introduction Media Home: -Growing amount of digital assets (text, images, video, audio) complex, high-dimensional -User: non-/semi-professional individual conceptualizations usually minimum efforts to annotate/organize -Tools mostly apply administrative (objective) metadata  vs. user interpretation/conceptualization Goal: automated modeling of essential semantic descriptions from existing metadata to ease annotation work

AST Introduction Solutions: -Automated feature extraction and indexing techniques to manage complexity of huge multimedia collections low-level features  Similarity-based Search (Query-by-Example, etc.) higher-level features  Pattern Recognition/Classification (objects, people, genres, etc.)  requires prototype features (user context, user feedback)  “Semantic Gap” -Media descriptions based on common metadata standards to enable interoperability & comparability MPEG-7 Multimedia Description Schema RDF/OWL representation for semantic modeling -Semantic Desktop Personal Information Management based on Semantic Web Technologies

AST Approach :: Semantics for Personal Media Management Knowledge representation -Allow description of background-knowledge (context information like events, activities, actors, locations, etc.) -Allow integration of domain models and individual extensions (e.g. integration of MPEG-7 MDS, FOAF for description of people, etc.) -Manageable, straightforward  for non-professionals ABC [Hunter, et al.] -“facilitate interoperability between metadata ontologies“, “conceptual basis for automated mapping …”, “guidance to communities…” -Base classes to attach domain-specific entities  MPEG-7 MDS -Temporal semantics and related actualities (who, what, when, where)  evolution/transition of objects

AST Approach :: Semantics for Personal Media Management Acquisition of as much information as possible from content + context -File information, administrative metadata, structural features, etc. Information sources depend on application context -naming, availability/accessibility, semantics, etc. of features/attributes e.g. “Date Time Original”/”Date Created”/…, “Creator”/”Author”/…, etc. -properties might be relevant, or redundant in some context e.g. “EXIF:ExposureTime” and “EXIF:ShutterSpeedValue”

AST First step: -Apply sets of mapping rules to construct media-centric description (MDS) -Mapping rules expandable regarding supported metadata schemes (  developer) Approach :: Information Instantiation and Integration Make=FUJIFILM Model=FinePix F601 [(urn:Exif:Make rdf:value ?j) (urn:Exif:Model rdf:value ?k) (urn:Filepath rdf:value ?l) hashCode(?l, ?t) makeTemp(?x) -> (?t mds:creationTool ?x) (?x mds:make ?j) (?x mds:model ?k) (?x rdf:type mds:Device)] FUJIFILM FinePix F FUJIFILM FinePix F601

AST Approach :: Information Instantiation and Integration Next steps: -Apply derivation rules (schema level) to establish relationships to actualities and temporalities (  developer, user) -Apply classification rules (instance level) to refine description model regarding known pattern (  user)

AST Approach :: Implemented System Architecture Design rationale: -Automated import and indexing of media items triggered by file system actions -Registry of media type specific analyzing components  providing metadata and feature extraction -Processing of semantics and ontology model wrapped by generic model API  independent from RDF/OWL storage and querying solution (Jena, Sesame, …) -Extensible, flexible service architecture  plug-in based setup, extension at run-time (OSGi service platform)

AST Conclusion and Future Work Our approach: -Part of the K-IMM (Knowledge through Intelligent Media Management) project  managing private media collections with the help of semantic technologies -Stepwise modeling of resource descriptions to separate distinct problems of construction of media semantics Building objective media descriptions from various sources Deriving subjective, user-oriented semantics Future Work: -Strategies to extend and refine basic rule sets by means of user feedback, context capture -Distribution of modeling steps (Web Services/Agents) -Detailed, comprehensive evaluation

AST Thank you for your attention! Annett Mitschick Multimedia Technology Group, Department of Computer Science, TU Dresden