1. CS 586 – Distributed Multimedia Information Management Prof. Dennis McLeod

2. About the paper... Towards Ontology-Driven Discourse: From Semantic Graphs to Multimedia Presentations In Proceedings of the 2nd International Semantic Web Conference, 2003 By: Joost Geurts, Stefano Bocconi, Jacco van Ossenbruggen, and Lynda Hardman Presented by: Gabriela Gutierrez, February 11, 2004

3. Overview Introduction Example Scenario: Rembrandt Process 1: From Semantic Graph to Structured Progression Process 2: From Structured Progression to Multimedia Presentation Conclusion

4. Introduction Traditionally –Applying Semantic Web technology to multimedia information systems focuses on using annotations and ontologies to improve retrieval process –Presentation of data is “detail” best left to CSS or XSLT style sheets In this paper –Claim that importance and complexity of effective presentation design is grossly underestimated –Concentration on improving the presentation of the retrieval results

5. Introduction Human professional designers must understand: –Underlying semantics of the client’s information –Most effective order, grouping and priorities for structuring this information –Most effective means of using the chosen medium to convey the information

6. Introduction Information presentation design is a knowledge-driven process. It requires: –Sufficient knowledge about domain –Knowledge on ordering, grouping and prioritizing information –Knowledge about media design Selection of most appropriate medium Understanding of medium characteristics in order to choose an effective means to achieve the communication goal

7. Introduction Problem: –Professional designers can only design data-driven web sites if the underlying data, its semantics and target audience are relatively homogeneous. –Variety of data sources, semantic relations, output devices, and user profiles forces content providers to adopt one-size-fits-all approach. Automation is needed in order to make the presentation of information knowledge-driven.

8. Introduction Assumptions: –Multimedia items are properly annotated –Annotations represent domain relations in a semantic graph (e.g. RDF) –Graph has associated Domain ontology –There is a Discourse ontology containing information about different document genres and building blocks for creating documents for each genre –There is a Design ontology containing media design knowledge

9. Introduction

10. Example Scenario: Rembrandt Web query: “life and work of Rembrandt” User-selected type of structured progression: disc:Biography User-selected output medium: non-interactive multimedia presentation Semantic graph = retrieval component’s results + domain ontology semantics relations Structured progression = typical facts (name, DOB,…) + career facts + personal life info

12. Process 1: From Semantic Graph to Structured Progression CSS and XSLT operate purely on the XML level of RDF’s serialization syntax w/o any understanding or support for semantics of RDF data model Transformation process needs access to knowledge on RDF Schema level –For querying underlying domain ontology –For access to its own operating knowledge

13. Process 1: From Semantic Graph to Structured Progression Several transformations prototyped in Java and Prolog environments Direct access to a Sesame RDF Schema-based repository Can use any query language supported by Sesame (RQL, RDQL, SeRQL) to gain direct access on the RDF instance level and the RDF Schema Level Transformation process uses (declarative) domain and discourse-specific knowledge, while (procedural) transformation code remains generic

14. Process 1: From Semantic Graph to Structured Progression Transformation code uses RQL query to retrieve classes that Rembrandt instance belongs to... dom:Artist Discourse ontology defines instance of disc:ArtistBiography that has disc:Subject property with value dom:Artist Structured progressions have a disc:narrativeUnits property that specifies the disc:NarrativeUnits that can be used to construct it (e.g. disc:PersonalData, disc:PrivateLife and disc:Career)

15. Process 1: From Semantic Graph to Structured Progression Narrative Units have associated rules used to select matching content Example: disc:PrivateLife –Rules to select information about family relations from semantic graph –Graph includes relation dom:isMarried between Rembrandt and Saskia_Uylenburgh –Rule #3 in following table can use domain relation to select Saskia in the disc:Role of disc:Spouse Rules can be applied recursively –Rule #3 specifies that PrivateLife is the narrative unit that can be used for a subsequent nested story line –Process continues until no more rules can be applied or a rule specifies that no further expansion should happen

17. Process 1: From Semantic Graph to Structured Progression After all rules have been applied: –Biography w/ 3 narrative units –disc:PersonalData (Rembrandt in role of disc:MainCharacter) –disc:Career (Chiaroscuro in role of disc:Technique) –Disc:PrivateLife (Saskia_Uylenburgh in role of disc:Spouse)

18. Process 2: From Structured Progression to Multimedia Presentation Two-step process: 1.Structured progression transformed into Document Structure –Decisions on output medium (e.g. text, interactive hypermedia, passive multimedia) 2.Document Structure transformed into a tree of formatting objects –Detailed layout and formatting decisions (e.g. timing of presentation, transition effects)

19. Advantage: –Mapping discourse-specific narrative units to more general document elements allows for more commonly applicable formatting rules (e.g. disc:PrivateLife can be mapped to document section element, relying on common formatting rules for section-level elements) Disadvantage: –There is always a level that can no longer be specified in terms of document structure (e.g. a figure w/ too much detail) –Solution: detailed structures copied directly into document structure in step 1 in order to define specific rules in step 2 to deal w/ formatting directly Process 2: From Structured Progression to Multimedia Presentation

20. From Structured Progression to Document Structure

21. Conveying Document Structure Transforming a document structure into presentation constructs uses Cuypers library –Uses constraint solving techniques to verify that a presentation construct conforms to delivery-context constraints (e.g. screen size) –Allows alternative formatting specification if constraints are violated A rule that transforms a document structure into presentation construct has 2 discourse parameters: –disc:NarrativeType –disc:Role –Parameters allow system to adapt formatting of presentation to convey message more effectively

22. Conveying Discourse Semantics Directly Depending on their function, we need to define formatting for different media types –Rembrandt self-portrait (disc:Portrait in disc:PersonalData vs. disc:Painting illustrating Chiarocuro) Awareness of impact of different media modalities Fall-back rules –Image not identified as either disc:Portrait or disc:Painting should be applied generic formatting for images since mm:Painting and mm:Portrait are subclasses of mm:Image

23. Conclusion Only short presentations have been generated to date, based on restricted domain ontology Focus has been on single discourse structure (biography) and single document structure (multimedia presentation) Additional research required to scale the system to more realistic scenarios Under investigation: how knowledge about the user interacts w/ discourse and design knowledge used in current prototype