1. Berliner XML Tage. Humboldt Universität zu Berlin, 11-13 Oktober 2004 SWEB2004 – Intl Workshop on Semantic Web Technologies in Electronic Business Intelligent information services The marriage of multimedia presentation and the Semantic Web Dipl.-Inf. Lyndon J B Nixon AG Netzbasierte Informationssysteme Freie Universität Berlin Takustrasse 9, D 14195 Berlin nixon@inf.fu-berlin.de

2. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop Introduction: multimedia presentation Multimedia presentation is about communicating an idea to a user through the use of synchronized mixed media The World Wide Web is becoming both ubiquitous and multimedia based Users want access to the information on the Web, and that access must be flexible to differing needs and contexts TextVideoImageAudio

3. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop Introduction: Semantic Web Knowledge representation techniques allow machines to process data according to its meaning, as modelled in an ontology The Semantic Web is about extending the existing Web of content into a Web of knowledge that can be understood, reasoned about and intelligently processed by computers. An ontology for geneaology (1) A person can be a parent or child of another person (2) If another person is my child, I am their parent (3) The parent of a parent is a grandparent Reasoning on knowledge John is a parent of Lucy Maria is a child of Lucy It follows that -> Lucy is a parent of Maria John is a grandparent of Maria

4. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop Introduction: a beautiful marriage? Idea: combine Semantic Web and multimedia presentation to offer users a flexible means for access to their desired information through Web-based knowledge acquisition and reasoning Multimedia-based presentation of the information result User expresses an information wish Web of content Internet Service

5. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop State of the Art: IMMPS* KNOWLEDGE SERVER semantics presents LAYERS application context user design control content design realizes presents Goal formulation User KNOWLEDGE SOURCES EXTERNAL CONTENT EXTERNAL APPLICATION * A Standard Reference Model for Intelligent Multimedia Presentation Systems, Bordegoni, M. et al A proposed generic model for Intelligent Multimedia Presentation Systems (IMMPS) Decision making at every stage based on knowledge provided from a server No concrete specification for knowledge representation, querying or manipulation

6. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop State of the Art: Cuypers* Multimedia IR System Semantic Structure Communicative Devices Qualitative Constraints Quantitative Constraints Final Form Presentation HTTP Web Client * Towards Second and Third Generation Web-Based Multimedia, van Ossenbruggen, J. et al A multimedia generation system using a constraint logic programming approach to support backtracking and best fit results Extensions into Semantic Web technologies: RDF metadata used as basis to infer spatio-temporal relationships between media OWL ontologies used to define domain and discourse models for the organisation and presentation of media Acts as an interface to a multimedia database Semantics limited to annotated media objects Knowledge for use in generation is fixed before execution

7. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop State of the Art: CHIME* * CHIME: Service-oriented Framework for Adaptive Web-based Systems, Chepegin, V.; Aroyo, L.; De Bra, P.; Houben, G.-J. An Adaptive Hypermedia System A modular, interoperable architecture An ontological approach for a shared conceptual understanding Research direction multi- agent systems / Semantic Web Services Project focus limited: user modelling & personalisation

8. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop SWeMPS: architecture 1 Query Tool Reasoner Service Planner ModellerFormatter Multimedia presentation Conceptual space Multimedia model Service Space Resource Space Rules-based Application Logic Information request Based on a: Generic rule-base Conceptualisation of process Sets of resources and services Abstraction of the result presentation Using plugged-in components to interface the application with external, distributed, heterogeneous data

9. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop SWeMPS: ontology The ontology provides a shared understanding of the definition of the multimedia generation process An instantiation of the ontology (knowledge base) defines the operation of an individual multimedia generation task Three key distinctions made in the model, between subjects, resources and services.

10. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop SWeMPS: rules Rule base written using PROVA, a language combining rule-based (Prolog) and object-oriented (Java) approaches. It supports extension with all Java APIs, such as Jena (Semantic Web) or Web Services. Rulebase assert/retract trigger rules query handler conceptual space service space knowledge acquisition reasoner (Jena) service invocation Fact(X) :- Block of Prova code

11. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop Case Study: Searching on a user goal Query handler asserts an information goal in the factbase assert(goal(me:betty,gen:relationship,X)) This means who is related to Betty? A rule is triggered by the goal assertion goal(X,Y,Z) :- [code for querying conceptual space] The executed code queries on the conceptual space, including the ontology for family relationships and metadata on my family SELECT ?k_obj WHERE (me:betty, gen:relationship, ?k_obj)

12. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop Case Study: Mediation and Reasoning The family metadata is using the GEDCOM ontology which states that Individual s are spouseIn or childIn a Family The query fails, as the metadata has no gen:relationship properties The rulebase supports backtracking: as the rule fails, an alternative is triggered which attempts to map an unknown to a known ontology, e.g.: X gen:wife Y <- X spouseIn Z, X sex F, Y spouseIn Z The ontology for family relationships states that the derived properties are all subProperties of gen:relationship, and hence it can be reasoned that: X gen:relationship Y <- X gen:wife Y

13. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop Case Study: Resource resolution The information goals answer is that Betty is the wife of Bob, sister of Barb and daughter of Ben and Bessy. The conceptual space references a family photo album, in which the photos are annotated with RDF. A service is invoked to extract knowledge about the family members. Found resources are asserted in the factbase triggering a rule to adapt and add them to the multimedia model. The multimedia model has mappings from the conceptual relationships to communicative abstractions e.g. m_directLeft(R1,R2) :- represents(R1,S1), represents(R2,S2),query(S1,gen:wife,S2)

14. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop Case study: Presentation result * BenBessy BobBettyBarb * Images from clipart.freedesktop.org Notes result derived from existing knowledge and annotated media information goal can be changed and other presentations generated gaps or mismatches in knowledge can be resolved with dynamic acquisition / mapping presentation can be adapted to additional knowledge

15. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop Conclusion: Needs A complete framework needs extensions to meet real world needs, e.g.: Mediation services for resolving ontology mismatches Semantic Web services framework for the discovery and invocation of services Ontology and rule management tools

16. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop Conclusion: Gains The SWeMPS framework intends to allow implementers to leverage the (soon..) existing knowledge infrastructure of the Semantic Web It offers the Semantic Web community a means for making the stored knowledge intuitively accessible to users It is to be built upon to realise individual Intelligent Information Services for communicating knowledge from given domains to users

17. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop Conclusion: An e-business vision Data providers contractually agree to provide trustworthy content and knowledge to a service provider, who is acting as the host for an Intelligent Information Service Users make information requests to the service (for a charge, through registration, with an agreement to receive advertising) through various UIs on various devices, to which the final multimedia presentation is adapted Intelligent Information Service Document Repository Media Repository Knowledge Repository Interactive Television PC-based Web UI Mobile Phone

18. Lyndon Nixon Intelligent information services Berliner XML Tage, 11-13 Okt. 2004 SWEB2004 Workshop Thats all folks! More at http://page.mi.fu-berlin.de/~nixon/ Thank you for your attention!