1. German Research Center for Artificial Intelligence DFKI GmbH Stuhlsatzenhausweg 3 66123 Saarbruecken, Germany phone: (+49 681) 302-5252/4162 fax: (+49 681) 302-5341 e-mail: wahlster@dfki.de WWW:http://www.dfki.de/~wahlster Intelligent Information Processing (IIP-2002) Montreal, 29 August 2002 Wolfgang Wahlster Personalized Web Interaction

2. © W. Wahlster First GenerationSecond GenerationThird Generation Static Webpages Fossils cast in HTML Interactive Web- pages JavaScripts and Applets Database Access and Template-based Generation Dynamic Web- pages Virtual Webpages Netbots, Information Extraction, Presentation Planning Personalized Webpages User Modeling, Machine Learning, Online Layout Three Generations of Webpages

3. © W. Wahlster 1.3rd Generation Websites: Adaptive and Virtual Webpages 2.The Need for Personalization 3.Component Technologies for Personalized Web Interaction - Constraint-based Layout - Plan-based Presentation Generation 4. Multimodal Interaction with Personalized Presentation Agents 5. Affective Personalization 6. Conclusions Outline of the Talk

4. © W. Wahlster A Virtual Webpage is generated on the fly as a combination of various media objects from multiple websites or as a transformation of a real webpage. looks like a real webpage, but is not persistently stored. integrates generated and retrieved material in a coordinated way. can be tailored to a particular user profile and adapted to a particular interaction context. has an underlying representation of the presentation context so that an Interface Agent can comment, point to and explain its components. Virtual Memory, Virtual Relation, Virtual Reality... What is a Virtual Webpage?

5. © W. Wahlster Virtual Webpage Retrieved from 5 Different Servers

6. © W. Wahlster The Basic Structure of an Adaptive Software System adapted from: P. Brusilovsky, M. Maybury: From Adaptive Hypermedia to the Adaptive Web, CACM, May 2002, Vol. 45, Nr. 5 System Data about user Collects User Modeling User Model Constructs Produces Adaptation Effect Adaptation

7. © W. Wahlster Personalization = Adaptation of a Webpage to a Particular User Personalization = Presentation of a Webpage by a Life-Like Character Two Meanings of Personalization

8. © W. Wahlster Information Retrieval IR Personalized Web Interaction: A Transdisciplinary Field Personalized Web Interaction Human Computer Interaction HCI Artificial Intelligence AI

9. © W. Wahlster Major Application Areas for Personalized Web Interaction Mobile Services E-Learning Info- tainment Recommender Systems E-Business Adaptive Navigation and Search Adaptive Content Selection Adaptive Presentation

10. © W. Wahlster ? e.g. 60 x 90 pixel b/w e.g. 1024 * 768 pixel 24-bit color The Need for Personalization: Adaptive Interaction with Mobile Devices

11. © W. Wahlster From Explicit to Implicit Input for User Model Acquisition Self-Reports on Personal Characteristics cf.: A. Jameson, Designing Systems that Adapt to their Users, CHI 2002 Self-Assessments on General Dimensions Evaluation of Specific Items Response to Test Items User Model Acquisition Previously Stored Information Naturally Occuring Actions Evidence for Affective And Cognitive States Evidence about Context Explicit Implicit

12. © W. Wahlster A “Web of Meaning“ has more Personalization Potential than a “Web of Links“ Three Layers of Webpage Annotations cf.: Dieter Fensel, James Hendler, Henry Liebermann, Wolfgang Wahlster (eds.) Spinning the Semantic Web, MIT Press, November 2002 Personalization Potential OWL DAML + OIL Content high Structure XML medium Layout HTML low

13. © W. Wahlster Personalization Mapping Web Content Onto a Variety of Structures and Layouts From the “one-size fits-all“ approach of static webpages to the “perfect personal fit“ approach of adaptive webpages Structure XML 1 XML 2 XML n Content OWL Layout HTML 11 HTML 1m HTML 21 HTML 2o HTML 31 HTML 3p

14. © W. Wahlster The Architecture of WIP (cf. [Wahlster et al. 91]): The First System for the Plan-based Personalization of Presentations Document Design Plan Graphics Design Graphics Realization Text Design Text Realization Presentation Planner Layout Manager Illustrated Document Illustrated Document Presentation Strategies TAG Graphics Design Strategies... User Model Basic Ontology RAT Presentation Goal Generation Parameters Mower Espresso Machine Modem Application Knowledge coded in RAT

15. © W. Wahlster Component Technologies for Personalized Web Interaction User Modeling Machine LearningIntelligent TutoringAdaptive LayoutPresentation PlanningLanguage Technology Agent TechnologyOntologiesHypermedia Personalized Web Interaction

16. © W. Wahlster MyArt www Portal Webpages Content Authors Portal Provider Content Customers Tailoring to the constraints of various display devices (cf. EU project IMAGEN: www.imagenweb.org) Creating Personalized Webpages

17. © W. Wahlster Creating a New Webpage from Available Media Objects 1. Content Selection 2. Content Packaging 3. Layout Revision

18. © W. Wahlster ensure-contrast title-size-min title-size-hierarchy regular-less-title regular-size-equals Finding Adequate Layouts Options by Constraint Solving Package CSP Finite domain constraint solver Possible Solutions

19. © W. Wahlster Example of a Layout Constraint (Kröner 2002) Purpose –use a title font that is larger than all fonts of subtitles Approach –Add a binary constraint between the related font sizes Constraint Formulation <lm:constraint name = “title-size-hierarchy” src = “JCL.BC_ID_LessThan” target-var = “lm:font-size” target-node = “//ims:title” source-node = “../ancestor::*/child::ims:title” />

20. © W. Wahlster Exploiting User Preferences to Select a Solution Possible Solutions DeliverySelecting a Solution

21. © W. Wahlster AiA: Information Integration for Virtual Webpages Yahoo Weather Server PAN Travel Agent Andi Car Route Planner Yahoo News Server Gault Millau Restaurant Guide Hotel Guide

22. © W. Wahlster Trip Data The Generation of Virtual Webpages with PAN and AiA Netbot PAN Pictures and Graphics Pieces of Text Coordinates for Pointing Gestures Input for Speech Synthesis Icons for Hyperlinks Hotel Agent Address Weather Agent Train & Flight Scheduling Agent Major Event Agent Virtual Web Presentation AiA Constraint- based Online Layout Presentation Planner Persona Server Components of virtual Webpages Map Agent

23. © W. Wahlster Persona as a Personal Travel Consultant

24. © W. Wahlster Information Structures Relations, Lists KR Terms Multi-Domain Problem Specs NETBOT Retrieved Results Media Objects Texts, Sounds, Videos Pictures, Maps, Animations Distributed Information Multiple Data Sources The Combination of Retrieved and Generated Media Objects for Virtual Webpages

25. © W. Wahlster Select & Design Select Canned Media Objects Design New Media Objects Graphics, Animation Text, Speech, Facial Expr. Icons, Clip Art Frames, Sounds Reuse & Transform Coordinate Media Objects Transform Media Objects Temporal Synchroni- zation Spatial Layout Clip, Convert, Abstract Zoom, Pan, Transition Effects The Combination of Retrieved and Generated Media Objects for Virtual Webpages Information Structures Relations, Lists KR Terms Retrieved Results Media Objects Texts, Sounds, Videos Pictures, Maps, Animations

26. © W. Wahlster Tasksself-behavior Script Presentation Planner Persona Player Events & UserIntervention Our Approach to the Autoanimation of Life-Like Characters

27. © W. Wahlster Describe-item S-SpeakS-Point S-Include- image Describe-feature It has broad tires S-Include-images S-Include- header S-Include- Hyperlink Introduce illustrate Design-Intro- Page Describe item S-Speak Emphasize Welcome to the Porsche website 911 & Boxter The New Models The planning process decomposes complex presentation tasks until elementary presentation acts are worked out

28. © W. Wahlster e.g., describe a location determine scriptdetermine acts presentation task play presentation From the Presentation Task to the Executable Script

29. © W. Wahlster APMLSMIL Agent Script PET- PML MPEG4 Greta MS-Agent Controller Java Persona Player PET SMIL Player (Real, QuickTime) Presentation Planner DFKI’s Presentation Planner Can be Used to Generate Scripts for Various Player Technologies

30. © W. Wahlster Plan-based Generation of Input to a MPEG4 Player (e.g., Greta-Player by C. Pelachaud) Affective Presentation Markup Language (de Carolis et al.) I have been in jail for a very long period. FAP(Facial Animation Parameter) 2.1 a1_1_2 25 163 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 0 0 2 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 9 9 -1 -1 9 10 7 0 0 0 0 0 0 0 0 0 0

31. © W. Wahlster Personalization Through Multimodal Dialogue Agents: The User Converses with a Presentation Agent August (KTH) Cyberella (DFKI) Peedy (Microsoft) REA (MIT) Smartakus (DFKI) …

32. © W. Wahlster SmartKom`s SDDP Interaction Metaphor SDDP = Situated Delegation-oriented Dialogue Paradigm User specifies goal delegates task cooperate on problems asks questions presents results Service 1 Service 2 Service 3 Webservices Personalized Interaction Agent See: Wahlster et al. 2001, Eurospeech

33. © W. Wahlster SmartKom-Public: A Multimodal Communication Kiosk SmartKom-Mobile: A Handheld Communication Assistant SmartKom: A Transportable Interface Agent Media Analysis Kernel of SmartKom Interface Agent Interaction Management Application Manage- ment Media Design SmartKom-Home/Office: Multimodal Portal to Information Services

34. © W. Wahlster Personalized Interaction with WebTVs via SmartKom (DFKI with Sony, Philips, Siemens) User: Switch on the TV. Smartakus: Okay, the TV is on. User: Which channels are presenting the latest news right now? Smartakus: CNN and NTV are presenting news. User: Please record this news channel on a videotape. Smartakus: Okay, the VCR is now recording the selected program. Example: Multimodal Access to Electronic Program Guides for TV

35. © W. Wahlster - different role castings, e.g. to emphasize: - different point of views / level of expertise, teacher/student, seller/buyer, - the indexing of contributions, e.g. according to: - Type of info: “A reports on X, B on Y” - Source of info: “A says X, B says Y” Require: - appropriate distribution of contents to the different characters - believable dialogues Presentation Teams

36. © W. Wahlster Presentation Teams in the MIAU Project (Wahlster, Rist, André, Baldes 2001)

37. © W. Wahlster The Adaptation of Presentation Agents in MIAU Character-Centered Approach –Story is not defined by a script, but by the character‘s role, personality, status, attitude etc.

38. © W. Wahlster Interactive Presentation Teams in MIAU (DFKI) Hello, I am Peter.

39. © W. Wahlster 2002 Non-interactive presentation Hyper- presentation / dialogue Presentation teams interactive performances, iTV multiparty, multithreated conversations PPPAIAMIAU The Development of Personalized Webpages

40. © W. Wahlster Personalized Multiparty Interaction with Multiple Interface Agents

41. © W. Wahlster Tailoring Web Presentations to the User's Affective State IBM's prototype Emotion Mouse measures heart rate, temperature, general somatic activity, and the galvanic skin response. In the place where the fingertips touch there is an area sensitive to sweat. Movements are analyzed by a sensor next to the mouse's ball. Heartbeats are measured by the pulse of the thumb. Inside the instrument there is a thermometer that measures the hand's temperature.

42. © W. Wahlster Using Facial Expression Recognition for Affective Personalization (1) Smartakus: Here you see the CNN program for tonight. (2)User: That’s great. (3)Smartakus: I’ll show you the program of another channel for tonight. (2’)User: That’s great. (3’) Smartakus: Which of these features do you want to see? Processing ironic or sarcastic comments  

43. © W. Wahlster MP3 music files from the Web Rist & Herzog for Blaupunkt Personalized Car Entertainment (DFKI for Bosch)

44. © W. Wahlster Peter Brusilovsky, Oliviero Stock, Carlo Strapparava (eds.) Adaptive Hypermedia and Adaptive Web-based Systems, First International Conference, AH 2000, Trento, August 2000, LNCS 1892 Peter Brusilovsky, Mark Maybury (eds) The Adaptive Web. CACM, Vol. 45, Issue 5, May 2002 Peter Brusilovsky, Carlo Tasso (eds.): Special Issue on User Modeling for Web and Hypermedia Information Retrieval, In: User Modeling and User- Adapted Interaction: The Journal of Personalization Research, Kluwer, 2002 Jameson, Anthony: Designing Systems that Adapt to Their Users. Tutorial presented at CHI 2002, Minneapolis, May 2002, ACM, Tutorial Notes 13 Paul de Bra, Peter Brusilovsky, Ricardo Conejo (eds): Adaptive Hypermedia and Adaptive Web-based Systems, Second International Conference, AH 2002, Malaga, May 2002, LNCS 2347 Recent Overviews about Personalized Web Interaction

45. © W. Wahlster The goal of adaptive personalization is to increase the usage and acceptance of ubiquitous webservices. The intelligent adaptation to cognitive and technical resource limitations of the user is an important prerequisite of user-friendly web interaction. Plan-based and constraint-based methods are now available for the advanced personalization of adaptive and virtual webpages. The Semantic Web is increasing the potential for the effective personalization of webpages. Conclusions