1. An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen WanPeterWilkins Ross Wilkinson Mingfang Wu

2. Outline Problem statement System Description Experiment Discussion

3. Finding Information about a Corporation Often done through the web, searching and browsing Differences between results from such search-and- browse and printed brochures: –Printed brochures are coherent documents – and include more than just information about a specific topic –Printed brochures are typically tailored to an audience

4. Challenges in accessing an organization’s website Browse –know the organization’s structure –know the web data structure –otherwise -> users feel lost Search –It works for the “right” query or someone with the “right” mental model how querying/searching works –Otherwise, it may cause zero results or too many results –Result lists is disorganized - each result is presented independently, although each result may be a piece of the overall picture

5. Our research question and approach Can a tailored hypermedia system offering tailored and coherent information provide a good alternative to search-and- browse? Approach: Given a modest user model and a query: –Present information that not only matches the query but also is tailored to the user. –Present other information that is relevant to the context of the query (here information about an organization), like printed brochures do (e.g., introduction, contact details, etc.) –Deliver the result as a concise and readable summary. –Note that the summary may not cover all information a user needs, but is an informative guide that will lead the user to the needed information.

6. Our Tailor Hypermedia System: Percy Data Interface User Model Stereotype Individual Virtual Document Planner Discourse Rules Content Planner Data Sources ‘Documents’ VDP Data Interface Interface Specification Grammar(s) Lexicon(s) Templates Stylesheets Surface Realiser Presentation Rules Presentation Planner

7. Percy – user model Based on a requirements analysis Contains: –Stereotypical information Current job position (e.g., scientist or CEO) Industry sector (e.g. science, engineering, finance) –Individual information Interests (as represented by a query term, e.g., language technology) Preferred delivery channel (e.g web page, or hand-held device)

8. Percy – discourse planner Discourse rules or Schemas –They indicate which information to present and how to organize it –They are used by the discourse planner to build a discourse tree –Each rule extends a branch in the discourse tree –Coherence relations amongst siblings indicate how the information fits together

9. Percy – presentation planner Discourse tree (which indicates content and organization) is extended with presentation markup Uses presentation plans P Palm C P Web C

10. Percy – surface generator Traversal of final discourse tree to generate document Conversion from tree markup of layout to device dependent markup –Device dependent markup may include use of further style sheets Surface Realisation XSLT templates ~~~~~~~~~ P Web C

11. PERCY – an example delivery to a scientist Introduction to CSIRO Short introduction to the divisions that have the projects related to the query Show the expanded query Short description of the matched project

12. PERCY – an example delivery to a scientist Link to the detailed description Link to the online demonstration Publications Contact person

13. PERCY – an example delivery to a CEO Short introduction of how to do business with CSIRO

14. PERCY – an example delivery to a CEO The project description and business relevance

15. Preliminary Evaluation Question Is this coherent and tailored delivery is useful interface to organizational websites ?

16. Possible evaluation criteria User’s interactive experience –format –content User’s attitudes towards the received message –intention The influence of the delivery message –The further interaction with the corporate

17. Experimental setting Test systems: PERCY vs. search-and-browse mechanism: a website search engine (Panoptic) Subjects: 20 university students Search topics: –T1: Human computer interaction –T2: Mathematical modeling –T3: Image analysis –T4: Language technology

18. Evaluation instruments Post-system questionnaire (PSQ) –To get users’ opinion on the evaluation criteria independently on each test system Exit questionnaire (EQ) –To get users’ opinion of the second system compared with the first system All questions are evaluated on a 5-point Likert scale

19. Experiment design Latin square design to reduce the sequence effect of systems SubjectsSystem, topic No. 1-5Percy(t1,t2) PSQ Search (t3,t4) PSQ EQ No. 6-10Percy (t3,t4) PSQ Search (t1,t2) PSQ EQ No. 11-15Search(t1,t2) PSQ Percy (t3,t4) PSQ EQ No. 16-20Search (t3,t4) PSQ Percy (t1,t2) PSQ EQ

20. Post-system questionnaire – Measuring the content of the delivered information Q1: The system provides me accurate information Q2: The system provides sufficient information Q3: The information provided by the system meets my need. Q4: The system provides me comprehensive information Q1Q2Q3Q4 Search3.503.603.353.70 Percy3.653.303.653.55

21. Post-system questionnaire – Measuring the format of the delivered information Q5: The structure of the presented information is clear to me. Q6: I think the presented information is organized in a useful format. Q7: I think the presented information serves well as a useful online brochure. Q5Q6Q7 Search3.653.502.75 Percy4.003.803.50

22. Post-system questionnaire – Understanding the intention of the user Q8: I would like to get more information on some specific projects presented. Q9: I would like to get more information because I have not got the needed information. Q8Q9 Search3.953.15 Percy3.902.90

23. Exit-questionnaire – Evaluating users’ preferences Q1Q2Q3Q4Q5Q6Q7 Search2.703.003.102.803.303.502.70 Percy3.703.603.504.20 3.404.00 P <0.010.030.04 focus on the search topic better make search task easier deliver higher quality info attract attention better provide better explanation provide more comprehensive info preference

24. Summary of experimental results With Percy, subjects can better focus on their tasks. With Percy, subjects understand better why a piece of information is presented. Subjects preferred Percy.

25. Discussion About tailored hypermedia (anything to say about the delivery method itself or limitations?) –Delivery – who is in charge, user/author? or it is situation dependent? –How to apply AH to a dynamic resource (or even a different domain) with minimum effort? –Is user’s expectation of AH system different from IR? About the evaluation –What to compare – Is it fair to compare the adaptive system with the non-adaptive system? Or should we compare two adaptive systems? –Will the preference brings the enhancement of user performance? –(How to measure long term benefit?)