1. D AFFODIL Strategic Support Evaluated Claus-Peter Klas Norbert Fuhr Andre Schaefer University of Duisburg-Essen

2. 15.09.2004University of Duisburg-Essen Outline Introduction to D AFFODIL Evaluation Procedure Hypotheses Main Results Outlook

3. 15.09.2004University of Duisburg-Essen DAFFOFIL is Virtual Digital library with strategic support given by high-level search functions an agent-based system with currently over 35 different agents providing the low-level and high-level functions connects ~15 data sources for searching and browsing from the computer science area

4. 15.09.2004University of Duisburg-Essen

5. 15.09.2004University of Duisburg-Essen DAFFODIL: Concepts Strategic Support Scientific Workflow Personalization Personal library Collaboration Awareness & Annotation Proactivity

6. 15.09.2004University of Duisburg-Essen Strategic Support is provided by high-level functions like metadata search duplication elemination citation search author network based strategems journal and conference run in a integrated way

7. 15.09.2004University of Duisburg-Essen Scientific workflow Choose data source structured storing interpret information create new knowledge retrieve knowledge

8. 15.09.2004University of Duisburg-Essen Structured folders document metadata fulltext authors conferences journals websites queries Collaboration through group folders Personalization: Personal Library

9. 15.09.2004University of Duisburg-Essen Personalization: Awareness & Annotation Long term queries: Query-Profiles: interval processing Authors: new publications Journals, conferences: new issues, proceedings See whats new: Message via Email, Fax, SMS,... Annotation for scientific discurs

10. 15.09.2004University of Duisburg-Essen Proactivity Contextual monitoring of users & system Direct action Spellchecking Authorname completion Related terms Recommend next step based on current stage based on collaborative usage

11. 15.09.2004University of Duisburg-Essen Evaluation framework Formal bases given by [Saracevic&Covi] Why to evaluate ? What to evaluate ? How to evaluate ?

12. 15.09.2004University of Duisburg-Essen D AFFODIL Evaluation Scheme Hypo I: More effective IR through strategic support Hypo II: More effective IR through (pro)-active system Hypo III: More effective IR through personalisation and collaboration Criteria Effectiveness Efficiency Measure Time Relevance User Satisfaction

13. 15.09.2004University of Duisburg-Essen Evaluation Phases 1st Phase 14 students in parallel sessions System efficiency & bug hunting 2nd Phase 8 researchers Focus on user: Efficiency and Effectiveness 3th Phase (expert phase) 3 librarians, 3 professors Comments, critics, hints

14. 15.09.2004University of Duisburg-Essen Gathering Evaluation Data Tasks on questionnaire 1st phase: protocolls Other phases: video capturing of monitor and face store usage see reaction store comments 11h video data

15. 15.09.2004University of Duisburg-Essen Real world task classified 5 Tasks (4 presented) on 4 levels, classified by a „information need typology matrix“ work task information need no knowledge full knowledge one object heterogenous set of objects 1L 2L3L 4L

16. 15.09.2004University of Duisburg-Essen Result (1st Phase) Phase 1: 14 students Error hunting, system efficiency Outcome: speed up 5-7 times Find a lot of bugs Comment: „Even so slow when searching, i want to use it, since it saves much time, handling the found knowledge“

17. 15.09.2004University of Duisburg-Essen Main Hypotheses more effective and efficient IR through strategic support given with DAFFODIL compared to other (DL) information system more effective, because user can cognitivly process and interpret information more easily more efficient, because user needs to invest less time

18. 15.09.2004University of Duisburg-Essen Result (2nd phase) Task 1: Known item instantiation (L1) Find document by given title (precise) „Search strategies in content-based image retrieval“

19. 15.09.2004University of Duisburg-Essen Task 2: Known item instantiation (L2) Find document by given title (not precise) „Methods of automated reasoning“

20. 15.09.2004University of Duisburg-Essen Result Task 3: A real search (L3) Find articles by author X in the area of digital libraries within the last 4 years. Results All participants found with Daffodil at least 12-19 relevant articles. With Google only two persons found articles any articles, the other gave up. Actually, user said, it will be hard with google prior to search.

21. 15.09.2004University of Duisburg-Essen Result Task 5: New area (L4) find literature about a specific subject: interactive information retrieval;information searching; information seeking not known by any participant prior to the evaluation Explorative Search Scanning result lists; related terms feature Results: Participants at least found a couple of entry points, to further search Got very soon an idea, what they don‘t want.

22. 15.09.2004University of Duisburg-Essen Result (3th phase) Experts: Free Search Three librarians and three professors Free Search with own theme Results: input for improving satisfied Comments: librarian „rich functionality“ „D. mediates the knowledge about the tactics to the user“ professor „D. seems in a state that it can be used“ „I will propose it to my assistents“

23. 15.09.2004University of Duisburg-Essen Summary of evaluation Evaluation with 28 participants showed, that a user is more effective when searching for scientific literature with the strategic support, here given by the DAFFODIL system, but not always more efficient (task1). But, as the tasks become more complex, the hypotheses hold.

24. 15.09.2004University of Duisburg-Essen Outlook Continue Evaluation other systems Long-Term Evaluation One task from start to end new paper for ECDL 2005 Participants are welcome Try it out http://www.daffodil.de