1. Evaluation of an Information System in an Information Seeking Process Lena Blomgren, Helena Vallo and Katriina Byström The Swedish School of Library and Information Science

2. Holistic evaluation of an IR system: u Effectiveness of the system (system perspective) u Usefulness of the system (user perspective) u Integration of the system (contextual perspective)

3. Information seeking and searching (and retrieval) embedded in a work task Legend: IST stands for information-seeking task; IRT stands for information-search task Source: Byström & Hansen, 2002, 2004

4. Borlund’s IIR (Interactive Information Retrieval) Systems Evaluation Package u an appropriate research setting u empirical recommendations for simulation u alternative relevance measures

5. Setting The participants: 20 (280) reporters employed at different editorial departments at Göteborgs- Posten (GP) The information system: NewsLink, - manually indexed full-text database (1994) - employs the Boolean search technique - ranking both by date and relevance

6. Data collection u Questionnaires u Search protocols u Post-search interviews u Web form (all journalists)

7. Simulated work task situation: The Terror Attacks in USA September 11 th Some time has passed since the terror attacks in USA. GP is now planning a follow-up and a summarising series of articles on the subject, which will illustrate how different areas have been affected and the long term consequences. You have been asked to write an article that will illustrate this issue from your particular subject field.

8. Analysis Performance measures: Precision at DCV 15 : situational / perceived relevance - non-binary scale: not relevant = 0, partly relevant = 0.5, very relevant = 1 RHL (Ranked Half-Life): capability of ranking output according to users’ situational relevance assessments Novelty: ratio of known/unknown retrieved documents Satisfaction: level of satisfaction after task completion

9. Results The most important information sources:  oral sources (formal / informal)  Internet (official web sites / search engines)  NewsLink - an important information source - used in combination with other sources - used for checking what has been written in GP  other media

10. The respondents turn exclusively to NewsLink to:  check what GP has written, in order to - get an update on a subject (interviews, follow ups) - get background material for articles - avoid duplicate articles  find a specific already known article  find oral sources, names  check facts: years, names, places etc

11. What is the role of the system for its users, as a resource among other information sources available?  NewsLink is important for the journalists in their daily work, but does not function as their only or primary information source.  NewsLink is experienced as effective from the users’ point of view and judged as an important information source among others.

12. Results Precision and RHL index NewsLink generally performs better on the real work tasks and in ranking by relevance:  Real work task - familiarity with topic  Simulated work task - wide topic - difficult to place in time

13. Novelty/post-search Interview The relevance assessments are:  usually affected by whether or not a document is new to the user  depending on the work task at hand

14. How well do the relevance assessments of the system correspond with those of the users? Poor performance but effective from a user perspective:  High satisfaction but poor Precision and RHL-index values  Dynamic information needs affecting the values for Precision and RHL

15. Method evaluation  The simulated work task situation functioned well  Compose simulated work task situations with great care  Important to use at least one real work task  Evaluations should include the users of the system and user-oriented measures

16. The Performance Measures:  Similar results for RHL and Precision  The RHL measure indicates which system is most effective even where the Precision values are identical  RHL-indicator ought to be normalised into an RHL index value to produce more easily interpreted and comparable results

17. Novel results in evaluation context:  The type of information retrieved from the system - important for correct evaluation  As an information source among others available, the system can be satisfactory even if it does not provide all the information needed  Combining both user- and system oriented measures and adding post search interviews, enables a fuller and more accurate picture of the system, its users and context

18. Is the evaluation method applied well suited for evaluations of operational IR systems in real-life settings?  The approach has functioned well and provided a solid methodological base  The measures used complete each other  Ensures both realism and control  Covers system, user and context

19. General conclusions  Results for Satisfaction and Novelty confirm importance of qualitative measures in real life settings  Knowing the users and their context is a key factor in successful system design.  Systems must be well suited to their users

20. Recommendations for future work  Mapping what kind of information is retrieved from the information system  Study if and to what extent the information retrieved actually is used  Test whether the method used works as well in other user scenarios  Complement RHL, Precision, Novelty and Satisfaction with Relative Recall