Presentation is loading. Please wait.

Presentation is loading. Please wait.

User-centered Information System Evaluation Tutorial.

Similar presentations

Presentation on theme: "User-centered Information System Evaluation Tutorial."— Presentation transcript:

1 User-centered Information System Evaluation Tutorial

2 Goal Understand how to do user-centered information system evaluation Using query performance of different queries generated by users to measure the efficiency of different information retrieval systems. Evaluating systems using precision and recall based measures

3 Example Scenario: Finding expert is important, especially when you want to find potential collaborators, ask for consultancy, look for potential reviewers, etc. Aim: comparing two expert finding systems – Pair 1: vs. – Pair 2: vs. – Pair 3: Google vs. Yahoo Goal: which one is a better expert finding system based on the following criteria – easy to use, user-friendly – quality of search

4 Quantitative data analysis

5 How Approach (2 persons form a group) – Demographic questionnaire About your user, whether he/she is familiar with this system – Pre-evaluation questionnaire whether you are familiar with the topic – Evaluation Easy to use, user-friendly – Develop a survey/questionnaire with scale questions and 1-2 open-end questions – E.g., easy to find help file, easy to know where you are, focusing on the design of the system Quality of search (2 queries) – Pair 1: Q1: find experts working on “stem cell”, Q2: create another query – Pair 2: Q1: find experts good at “data mining”, Q2: create another query – Pair 3: Q1: create your own query, Q2: create another query – Post-evaluation questionnaire Usability – Whether the system is easy to search, does it take too long to get the result, whether the results are ranked, whether each ranked URL has short summary – Exit questionnaire preference

6 Measuring quality of search QuerySyste m RankPage titleRelevance Judgment (0/1) Q1: ?????? ? Same for Query 2

7 User judgment of relevance RankPage title + short summary (random order) Relevance Judgment (0/1) Fill in the following table with the title and short summary of top 5 results Show the results to your partner and let your partner decide the relevance (0=irrelevant, 1=relevant)

8 Calculate Query System IDEvaluation Metric Q1 System = = System = = Q2 System = System 2 = RankRelevance Judgment (0/1) System ID Evaluation Metric System = = System = =

9 Compare your results with your partner’s Will your results be consistent with your partner’s? – Compare the and – If your results are consistent, you can draw the conclusion, which system performs better – If your results are not consistent, what can you draw, why? – Limitation and potential reasons.

10 Evaluation Report Write a short report (1-2 pages) about this evaluation – Including the evaluation outputs – Quantitative analysis – Qualitative analysis – Final conclusions – Limitation – Lessons-Learned – If you want to have this as a real evaluation, what can you improve.

Download ppt "User-centered Information System Evaluation Tutorial."

Similar presentations

Ads by Google