Presentation on theme: "User-centered Information System Evaluation Tutorial"— Presentation transcript:
1 User-centered Information System Evaluation Tutorial
2 Goal Understand how to do user-centered information system evaluation Using query performance of different queries generated by users to measure the efficiency of different information retrieval systems.Evaluating systems using precision and recall based measures
3 ExampleScenario: Finding expert is important, especially when you want to find potential collaborators, ask for consultancy, look for potential reviewers, etc.Aim: comparing two expert finding systemsPair 1: vs.Pair 2: vs.Pair 3: Google vs. YahooGoal: which one is a better expert finding system based on the following criteriaeasy to use, user-friendlyquality of search
5 How Approach (2 persons form a group) Demographic questionnaire About your user, whether he/she is familiar with this systemPre-evaluation questionnairewhether you are familiar with the topicEvaluationEasy to use, user-friendlyDevelop a survey/questionnaire with scale questions and 1-2 open-end questionsE.g., easy to find help file, easy to know where you are, focusing on the design of the systemQuality of search (2 queries)Pair 1: Q1: find experts working on “stem cell”, Q2: create another queryPair 2: Q1: find experts good at “data mining”, Q2: create another queryPair 3: Q1: create your own query, Q2: create another queryPost-evaluation questionnaireUsabilityWhether the system is easy to search, does it take too long to get the result, whether the results are ranked, whether each ranked URL has short summaryExit questionnairepreference
6 Measuring quality of search QuerySystemRankPage titleRelevance Judgment (0/1)Q1: ???????12345Same for Query 2
7 User judgment of relevance Fill in the following table with the title and short summary of top 5 resultsShow the results to your partner and let your partner decide the relevance (0=irrelevant, 1=relevant)RankPage title + short summary (random order)Relevance Judgment (0/1)12345
9 Compare your results with your partner’s Will your results be consistent with your partner’s?Compare the andIf your results are consistent, you can draw the conclusion, which system performs betterIf your results are not consistent, what can you draw, why?Limitation and potential reasons.
10 Evaluation ReportWrite a short report (1-2 pages) about this evaluationIncluding the evaluation outputsQuantitative analysisQualitative analysisFinal conclusionsLimitationLessons-LearnedIf you want to have this as a real evaluation, what can you improve.