1. Usability Engineering: Formative Evaluation of NSDL Projects Dr. Manuel A. Pérez-Quiñones Dept. of Computer Science Virginia Tech Blacksburg VA perez@cs.vt.edu | www.citidel.org Dr. Manuel A. Pérez-Quiñones Dept. of Computer Science Virginia Tech Blacksburg VA perez@cs.vt.edu | www.citidel.org

2. Usability Engineering Methods Formative evaluation - evaluation of interface as it is being designed We have used formative evaluation methods to several existing Digital Libraries: iLumina, NCSTRL, CITIDEL The results showed some consistency on types of errors across digital libraries Wording, user’s mental models, support for user tasks Formative evaluation - evaluation of interface as it is being designed We have used formative evaluation methods to several existing Digital Libraries: iLumina, NCSTRL, CITIDEL The results showed some consistency on types of errors across digital libraries Wording, user’s mental models, support for user tasks

3. Why Usability Evaluation? You can’t manage what you can’t measure What to measure: throughput, error rate, task completion, completion rate, user satisfaction, learnability, etc. Why? Interactive applications have high development cost and acceptance often depends on usability You can’t manage what you can’t measure What to measure: throughput, error rate, task completion, completion rate, user satisfaction, learnability, etc. Why? Interactive applications have high development cost and acceptance often depends on usability

4. Usability Engineering Methods When?Who?Why? Inspection Early and deployed designs Done by expert Predict usability problems Heuristic Evaluation Best for early designs Done by evaluators Evaluate using to principles and guidelines Lab-based Evaluation Deployed designs or advanced prototypes Done by evaluator with users Evaluate some tasks “Use them early and use them often”

5. Inspection Findings 32 usability problems found using co- inspection

6. Heuristic Evaluation Asked users to evaluate interface with respect to given rules/heuristics Sample heuristics Use single natural language; speak the user’s language; minimize memory load; be consistent; provide feedback; provide clearly marked exits; etc.

7. Heuristic Evaluation Results Done with Jr/Sr Computer Science students User language: cc, ccs, corr, msc, citidel, acm, nsf, nsdl, siguccs, collection, digital library… 12/3/03 - NSDL Opening, CITIDEL debuts! “What does this mean? What is NSDL?” …allows you to sequence resources “What does ‘sequence’ means?”

8. Wider User Testing: CITIDEL

9. Conclusions Common problems found: technical jargon, understanding users’ mental model, search vs. browsing usability, portal pass-through problems Understanding of the potential of UE methods when applied at the end of the project we can only give you a “ok or not ok” answer, however …when applied earlier, it can help formulate and validate design solutions Common problems found: technical jargon, understanding users’ mental model, search vs. browsing usability, portal pass-through problems Understanding of the potential of UE methods when applied at the end of the project we can only give you a “ok or not ok” answer, however …when applied earlier, it can help formulate and validate design solutions