ICS 463, Intro to Human Computer Interaction Design: 8. Evaluation and Data Dan Suthers

Evaluation Formative: early and often; informs design Summative: near end; have we done well? We focus on formative

When and why? Early in the life cycle –understanding target application –check understanding of requirements –quick filtering of ideas Middle –predicting usability –comparing alternate designs –engineering towards a usability target Later –fine tuning of usability –verifying conformance to a standard

Preview of Methods of Evaluation Collecting users' opinions  attitudes Observing and monitoring use  how users interact Experiments  hypothesis testing Interpretive evaluation  how used in natural settings (ecological validity) Predictive evaluation  anticipated usability issues

Typical Procedure (formal evals) Identify questions Plan the evaluation Pilot the evaluation and revise if needed Run the sessions and collect the data Analyze the data Draw conclusions Redesign and revise system Details next week (Hix & Hartson)

Dimensions to consider Evaluation planning should consider Characteristics of users Types of activities Environment of use Nature of artifact So should design … from the outset!

Data Types of Data –Objective versus Subjective –Quantitative versus Qualitative What are the independent variables? Dependent variables? Controlled variables? Now on to details of evaluation methods …

Collecting Users’ Opinions Tells us about attitudes Caveat: "First rule of usability: don't listen to users, watch what they do!" Two major methods Interviews - qualitative analysis Surveys - quantitative analysis

Interviews Structured: –fixed questions, fixed or conditional sequence –easier to conduct and analyze –may miss opportunistic information Semi-structured –set of questions “to get to” Flexible –no set questions or sequence

Questionnaires and Surveys Large numbers, analyzed quantitatively Design with your analysis in mind Piloting important Closed questions versus open questions Types of closed questions –Checklists: background information –Likert scales: range of agreement or disagreement with a statement –Semantic differentials: place on scale of adjectives –Ranked order: e.g., rank in order of usefulness

Observing Users Observing and monitoring use of artifact –in laboratory –in natural setting  how users interact with system  usability issues

Direct Observation Researcher watches use, takes notes Hawthorne Effect (users act differently under observation) may contaminate results Record may be incomplete Only one chance Helpful to have shorthand and/or forms which which you are fluent

Indirect Observation Video logging –User(s) body language, gestures –Screen activity Two uses: –Exploratory evaluation: review tapes carefully and repeatedly to discover issues –Formal studies: know what you are looking for! Interaction logging (software) –Often use two or more together –Must synchronize all data streams High volume of data can be overwhelming

Verbal Protocols Audio record of spoken language –Spontaneous utterances –Conversation between multiple users –Think-aloud protocol –Post-event protocols Dangers of introspection, rationalization Analyze along with video

Video/Verbal Analysis Diversity of approaches Task-based –how do users approach the problem –difficulties in using the software –need not be exhaustive: identify interesting cases Performance-based –frequency and timing of categories of actions, errors, task completion

More on Analysis of Video/Verbal Requires classification scheme, invented or borrowed May involve inter-rater reliability Often exhaustive and time intensive! Tools important –we transcribe conversation to text merged with transaction log –better approach would be direct analysis of digital video

Software Instrumentation/Logging Time-stamped logs –key-presses or higher level actions –record what happened but not replayable Interaction logging –replayable Synchronize with video data for rich but overwhelming data Analysis issues are similar

Methods of Evaluation Collecting users' opinions –  attitudes Observing and monitoring use of artifact –in laboratory –in natural setting –  how users interact Experiments –testable hypothesis –comparative with controlled variables –quantitative analysis Interpretive evaluation –how used in natural settings –qualitative data and analysis Predictive evaluation –predict usability issues based on model –applied to specifications or prototypes

Additional Methods Experiments –testable hypothesis –comparative with controlled variables –quantitative analysis –we’ll get into this next week Interpretive evaluation –how used in natural settings –qualitative data and analysis Predictive evaluation –predict usability issues based on model –applied to specifications or prototypes

Assignment 7 (project groups) The good news: your project has been accepted by The Boss! The bad news: you have 6 weeks to finish it!

Assignment 7 continued Write a 2 page plan expressed in terms of the Star model –What is the expected product? –At which phase will you start? –What is your estimated timeline? –How will you incorporate evaluation and other user-centered techniques, and use the outcomes to adjust the design? –What resources do you need? –This can be in outline form and will be assessed for its value as an efficient briefing for The Boss