Presentation on theme: "Copyright D. Petkovic1 Software Engineering CSc640/848 Fall 2006 Usability Testing Prof. Dragutin Petkovic."— Presentation transcript:
Copyright D. Petkovic1 Software Engineering CSc640/848 Fall 2006 Usability Testing Prof. Dragutin Petkovic
Copyright D. Petkovic2 Important Human Characteristics related to GUI: Memory (Miller, 1995) Sensory memory: buffer that stores sensory input. Forgotten easily Short-term memory: holds limited amount of data for several minutes. Experiments showed that is holds up to seven concepts Long term memory: large storage capacity, but can be unreliable
Copyright D. Petkovic3 Important Human Characteristics related to GUI: Memory Recall: getting information from long term memory. It is hard Recognition: selecting from multiple choice or recognizing a pattern. Easier – Leverage speed of short-term memory by proper graphics and layout –Use pull down menus to “remind” people and help access long term memory
Copyright D. Petkovic4 Important Human Characteristics related to GUI: Information processing High level –Identified with conscious thinking –Slow, sequential, logical –Used for reading and understanding Low-level –Rapid information processing, “pattern processing” –Without conscious effort –Look vs. seeing it, perceive vs. read If you know the screen you simply verify it (fast, easy). If you are novice, you read it (much slower) leverage this by proper screen design, make it easily recognizable and distinct, reduce clutter and ambiguity
Copyright D. Petkovic5 Important Human Characteristics related to GUI: Vision Visual acuity: capacity to resolve details Foveal and peripheral vision Relative acuity is halved at 2.5 degrees from the center influences character spacing on the screen etc. Eyes jitter (this increasing accuracy). However: Some patters on the screen can seem jittery Care has to be taken in designing patterns so as not to cause jitter and visual distraction Busy periphery can distract main foveal vision implications on screen design Acuity drops with age Font type contrast and size to be adjusted for older users. Same for URLs
Copyright D. Petkovic6 Important Human Characteristics related to GUI: Mental models Users have mental models of everyday things as well as established computing models (file systems, document model, transaction model, cars, TVs etc.). If GUI matches user or established mental model with the applications it is easier to learn, feels more intuitive etc. It is highly recommended to develop GUIs and applications with established models for the intended application and stick to them
Copyright D. Petkovic7 Important Human Characteristics related to GUI: Movement Control Keyboard, mouse, trackball usage involves movement Fitt’s Law (1954) is one of the rare exact laws related to usability: Time-to-acquire-target = a + b*Log2(2A/W) (A is distance, W is width of the target) Large objects for important functions Group related buttons Pining action of the sides, top, bottom and corners of screens Avoid small clickable objects
Copyright D. Petkovic8 Important Human Characteristics related to GUI: Learning Moving information from short to long term memory Designs that minimize the need for learning accelerate human performance Long learning times are unproductive People like to explore and do trial and error Inconsistencies in GUI frustrate people Allow transfer of skills from application to application, I.e. consistent GUI design and behaviors Provide feedback Allow UNDO (then users feel fere to explore and learn) Phase the information: people need only particularly needed info to accomplish the current task
Copyright D. Petkovic9 Important Human Characteristics related to GUI: Interactivity Response TimeUser reaction and perception < 0.1 sec.Instantaneous, no interruptions < 1.0 sec.Delay noticed, but no break in thought stream > 10.secInterruption, user switches to another task
Copyright D. Petkovic10 Important Human Characteristics related to GUI: People differences We are all different: motor and perception abilities, skills, education, learning abilities, age etc. How to design for wide population? –Lowest common denominator? –Provide varying levels of GUI that can be adjusted to various people –Allow people to customize
Copyright D. Petkovic11 User/task considerations (for categorizing and surveying users) Knowledge and Experience Computer Literacy System experience Application experience Task experience Education Reading level Typing skill Native language or Culture
Copyright D. Petkovic12 Types of users vs. skill level Novice users Depend on systems features to assist (help etc.) Restricted vocabularies Simple tasks Need learning time Sometimes high “startup” curve that may lead to dropping the SW Novice users are never “stupid”. Systems must be designed for them. They can be your key market segment especially in early stages
Copyright D. Petkovic13 Types of users vs. skill level Expert users Leverage experience Use recall (fast) rather than recognition Fast Need less help and feedback Like to reduce keystrokes, have shortcuts etc. Allow experts to leverage their expertise. Systems also need to be design to accommodate them. How to design good UI for novices and expert users?
Copyright D. Petkovic14 User/task considerations (for categorizing and surveying users) Job/task/Need Type of system use Frequency of use Task importance Task structure Social interactions Training Turnover rate Job category Lifestyle
Copyright D. Petkovic16 User/task considerations (for categorizing and surveying users) Physical characteristics Age Gender Handedness Disabilities
Copyright D. Petkovic17 Human interaction speeds Reading: paper 200 w/minute; computer 180 w/min Listening: 150 w/minute Typing (classic): 60 w/minute average Keyboard typing: transcripts 33 w/minute; composition 19 words/minute Hand printing: memorized text 31 w/minute; copying 22 w/minute Expects system response at better than 0.5 sec. To be felt fully interactive
Copyright D. Petkovic18 About users User will not be able to tell you how to design good GUI BUT They will immediately tell you if your GUI is good or bad You are in charge of the design and you have to iterate with users
Copyright D. Petkovic19 More about the users At work, they like to do the task in shortest possible time, with as little learning as possible User perception IS the reality, no matter how wrong they are Long text messages and long help files are not read Nobody reads the documentation Users are getting accustomed to well designed professional SW Users also learned standard Windows and WWW behavior, and expect same metaphors and quality from any other SW. Leverage this! Nobody likes surprises User opinion is formed at the first point of contact (order, install…) Novel applications have to deal with “startup” learning Poorly designed or behaving WWW sites are dropped immediately, usually forever Nobody likes to install plug-ins any more Users expect very easy installation and upgrades, with ability to use old data files etc. (legacy issues)
Copyright D. Petkovic20 Usability Testing and Evaluation
Copyright D. Petkovic21 What can be tested/validated All components of requirements, conceptual design and high and low level design: –Requirements –Use cases –Sketches, mockups, storyboards, prototypes, docs, help Particular isolated features Almost ready products Beta products, release candidates…
Copyright D. Petkovic22 Some types of testing/validation 1.Surveys: good to survey customers to validate requirements 2.Heuristic: performed by GUI expert. 3.Cognitive walkthrough: performed by GUI developers who execute test tasks 4.Think-Aloud evaluations: Users talk as they use the systems, feedback is recorded 5.Usability tests: performed by users following test plan questionnaire 6.Experiments: certain parameters are measured (I.e. response time etc.) Suggestion: must do 1-3 in house; do 5 with as many users outside of development; do 6 if needed
Copyright D. Petkovic23 What is tested/validated Validation of requirements Validation of style guidelines and basic assumptions Identification of design problems Usability –Ease of use –User satisfaction –Time to complete tasks –Error rates –Performance Reliability (via stress testing) Typical QA (defects, robustness, corner cases…) Compliance with standards (e.g. Microsoft logo certification) (Validate key high level decisions as early as possible)
Copyright D. Petkovic24 Usability testing: practical instructions We will present some practical and simple instructions for usability testing Note that for many culminating projects you will be required to do this Formal plan has to be submitted for review to SFSU Human Subject Protocol committee (Federal rules)
Copyright D. Petkovic25 Typical test plan format Purpose: what is the main purpose of the test Problem statement: specific questions you want resolved Test plan and objectives: tasks the user will do User profile: who will be the users Method and test design: how will you observe it, how will you collect the data etc. Test environment and equipment Test monitor role Evaluation measures and data to be collected: how will you collect the feedback and how will you evaluate it Report: what will final report contain
Copyright D. Petkovic26 Task selection for evaluation Tasks to be evaluated are functions users want to do with the product. Focus is on user view of the tasks and NOT at the components and details that are need it to accomplish it. This is deliberately left the the user to evaluate! Examples: –Create and file document –Import several images –Find the right document Objective is to indirectly expose usability flaws by asking the user to perform typical tasks and NOT telling them exactly how to do it. Choose key and most frequently done tasks The task has to be specific and measurable (quantitatively or qualitatively)
Copyright D. Petkovic27 Task components TASKDESCRIPTION TaskLoad paper into copier Machine statePaper tray empty Successful completion criteria Paper properly loaded BenchmarkCompleted in 1 minute Do NOT tell them how!
Copyright D. Petkovic28 Task description Correct: –Do X (I.e. load paper) Not correct: –Do X by clicking here, then using this, then typing that… It is deliberately left to the user to figure out HOW to accomplish the task and to grade ease of use, intuitiveness etc.
Copyright D. Petkovic29 Selection of evaluators and test groups Evaluators should be representative of the targeted users Independent groups or within-subject design (but be careful to avoid exposing users to same tests since this will bias the results) Adequate numbers of testers Offer motivation and rewards
Copyright D. Petkovic30 Measurements and Questionnaires Performance data: measures of user behavior such as error rates, number of accesses to help, time to perform the task etc. –Usually can and should be objectively and automatically measured Preference data: measures of user opinion, thought process such as rankings, answers to questions, comments etc. –Use questionnaires.
Copyright D. Petkovic31 Some performance measures (measure what can be measured) Time to complete each task Number and percentage of tasks completed successfully/unsuccessfully Time required to access information Count of incorrect selections Count errors Time for system to respond …. Data should be collected automatically or manually in an objective way.
Copyright D. Petkovic32 Summarizing performance results Performance data –Mean time to complete –Median time to complete –Range (high and low) –Standard deviation of completion times –System response time statistics Task accuracy –% of users completing the task within specified time –% of users completing the task regardless of time –Same as above, with assistance –Average error rate
Copyright D. Petkovic33 Summarizing preference results For limited choice questions –Count how many participants selected each choice (number and %) –For Likert scale or semantic differentials provide average scores if there are enough evaluators For free form questions –List questions and group answers into categories, also into positive and negative answers For free comments –List and group them at the end of the report
Copyright D. Petkovic34 Analyzing Data Identify and focus on tasks that did not pass the test or showed significant problems Identify user errors and difficulties Identify sources of errors Prioritize problems by criticality = severity AND probability of occurrence Analyze differences between groups (if applicable) Provide recommendations and prioritization at the end
Copyright D. Petkovic35 Problems statements and performance data to collect Problem Statement Performance Data Collected How effective is the tutorial Compare error rates of users who used and not used this How easy is it to perform task X Error rate OR Number of steps needed Note: this is Performance data measurement only. You also need to asses user Preference data (see next slide)
Copyright D. Petkovic36 Problems statements and preference data to collect Problem Statement Preference Data Collected How effective is the tutorial Ask user to rate it from very ineffective to very effective (Lickert scale or semantic differentials) + free comments How easy is it to perform task X Ask user to rate it from very easy to very difficult (Lickert scale or semantic differentials) + free comments
Copyright D. Petkovic37 Relate problem statements with tasks Problem Statement Task How effective is the tutorial GroupA: Import image w/o using tutorial GroupB: Same but use tutorial first How easy is it to Create Virtual Machine Create Virtual machine with “this” properties using New VM Wizard
Copyright D. Petkovic38 Task components TASKDESCRIPTION TaskCreate VM using New VM Wizard Machine stateVMware WS SW just loaded Successful completion criteria Working VM created BenchmarkCompleted in 30 sec.
39 How easy was it to create VM? Very difficult Average Very easy Question 15: Example questionnaire What is wrong here? Comments: << Previous Next >> END **** This OK? ****
40 It was easy to create VM? Strongly agree Neutral Strongly disagree Question 15: Example questionnaire CORRECTED Comments: << Previous Next >> END AgreeDisagree
Copyright D. Petkovic41 Some Suggested GUI issues to cover in preference data collection Use GUI principles as general measures of quality to evaluate Screen layout matches tasks Amount of information is adequate Good use of text (headlines, help,messages, warnings…) Good use of colors Proper grouping of related info Navigational problems Users get lost in the system Organized by user tasks Icons are self-explanatory Consistency Note: ask these questions in the context of concrete user tasks, not in a vacuum
42 Hedgehog text annotations - example of usability results (S. Raghavendra)
Copyright D. Petkovic43 API usability and requirements for the design of good APIs APIs are a form of human/machine UI More and more applications are developed by providing or using API but little is done to address the “usability” and design for “usability” of APIs Good article –http://www.codeproject.com/gen/design/APIUsabilityA rticle.asphttp://www.codeproject.com/gen/design/APIUsabilityA rticle.asp Use it for your final project
Copyright D. Petkovic44 API usability – what is it How well are APIs designed – do they provide all the functions, are the variables intuitive and useful etc. Documentation? Easy to download and link to Are simple application examples provided Do they run on all necessary platforms Managing API changes vs. legacy installations
Copyright D. Petkovic45 Usability testing - reading J. Rubin: “Handbook of Usability Testing”, John Wiley & Sons, 1994 (There are many tutorials on the WWW)
Copyright D. Petkovic46 Usability testing at SFSU Proper government mandates procedures must be followed In CS department usability testing is focused on testing SW and NOT the people => simple approval process allowed When submitting thesis proposal make sure you check the “human subject” button if necessary Instructions: