1. Chapter 4 – Expert Reviews, Usability Testing, Surveys, and Continuing Assessments

2. 4.1 Introduction Designers may fail to evaluate adequately.
Experienced designers know that extensive testing is necessary 
Many factors influence the evaluation plan
Evaluations might range from two-years to a few days  
Range of costs might be 10%  1% of a project budget
Customers are more and more expecting usability
4.1 Introduction
Designers can become so entranced with their creations that they may fail to evaluate them adequately.  
Experienced designers have attained the wisdom and humility to know that extensive testing is a necessity.  
The determinants of the evaluation plan include:
stage of design (early, middle, late)
novelty of project (well defined vs. exploratory)
number of expected users
criticality of the interface (life-critical medical system vs. museum exhibit support)
costs of product and finances allocated for testing
time available
experience of the design and evaluation team
The range of evaluation plans might be from an ambitious two-year test to a few days test.  
The range of costs might be from 10% of a project down to 1%.

3. 4.2 Expert Reviews Formal expert reviews have proven to be effective.
Experts may be available on staff or as consultants
Expert reviews may take one-half day to one week plus training
There are a variety of expert review methods to chose from
Expert reviews can be scheduled at several points in the development process
Try not to rely on just one expert.
While informal demos to colleagues or customers can provide some useful feedback, more formal expert reviews have proven to be effective.
Expert reviews entail one-half day to one week effort, although a lengthy training period may sometimes be required to explain the task domain or operational procedures.
There are a variety of expert review methods to chose from (see upcoming slide)
Expert reviews can be scheduled at several points in the development process when experts are available and when the design team is ready for feedback.
Different experts tend to find different problems in an interface, so 3-5 expert reviewers can be highly productive, as can complementary usability testing.

4. Expert Review Methods Heuristic evaluation Guidelines review
Consistency inspection
Cognitive walkthrough
Formal usability inspection
There are a variety of expert review methods to chose from:
Heuristic evaluation – expert reviews interface identifying where it doesn’t follow design heuristics – rules of thumb. E.g. Shneiderman’s 8 Golden rules, or Jacob Nielson’s heuristics
(Nielson’s discount usability engineering lowers the barriers to organizations considering usability by advocating quicker and cheaper approaches to task analysis, prototype development, and testing)
Obviously, the expert has to be an expert – familiar with the rules and how to apply them
Guidelines review – interface checked against the guidelines produced for the project or organization. Guidelines may have many items, so this can take time (and experience by the expert). If the guidelines are believed in, it seems like this should be done
Consistency inspection – examine interface for consistency in color, layout, fonts, terminology etc within the system or family of systems. Also examine training materials and help
Cognitive walkthrough – experts simulate users using the interface doing typical (or important) tasks (“walking though” them). Group meeting to discuss issues that arise (w/ expert, users, designers, possibly managers)
Formal usability inspection – more adversarial. Moderator runs meeting. Expert presents interface with evaluation. Design team has opportunity to rebut. May take longer to prepare for because of the formality. Adversarial nature is also a negative
In choosing a method to use, consider resources and time needed to do it, expertise and skill needed to do it, and its success in uncovering problems. You will of course, find advocates for each approach

5. Using Expert Reviews
Danger: Experts may not have an adequate understanding of the task domain or user communities.
Danger: may get conflicting opinions
Expert reviewers are not typical users, and may not relate completely.
Helps to chose experts who are familiar with the project and the organization.
Beneficial to do usability testing as well
The dangers with expert reviews are that the experts may not have an adequate understanding of the task domain or user communities. <Jim – looks like a WEB App – It IS a Web App>
E.g. the Interface Hall of Shame guy is anti-metaphor – Book – “For every PhD, there is an equal and opposite PhD”
Moreover, even experienced expert reviewers have great difficulty knowing how typical users, especially first-time users will really behave.
To strengthen the possibility of successful expert reviews it helps to chose knowledgeable experts who are familiar with the project situation and who have a longer term relationship with the organization. – can be brought back, have a stake in success

6. 4.3 Usability Testing and Laboratories
There is increasing attention to usability testing
Has benefits beyond usability
Not controlled experiments
The emergence of usability testing since the early 1980s is an indicator of the profound shift in attention to user needs and has resulted in construction of usability labs in many companies
The remarkable surprise was that usability testing not only sped up many projects but that it produced dramatic cost savings. Also, it helps force completion of the design phase since the usability testing creates a major deadline. Helps detect disasters in the making instead of later surprises to management
… not two alternatives and statistical tests … more like techniques of market research and advertising – focus groups – carefully planned set of tasks, a few “users”, results are feedback (in this case complaints and suggested changes). Frequently carried out in labs where users can be observed …

7. Usability Laboratories
Might be set up to allow observation via one-way mirror
Staffed by expert in usability testing and interface design
IBM was an early leader
Consultants available
A typical modest usability lab would have two 10 by 10 foot areas, one for the participants to do their work and another, separated by a one way mirror, for the testers and observers (designers, managers, and customers).
… staff can aid project before testing – w/ task analysis, design reviews, develop usability tests, provide references to tools and expertise

8. Usability Testing Process
Plan ahead
Pilot test
Choice of participants is important
Other factors to be controlled
Participants should be kept informed and respected
Think Aloud protocols useful
Videotaping is useful
Test can be repeated after significant improvements
Tasks to be carried out, participants to use, debriefing questions and questionnaires developed
Tryout testing with a 1-3 users before doing a large test (if doing a large test) – find out any problems with tasks, procedures and questionnaires before wasting a lot of time and money
… Should be representative in terms of – background in computing, knowledge/experience with the task, motivation, education, fluency with the language, eyesight, handedness, age, gender (this is why user analysis is important – so know what users are like – what is representative)
… If potentially relevant, may want to vary (be representative) time of day, day of week, noise, room temperature, distractions
… follow informed consent procedures for experiments – know the purpose, potential risks and benefits, allowed to back out; get signature. Ensure they know that the system is being tested, not them.
… participants asked to think aloud as they are working with the system – to give testers more insight into the problems the participants encounter
… allows designers to see the mistakes that users made (particularly repeated mistakes). Need logging to reduce time searching for particular sections of videotape – software to automatically log can be valuable – and what time user did what. Eye-tracking can even be used – to see where people are looking

9. Field Tests Real environments instead of labs Still useful to log
Beta testing is field testing
… capture #errors, productivity, amount of help requested
… partially deals with usability

10. Paper Prototypes Obtain very early feedback, inexpensively
Person plays the role of the computer, displaying screens
Allows capturing difficulties with wording, layout, and sequences involved in tasks
… without development effort

11. Competitive Usability Testing
Closer to controlled experiment
Compare interface to previous version or competitor
Ensure tasks are parallel
“Within Subjects” recommended
Counter balance order
… Psych jargon – same subjects used on both interfaces rather than separate groups of subjects on the two interfaces. Allows subjects to explicitly do comparisons between the interfaces for you
… equal number of subjects use each first – since performance on one may be influenced by having used the other

12. Issues with Usability Testing
Emphasizes first-time usage
Has limited coverage of the interface features.
Also use expert reviews
For all its success, usability testing does have at least two serious limitations:
it emphasizes first-time usage – users don’t have time to become experts
has limited coverage of the interface features – tests don’t hit every task
These and other concerns have led design teams to supplement usability testing with the varied forms of expert reviews.

13. Heuristic Evaluation and Discount Usability Engineering
Taken from the writings of Jakob Nielsen – inventor of both
Heuristic Evaluation is one form of expert reviews. Discount usability engineering is a broader concept - heuristic evaluation could be one aspect or one approach to discount usability engineering

14. Heuristic Evaluation Context – part of iterative design
Goal – find usability problems
Who – small set of evaluators
How – study interface in detail, compare to small set of principles
… preferably knowledgeable about usability.
… the principles are the heuristics – rules of thumb

15. Ten Usability Heuristics
Visibility of system status
Match between system and the real world
User control and freedom
Consistency and standards
Error prevention
Recognition rather than recall
Flexibility and efficiency of use
Aesthetic and minimalist design
Help users recognize, diagnose, and recover from errors
Help and documentation
<Below – direct quote from Nielsen’s WWW page>
Ten Usability Heuristics
Visibility of system status
The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
Match between system and the real world
The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.
User control and freedom
Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.
Consistency and standards
Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
Error prevention
Even better than good error messages is a careful design which prevents a problem from occurring in the first place.
Recognition rather than recall
Make objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
Flexibility and efficiency of use
Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
Aesthetic and minimalist design
Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
Help users recognize, diagnose, and recover from errors
Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
Help and documentation
Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.
Could use Shneidermans guidelines and principles – don’t want something with 1000 items

16. How to Conduct a Heuristic Evaluation
More than one evaluator to be effective.
Each evaluator inspects the interface by themselves
General heuristics may be supplemented
Results can be oral or written
Evaluator spends 1-2 hours with interface
Evaluator goes through interface > 1 time
Evaluators may follow typical usage scenarios
Interface can be paper
… Different evaluators will find different usability problems. By having 3-5 evaluators, should get good coverage, without too much expense and without getting a lot of repetition of the same problems (see next slide).
… only after all evaluators have done their work can evaluators talk to other evaluators or see what they came up with – to ensure independence and avoid bias – so new problems will be found by new evaluators
… w/ specific heuristics for the kind of product that it is (its domain)
… oral – have observer writing down the evaluator’s comments – then coalescing after all evaluators. Advantage – results may be available sooner. Written – evaluator must write their own report, then somebody has to create coalesced summary
… break into multiple sessions if big complex system requiring more time
Different experts tend to find different problems in an interface, so 3-5 expert reviewers can be highly productive, as can complementary usability testing.
… first get an overall feel for interface and scope of system. Second evaluate each interface element w/r/t heuristics
… going through steps user would follow to accomplish different tasks – drawn from task analysis – and chosen as important/common tasks
… Does not have to be a working prototype – this makes it possible to evaluate very early

17. Different Evaluators Find Different Problems
Some problems were found by most evaluators
Some evaluators found most problems (but don’t plan or them necessarily being somehow superior and toss the rest for next time – they might not be the superior ones next time – plus some of harder problems to find were found by evaluators who did not find many problems))
No problems were found by all evaluators
No evaluators found all problems
On average, individual evaluators found 35% of the problems that were found by the group as a whole
< See next slide then go Back to previous slide>

18. Number of Evaluators
Note that we don't get up to 100% of problems found
But there are diminishing returns as we add evaluators – and costs go up as we add evaluators
You could actually do a cost benefit analysis for your project based on your cost structure (cost of evaluators, overhead for getting started, costs for fixing errors later, saved costs from user efficiency (if in house project), or increased sales (tough to figure!), etc) and the complexity of the interface. 3-5 is itself only a rule of thumb!! 
<Now, go back two slides>

19. Heuristic Evaluation Results
List of usability problems
With principle violated
With severity
NOT fixes
May have debriefing later to aid fixing
Discount usability
… interfaces tend to have more minor errors than major ones. Major problems are a little easier to find.
… after all evaluators are done, have meeting w/ evaluators, (observer), design team representatives. Brainstorming focus
… not intended to find every last problem – intended to find problems in a cost effective manner – studies finding 50-1 ROI

20. Usability Problem Location
Single Location
Two/Several Locations
Overall Structure
Something Missing
… problems found by comparing the locations – such as consistency problems
Something missing problems are hard to find in paper prototypes – must really focus on finding those

21. Severity Help focus repair efforts Help judge system readiness
Factors in Severity:
Frequency
Impact
Persistence
Market impact
Scale severity to a number
May wait on severity
… which problems to work on
… if a lot of severe problems, may want to do another round of usability evaluation .
… is the problem common or rare?
… how hard will it be for the users to deal with?
… will the problem continue to be a problem or will users get used to it and no longer be affected (I.e. affect ease of learning only or ease of use too)
… will it affect sales (e.g. poor product reviews)
… 0 – I don’t believe it is a problem; 1 – cosmetic problem; 2 – minor problem; 3 – major problem; 4 – catastrophic – show stopper
may wait for after all evaluations in and ask all evaluators to rate the severity of all problems found – a) ease evaluators job while finding errors; b) get evaluators ratings on all problems found, not just their own. Synthesizing several opinions on this will give a better view of the importance than one person’s opinion

22. H.E. Complementary w/ Usability Testing
Each will find problems that the other will miss
H.E. Weakness – finding domain specific problems
Don’t H.E. and Usability Test same prototype version
… will find a lot of the same problems. Run H.E. early with paper prototype, then clean up before trying usability testing

23. Discount Usability Engineering
“It is not necessary to change the fundamental way that projects are planned or managed in order to derive substantial benefits from usability inspection”
6% of project budget on usability
18% of respondents used usability evaluation methods the way they were taught
… Jakob Nielsen
… Nielsen 1993 – in 31 projects with usability engineering efforts <still fairly substantial> Cost can be a barrier to using usability engineering
… … 68% made minor modifications, 15% made major modifications
(53% of respondents used less than the recommended 3-5 evaluators for H.E.)

24. More Discount Usability Engineering
Cost projection to focus on usability may be reduced
“Insisting on only the best methods may result in having no methods used at all”
35% of respondents used 3-6 users for usability testing
Nielsen and others suggest 50-1 ROI
… Mantei and Teorey $128K
… - not a full test – a discount version – more affordable in the real world than bringing in 10 or more users. To develop usable interfaces, don’t need enough users to derive statistically significant conclusions (which is what is needed to publish conclusions)

25. Elements of Discount Usability Engineering
Scenarios
Simplified Thinking Aloud
Heuristic Evaluation
… the first two to be discussed in turn; heuristic evaluation already discussed

26. Scenarios
Take prototyping to extreme – reduce functionality AND number of features
Small, can afford to change frequently
Get quick and frequent feedback from users
Compatible with interface design methods
… can even be paper mock-ups (as discussed earlier) …
… scenarios may have been developed as part of design process, so don’t represent additional effort required for testing

27. Simplified Thinking Aloud
Bring in some users, give them tasks, have them think out loud
Fewer users in user testing
… w/o videotape, detailed protocol analysis (analysis of subjects statements down to a detailed level), psychologists, labs. Nielsen says that CS people can be trained to apply thinking aloud observation to get useful results
… learn the most from the first few users – 3-5

28. Heuristic Evaluation Fewer principles etc to apply
Compare interface to previous version or competitor
Ensure tasks are parallel
“Within Subjects” recommended
Counter balance order
… hard to evaluate for 1000 guidelines; hard to train people to look for 1000 things (does take experience – even with Nielsen’s 10)
… Psych jargon – same subjects used on both interfaces rather than separate groups of subjects on the two interfaces. Allows subjects to explicitly do comparisons between the interfaces for you
… equal number of subjects use each first – since performance on one may be influenced by having used the other

29. Stages of Views of Usability in Organizations
Usability does not matter.
Usability is important, but good interfaces can surely be designed by the regular development staff as part of their general system design.
The desire to have the interface blessed by the magic wand of a usability engineer.
GUI/WWW panic strikes, causing a sudden desire to learn about user interface issues.
Discount usability engineering sporadically used.
Discount usability engineering systematically used.
Usability group and/or usability lab founded.
Usability permeates lifecycle.
<Direct from >
Usability does not matter. The main focus is to wring every last bit of performance from the iron. This is the attitude leading to the world-famous error message, "beep."
Usability is important, but good interfaces can surely be designed by the regular development staff as part of their general system design. <Imperialist developers know best attitude>. At this stage, no attempt is made at user testing or at acquiring staff with usability expertise.
The desire to have the interface blessed by the magic wand of a usability engineer. Developers recognize that they may not know everything about usability, so they call in a usability specialist to look over their design and comment on it. The involvement of the usability specialist is often too late to do much good in the project, and the usability specialist often has to provide advice on the interface without the benefit of access to real users.
GUI/www panic strikes, causing a sudden desire to learn about user interface issues. Many companies as they move from character-based user interfaces to graphical user interfaces, or more recently to WWW, realize the need to bring in usability specialists to advise on graphical user interfaces/WWW from the start. <usability experts may believe the organizations are putting the cart before the horse having made the decision to change interfaces/ platform before consulting user needs> Even so, GUI/WWW panic is an opportunity for usability specialists to get involved in interface design at an earlier stage than the traditional last-minute blessing of a design that cannot be changed much.
Discount usability engineering sporadically used. Typically, some projects use a few discount usability methods (like user testing or heuristic evaluation), though the methods are often used too late in the development lifecycle to do maximum good. Projects that do use usability methods often differ from others in having managers who have experienced the benefit of usability methods on earlier projects. Thus, usability acts as a kind of virus, infecting progressively more projects as more people experience its benefits.
Discount usability engineering systematically used. At some point in time, most projects involve some simple usability methods, and some projects even use usability methods in the early stages of system development. Scenarios and cheap prototyping techniques seem to be very effective weapons for guerrilla HCI in this stage.
Usability group and/or usability lab founded. Many companies decide to expand to a deluxe usability approach after having experienced the benefits of discount usability engineering.
Usability permeates lifecycle. The final stage is rarely reached since even companies with usability groups and usability labs normally do not have enough usability resources to employ all the methods one could wish for at all the stages of the development lifecycle. However, there are some, often important, projects that have usability plans defined as part of their early project planning and where usability methods are used throughout the development lifecycle.

30. End Nielsen insert for Chapt 4

31. 4.4 Surveys Users are familiar with surveys
Surveys can provide lots of responses
Surveys can be inexpensive
Survey results can often be quantified
Surveys can be complementary to usability tests and expert reviews.
… can be given after usability test , give different perspective than expert review

32. Successful Use of Surveys
Clear Goals
Preparation
Don’t forget to gather background info
Other goals concerning learning about the user …
Goals concerning the interface …
reasons for not using an interface
familiarity with features
their feeling state after using an interface
… in advance – know what you are looking to find out. Goals could be tied to a theory such as Objects and Action Interface model
… prepare survey – creating survey items that address the goals, have it reviewed by colleagues, pilot tested. Preview statistical analyses and presentations to ensure the kind of info desired will be available
… age, gender, origins, education, income … experience with computers (specific applications or software packages, length of time, depth of knowledge) …
… job responsibilities (decision-making influence, managerial roles, motivation) … personality style (introvert vs. extrovert, risk taking vs. risk aversive, early vs. late adopter, systematic vs. opportunistic)
… reasons for not using an interface (inadequate services, too complex, too slow)
familiarity with features (printing, macros, shortcuts, tutorials)
their feeling state after using an interface (confused vs. clear, frustrated vs. in-control, bored vs. excited).

33. Online Surveys Online surveys cut cost
Online surveys may boast response rate
Online survey may bias survey
Online surveys avoid the cost of printing and perhaps postage, and the extra effort needed for distribution and collection of paper forms.
Many people prefer to answer a brief survey displayed on a screen, instead of filling in and returning a printed form. If people are online they may be more receptive than somebody receiving unsolicited postal mail
… since survey is “self-selected” – those who go to the page. for example, this could bias towards those more comfortable with computing

34. Simple Survey Use a simple scale Ask a few questions addressing goals
Easy for users
Directly quantifiable for use in statistics
Ask a few questions addressing goals
Few questions lead to higher response rate
Low cost, quantifiable results makes survey repeatable
… such as Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree
… direct and specific questions help to make the results meaningful to designers e.g. <I have tried to make these less command oriented than in the book. Some of these presuppose real use as in a field test as opposed to a short amount of use>
I find the system easy to use
I feel competent with and knowledgeable about the system
<made more general than the book> When carrying out a new task, I am confident that I will be correct on the first try
When I get an error message, I find that it is helpful in identifying the problem
I think that there are too many options and special cases.
I believe that the system could be substantially simplified
I have trouble remembering the commands and options, and must consult the manual frequently
When a problem arises, I ask for assistance from someone who really knows the system
… can be repeated at various stages to see if answers improve

35. More Scaling Some surveys use bipolar alternatives
E.g. Error messages were
Hostile Friendly
… having the opposite ends have meaning to the participant (instead of the content free Strongly Agree etc) may help user know how to answer the question (and may help avoid a halo effect where respondent answers similarly to all questions based on a general impression and doesn’t give enough specific answers) … Having more than 5 options helps respondent distinguish their feelings more. Some psychologists have found that 7 is a good number of options to give people on a scale.

36. Not So Simple Survey
Shneiderman’s Questionnaire for User Interface Satisfaction (QUIS)
Detailed info – gives specific feedback on many things
Response rate will be lower
Response bias to those highly motivated to help, very patient, and/or not that busy
Short form available for less patient
IBM’s Post-Study Usability Questionnaire
Software Usability Measurement Inventory
… reproduced on pages (8 pages; 131 items). Copyrighted and licensed to businesses
… 48 items
… 50 items

37. 4.5 Acceptance Tests
Large (particularly custom) software projects have “acceptance tests”
It’s time for something more specific than “user friendly” for handling of usability
Time to learn specific functions
Speed of task performance
Rate of errors by users
Human retention of commands over time
Subjective user satisfaction
Multiple such tests - different components - different user communities.
After acceptance, field testing before full distribution..
The goal all usability evaluation is to improve interface in the prerelease phase, when change is relatively easy
… measurable goals for hardware and software performance, quality, and functionality. If the completed product fails to meet these acceptance criteria, the system must be reworked until success is demonstrated. Acceptance testing can be adversarial as both organizations promote their own interests
… Rather than the vague, subjective and misleading criterion of "user friendly," measurable criteria for the user interface can be established. Specific, measurable criteria benefits the customer, and the designer – avoiding arguments, establishing an endpoint. Specific, measurable usability criteria could include the following:
Time for users to learn specific functions – <somewhat dated >Book e.g. “The subjects will be 35 secretaries hired from an employment agency. They have no word-processing experience, but have typing skills in the range of words per minute. They will be given 45 minutes of training on the basic features. At least 30 secretaries should be able to complete, within 30 minutes, 80 percent of the typing and editing tasks in the enclosed benchmark test completely.”
Speed of task performance by users – use after learning. Book e.g “After four half days of regular use of the system, 25 of 35 secretaries should be able to carry out, within 20 minutes, the advanced editing tasks in the second benchmark test, and should make fewer than six errors”
Rate of errors by users
Human retention of commands over time
Subjective user satisfaction
In a large system, there may be eight or 10 such tests to carry out on different components of the interface and with different user communities. Choice of tests is very important
Once acceptance testing has been successful, there may be a period of field testing before national or international distribution..
The goal of early expert reviews, usability testing, surveys, acceptance testing, and field testing is to force as much of the evolutionary development as possible into the prerelease phase, when change is relatively easy and inexpensive to accomplish.

38. 4.6 Evaluation During Active Use
Must continue to evaluate usability under real use
Improvements are possible and are worth pursuing.
A carefully designed and thoroughly tested system is a wonderful asset, but successful active use requires constant attention from dedicated managers, user-services personnel, and maintenance staff.
Perfection is not attainable, but percentage improvements are possible and are worth pursuing.

39. 4.6.1 Interviews and Focus Groups
Interviews with individual users
After individual discussions, group discussions
Interviews and focus group discussions
Interviews with individual users can be productive because the interviewer can pursue specific issues of concern. Small number of representative users interviewed. Can get specific suggestions for problems / improvements
After a series of individual discussions, group discussions are valuable to ascertain the universality of comments.

40. 4.6.2 Continuous User-Performance Data Logging
Software should enable collecting data about system usage
Logged data provides guidance
E.g. Most frequent error message
E.g. Most frequently used capabilities
Pay attention to user’s privacy
The software architecture should make it easy for system managers to collect data about the  patterns of system usage, speed of user performance, rate of errors, or frequency of request for online assistance.
A major benefit of usage-frequency data is the guidance they provide to system maintainers in optimizing performance and reducing costs for all participants. E.g. may suggest New hardware; Changes in procedures; Improvements to training
… something is causing this to happen – problem with training, problem with help, difficulty in a procedure to accomplish a task, unclear onscreen instructions, problem with the message (message should explain how to avoid the problem) …
… can be made more accessible, simpler … saving future efforts … (least frequently used may be being avoided for some reason!)
… aggregates less potentially threatening – if info maintained about individual’s use, need to anonymize or inform users of the collection and use – and be open to users’ inspection of results/ implications

41. 4.6.3 Online or Telephone Consultants
Online or telephone consultants provide assistance to users
Helpful to users when problems arise.
Consultants can provide info about problems users are having
Online or telephone consultants are an extremely effective and personal way to provide assistance to users who are experiencing difficulties.
Many users feel reassured if they know there is a human being to whom they can turn when problems arise. Helps users.
At the same time, these consultants become experts on what the users have trouble with – can provide valuable info to developers and project managers

42. 4.6.4 Online Suggestion Box
Provide facility to allow users to send messages to the maintainers or designers.
Easy access encourages users to make productive comments
Electronic mail can be employed to allow users to send messages to the maintainers or designers.
Such an online suggestion box encourages some users to make productive comments, since writing a letter may be seen as requiring too much effort.

43. 4.6.5 Online Bulletin Board
Electronic bulletin board (newsgroups) permit posting of open messages and questions.
New items can be added by anyone, but usually someone monitors the bulletin board
Many interface designers offer users an electronic bulletin board or newsgroups to permit posting of open messages and questions. These allow users to get answers to their questions. But also allow designers to find out what is causing trouble.
New items can be added by anyone, but usually someone monitors the bulletin board to ensure that offensive, useless, or repetitious items are removed.

44. 4.6.6 User Newsletters and Conferences
Newsletters can help users, include requests for assistance, promote user satisfaction
Printed newsletters can be carried away from the workstation and have respectability.
Online newsletters are less expensive and more rapidly disseminated
Conferences allow workers to exchange experiences with colleagues
Obtaining feedback in these ways allows gauging attitudes and gathering suggestions (as well as being good PR)
Newsletters that provide information about novel interface facilities, suggestions for improved productivity, requests for assistance, case studies of successful applications, or stories about individual users can promote user satisfaction and greater knowledge.
Printed newsletters are more traditional and have the advantage that they can be carried away from the workstation.
Online newsletters are less expensive and more rapidly disseminated
Conferences allow workers to exchange experiences with colleagues, promote novel approaches, stimulate greater dedication, encourage higher productivity, and develop a deeper relationship of trust.

45. 4.7 Controlled Psychologically Oriented Experiments
Scientific and engineering progress aided by precise measurement.
Designs of interfaces will be improved if quality can be quantified
Scientific method as applied to HCI:
Deal with a practical problem – but within a theoretical framework
State a clear and testable hypothesis
Identify a small number of independent variables
Carefully choose the dependent variables
Carefully select subjects and assign to groups
Control for biasing factors
Apply statistical methods to data analysis
Resolve the practical problem, refine the theory, and give advice to future researchers
Controlled experiments useful in fine tuning the interface.
Scientific and engineering progress is often stimulated by improved techniques for precise measurement.
Rapid progress in the designs of interfaces will be stimulated as researchers and practitioners evolve suitable human-performance measures and techniques. E.g. having a ease of learning rating based on some standard of testing, will a) tell designers what to shoot for b) promote usability by giving customers an objective measure (book e.g like energy efficiency numbers on appliances) <Seems unlikely in the near term>
The outline of the scientific method as applied to human-computer interaction might comprise these tasks:
Deal with a practical problem and consider the theoretical framework
State a lucid and testable hypothesis
Identify a small number of independent variables that are to be manipulated
Carefully choose the dependent variables that will be measured. Dependent measures could include performance times, user-subjective satisfaction, error rates, and user retention over time.
Judiciously select subjects and carefully or randomly assign subjects to groups
Control for biasing factors (non-representative sample of subjects or selection of tasks, inconsistent testing procedures)
Apply statistical methods to data analysis
Resolve the practical problem, refine the theory, and give advice to future researchers
Managers of actively used systems are coming to recognize the power of controlled experiments in fine tuning the human-computer interface. -- comparing existing to proposed changes – change in menus, display format … subjects can use the proposed interface for a limited time, and then performance could be compared with the control group. Key here is a) focus, compare one difference or results are confounded b) comparing proposed to existing means that not developing alternatives (developing one new)

46. End Chapter 4