1. COM531 Multimedia Technologies
Lecture 10 – HCI Guidelines
Evaluation

2. The User Interface
All those parts of the system we come into contact with…
Physically we might interact with a device by pressing buttons or moving levers and the interactive device might respond by providing feedback through the pressure of the button or lever.
Perceptually the device displays things on a screen, or makes noises which we can see and hear.
Conceptually we interact with a device by trying to work out what it does and what we should be doing. The device provides messages and other displays which are designed to help us do this.

3. The User Interface Input Output
Methods are needed to enter commands (tell the system what we want it to do)
We also need to be able to navigate through the commands and the content of the system
We need to enter data or other content into the system
Output
So the system can tell us what is happening - provide feedback
So the system can display the content to us.

4. Key Issues
Accessibility
Usability
Acceptability
Engagement

5. Accessibility
Removal of the barriers that would otherwise exclude some people from using the system at all
Legislation requires software to be accessible.
UK’s Equality Act 2010 and Section 508 in the US
Web Accessibility Initiative
The Web Content Accessibility Guidelines (WCAG) 2.0, explain how to make Web content accessible for people with disabilities.
ISO 9241 Part 171 Guidance on Software Accessibility

6. Principles of Universal Design
Equitable Use: The design does not disadvantage or stigmatize any group of users.
Flexibility in Use: The design accommodates a wide range of individual preferences and abilities.
Simple, Intuitive Use: Use of the design is easy to understand, regardless of the user's experience, knowledge, language skills, or current concentration level.
Perceptible Information: The design communicates necessary information effectively to the user, regardless of ambient conditions or the user's sensory abilities.

7. Principles of Universal Design
Tolerance for Error: The design minimizes hazards and the adverse consequences of accidental or unintended actions.
Low Physical Effort: The design can be used efficiently and comfortably, and with a minimum of fatigue.
Size and Space for Approach & Use: Appropriate size and space is provided for approach, reach, manipulation, and use, regardless of the user's body size, posture, or mobility.

8. Acceptability and Engagement
Acceptability refers to fitness for purpose in the context of use
Unacceptable use of mobile phones
Engagement is concerned with all the qualities of an experience that really pull people in
A sense of immersion that one feels

9. Design Principles, Guidelines and Rules
International Standards
Design Principles
Universally applicable high level design goals, based on International Standards
Open to board interpretation
‘Design for Human Cognitive Limitations’
Design Guideline
Principles are interpreted; guidelines produced to assist with design situations
Must be interpreted within the context of the task. Usability must include task-context dependent feature
‘Recognition rather than recall’
‘Make it obvious which menu items are/are not active at any time’
Design Rule
Highly specific low level design rules
Found in corporate style guides and design manuals.
In menu design,
‘Max of 10 items per panel; inactive items should be greyed out’

10. HCI Guidelines “Broad brush” design rules
Useful check list for good design
Better design using these than using nothing!
Different collections:
Benyon and Turner’s 12 Principles
Nielsen’s 10 Heuristics
Shneiderman’s 8 Golden Rules

11. Benyon and Turner’s 12 Principles
Visibility
Consistency
Familiarity
Affordance
Navigation
Control
Feedback
Recovery
Constraints
Flexibility
Style
Conviviality

12. 1. Visibility Computers are good at remembering, people are not!
Try to ensure that things are visible so that people can see what functions are available and what the system is currently doing
This is an important part of a psychological principle that it is easier to recognize things than to have to recall them
If it is not possible to make it visible, make it observable
Consider making things ‘visible’ through the use of sound and touch
Use menus, icons, dialog boxes vs commands

13. 1. Visibility The common commands and defaults are made visible
Other commands are observable by using the drop down menus
Visibility and sensible grouping makes people aware of other options

14. 2. Consistency Be consistent in the use of design features
Be consistent with similar systems and standard ways of working
Consistent use of commands, sequence of tasks, use of terminology, layout and structure
Internal consistency within the system; interaction style and presentation
External consistency between packages; easier to move from one application to another; ‘look and feel’ (but don’t guarantee usability)

15. 3. Familiarity Use language and symbols that users are familiar with
Where this is not possible because the concepts are quite different from those people know about, provide a suitable metaphor
To help them transfer similar and related knowledge from a more familiar domain
User: “I just got a message Rstrd Info. What does it mean?”
Designer: “That’s restricted information.”
User: “But surely you can tell me!!!”
Designer: “No, no… Rsdrd Info stands for “Restricted Information.”
User: “Hmm… but what does it mean???”
Designer: “It means the program is too busy to let you log on.”
User: “Ok. I’m taking a break.”

16. 4. Affordance Design things so that it is clear what they are for
The properties of objects and how these relate to how objects are used
For example make buttons look like buttons so people will press them
Use textboxes for data entry, labels for displaying output

17. 5. Navigation
Provide support to enable people to move around the system
Maps, directional signs, informational signs
Menus are often used for navigation, signs (labels) indicate where else you can go in the system

18. 6. Control Make it clear who or what is in control
Allow user to take control of the system which responds to actions
Interleaving modes
Normal View, Print Preview
Design View, Code View
Tailor system to individual needs (Accessibility)
Window Size, Font Size, Colour, Toolbars
Good help and documentation
Tooltips, Context Sensitive Help
User Guides
Online help

19. 7. Feedback
Feed back information from the system to user so that they know what effect their actions have had
Continuously inform the user about
what it is doing
how it is interpreting the user’s input
Constant and consistent feedback will enhance the feeling of control
E.g. Cursor style, Status bar
User should know when completion is successful
Direct view as changes happen
Message box

20. 8. Recovery
Enable recovery from actions and errors quickly and effectively
Users know that they can return to the previous state by providing Undo and Cancel options
If they make a mistake, offer them clear and informative instructions to enable them to recover
E.g. Message boxes giving instructions
Users don’t like to feel trapped by the computer!
Provide clearly marked exits - should offer an easy way out of as many situations as possible

21. 9. Constraints
Provide constraints so that people do not try to do things that are inappropriate
Have allowable actions
Give confirmation of dangerous operations
Users are prevented from making mistakes by limiting the amount of typing required
Disable menu commands (grey = inactive)

22. 10. Flexibility
Multiple ways of doing things to accommodate different levels and needs of a range of users
Provide Shortcuts through use of Hot Keys
E.g. Ctrl C for ‘Copy’ , Ctrl V for ‘Paste’
Perform regular, familiar actions more quickly
Personalise the system by viewing/removing toolbars as needed

23. 11. Style 12. Conviviality Stylish designs Attractive
Polite, friendly and pleasant designs

24. Nielsen’s 10 Heuristics Visibility of system status
Match between system and the real world
User control and freedom
Consistency and standards
Error prevention
Helping users recognise, diagnose and recover from errors
Recognition rather than recall
Flexibility and efficiency of use
Aesthetic and minimalist design
Help and documentation

25. Shneiderman’s 8 Golden Rules
Strive for consistency
Enable frequent users to use shortcuts
Offer informative feedback
Design dialogs to yield closure
Offer error prevention and simple error handling
Permit easy reversal of actions
Support internal locus of control
Reduce short-term memory load

26. International Standards
ISO 9126 Software Engineering
International standard for the evaluation of the quality of software
4 Parts
Part 1 Software Quality
Functionality
Reliability
Usability
Efficiency
Maintainability
Portability

27. International Standards
ISO :2002 Software ergonomics for multimedia user interfaces
Part 1: Design principles and framework
Part 2: Multimedia navigation and control
Part 3: Media selection and combination
ISO :2006 Ease of operation of everyday products
Part 1: Design requirements for context of use and user characteristics
ISO 6385:2004 Ergonomic principles in the design of work systems
ISO :1999 Parts 1 and 2 "Ergonomic requirements for the design of displays and control actuators"

28. International Standards
ISO/IEC 25051:2005 Software engineering -- Software product Quality Requirements and Evaluation (SQuaRE)
Requirements for quality of Commercial Off-The-Shelf (COTS) software product and instructions for testing
ISO/IEC 25062:2006 Software engineering -- Software product Quality Requirements and Evaluation (SQuaRE)
Common Industry Format (CIF) for usability test reports
ISO/TR 16982:2002 Ergonomics of human-system interaction
Usability methods supporting human-centred design

29. International Standards
Specific for Icons
ISO Icon Symbols & Functions - General
ISO Object Icons
ISO Pointer Icons
ISO Control Icons
ISO Tool Icons
ISO Action Icons
ISO Icons for Controlling Multimedia Software
ISO Icons for World Wide Browser Toolbars

30. International Standards
ISO 9241 Ergonomic Requirements for Office Work with Visual Display Terminals (VDT).
32 Parts covering all aspects of Usability (hardware, software, processes)
Part 11 Guidance on Usability
Read ISO 9241 Bluffer Guide
For more information on standards:

31. Evaluation

32. User-Centred System Design
Problem
Statement
Observation of
existing systems
Task
Analysis
Requirements
Gathering
Requirements Statement –
Functional and non-functional
Usability Guidelines
& Heuristics
Design &
Storyboarding
Storyboard
Prototype
Implementation
Prototype
Evaluation
Transcript & Evaluation Report
Installation
Final Implementation

33. The Star Method

34. Evaluation
‘Evaluation is concerned with gathering data about the usability of a design or product by a specified group of users for a particular activity within a specified environment or work context’ (Preece, 1994)
Evaluation is central to designing interactive systems
Everything gets evaluated at every step of the process
For example, requirements are evaluated, storyboards evaluated and a prototype built. The prototype is then evaluated and some aspect of a physical design identified and implemented; this is then evaluated again and so the iteration continues until a final product is complete

35. ‘Users will evaluate your interface sooner or later.’
Why do we Evaluate?
‘Users will evaluate your interface sooner or later.’
To suggest improvements to the design
To confirm that the software meets all of the functional and usability specifications
To confirm acceptability of the interface and/or supporting materials
To compare alternative designs
To ensure that it meets the expectations of users
To match or exceed the usability of competitor’s products
To ensure that it complies with standards and any statutory requirements
Some of the good reasons why evaluation should be done include,
1. To suggest improvements to the design so that it fits better with the tasks it is to perform, the work environment in which it will be used, and the preferred ways of working of the target users.
2. To confirm that the software meets all of the functional and usability specifications that were made at the beginning (and throughout) the development process.
3. To compare alternative designs to determine which is ‘best’.
4. To ensure that it meets the expectations of your customers; that it will not damage your competitive reputation in the marketplace.
5. To see that it matches, or exceeds, the usability of your competitors’ products and/or of any earlier version of the same package.
6. To ensure that it complies with any statutory requirements (such as those incorporated into EU legislation.).

36. Evaluation is Often Performed Badly
Designers assume their own personal behaviour is ‘representative’ of that of an average user
Designers make assumptions about how people are able to operate the software, but these assumptions might well be unfounded.
Acceptance of traditional/standard interface design - assume style guides ensure good software design
Evaluation may be postponed until ‘a more convenient time’ when functionality is complete
Poor knowledge of evaluation techniques and lack of expertise in analysing experiments
Reasons for poor evaluation have been identified as:
1. The designers assume that their own personal behaviour in using the software is representative of that of the average user. Thus, if they can easily and effectively use the software, then they think anyone will be able to do so.
2. Designers make assumptions about how people are able to perform with software, but these assumptions might well be unfounded.
3. Some companies develop a standard style and layout (style guidelines) for their software. It is then assumed that if these guidelines are followed then good software will result. However, a style developed for one kind of product and for one set of users, will not necessarily be successful in a different product for a different user base.
4. Evaluation is often put off while developers concentrate on achieving the functional requirements of the software. Since this process usually takes longer than one planned for, then by the time they get round to thinking of evaluation, time and budget are running out and so the job is skimped.
5. A number of software designers have poor knowledge of evaluation techniques, due to a poor level of training in this area

37. What Do We Evaluate? Usability (Criteria)
Initial designs (pre-implementation)
Interfaces/Interaction (Heuristics)
Prototype at various stages
Final implementation of software system
Documentation

38. Types of Evaluation Formative Evaluation
Evaluation within the design process
Produce good usability through the process of evolution, forming/reforming the product
Informal, Structured
Summative Evaluation
Take the finished system & assess it for aspects of usability
Carry out experiments, after implementation
Purpose is Quality Control
Formal, Costly & Time-consuming

39. Formative Evaluation Ask the Experts (No users) Cognitive Walkthrough
Heuristic Evaluation
Ask the Users
Focus groups
Questionnaire
Interviews
User interaction
Think Aloud
Co-operative Think Aloud
Expensive
Cheap C O S T

40. Types of Data Quantitative data
‘Objective’ measures of certain factors by direct observation
E.g. time to complete certain tasks, accuracy of recall, number of errors made
User performances or attitudes can be recorded in a numerical form
Qualitative data
‘Subjective’ responses; Opinions rather than measurements
Reports and opinions that may be categorized in some way but not reduced to numerical values
Depending on the method used, evaluation can result in the collection of either quantitative or qualitative data.
Quantitative data:
‘Objective’ measurement of certain factors by direct observation. For example, the time to complete certain tasks, or accuracy in recalling how to do some function, or number of errors made in certain tasks. More ‘subjective’ measures can be quantified by using some form of rating scale.
Qualitative data:
More subjective in nature, opinions rather than measurements, so it is obviously more difficult to analyse. It may, in some cases, be possible to place sets of opinions into certain defined categories though they cannot be reduced to any numerical form. Despite the lack of objective measure, these kinds of data are just as useful to the design team in developing software that people will actually want to use.

41. Recording Methods Paper and pencil – cheap, limited to writing speed
Audio – good for think aloud, difficult to match with other protocols
Video – accurate and realistic, needs special equipment
Computer logging – automatic and unobtrusive, large amounts of data difficult to analyze
User notebooks – coarse and subjective, useful insights, good for longitudinal studies
Mixed use in practice. Audio/video transcription difficult and requires skill. Some automatic support tools available
There are a number of different methods by which data can be collected. Most of them will require data gathering from at least one representative user at a time and so they would properly be used for what we termed empirical evaluation. Some of these tools are outlined above.
Explanation of each of the tools will be provided in class.

42. Cognitive Walkthrough
‘Expert’ simulates user actions/thoughts and steps through the action sequence to complete the task
Has rough plan and explores the system for possible actions
Interprets system responses and assesses if each step is or is not good for a new user
Are the actions appropriate and visible? Is the feedback adequate?
Suits systems primarily learned by exploration e.g. walk-up-and-use such as ATM, ticket machines
Overall question:
How successfully does this design guide the unfamiliar user through the performance of the task?
This is another ‘analytic’ evaluation technique in that it is carried out by the designer, or an ‘expert’ user, who attempts to simulate the way an average user would react to the system. It is particularly useful for the design of ‘walk-up-and-use” systems, such as cash machines, where the user will have to learn how to operate it by ‘exploring’ the interface. That is, the designer must make sure that it is going to be obvious, just from looking at the interface, what action should be taken by the user at each stage, and that sufficient and clear feedback is given to the user to show that they have done the correct thing.
So, the main question to be answered by the technique is “How well does the design guide the unfamiliar user through the performance of some task?” There is no concern to measure, for example, the speed with which the user can perform the action, though the designer might want to try to predict, say, likely error rates and their consequences since these could greatly affect the user’s ‘satisfaction’ with the system.
To apply the ‘cognitive walkthrough; technique the designer selects a particular task and breaks it down into its constituent subtasks. For example, in an automated ticket machine, the task might by described as ,
“Purchase a ticket (single or return) to a particular destination.”
This can be broken down into sub-tasks such as,
1. Choose destination
2. Indicate journey type (single/return)
3. Deposit money
4. Get ticket.
The designer ‘walks through’ these stages in sequence and for each one will ask him/herself the questions,
(a) Will it be obvious to the user what action to take next?
b) Will the user correctly interpret the description given to the correct action, and connect it with what they are trying to achieve?
(c)Will the system give feedback so the user will interpret the system’s response correctly?
This process, of asking the three questions, is carried out through all the stages of the task. Any “No” answers are an indication of where the designer needs to improve the interface design.

43. Heuristic Evaluation
A ‘heuristic’ can be defined as a ‘rule-of-thumb’ or general rule
The idea is to assess the design against known criteria
A number of reviewers (3-5) go through product, screen by screen, and note any problems and note violations of these principles, with a severity rating (0-4)
All responses are collected and aggregated
About 5 reviewers can find about 75% of the problems
Nielsen’s 10 Heuristics
A ‘heuristic’ can be defined as a ‘rule-of-thumb’, or general rule. For example, in evaluating an interface, heuristics might be derived from the general guidelines that have been used in the system’s design. These might take the form of statements like “prevent errors”, provide feedback”, etc.
A number of reviewers then go through the product, screen by screen, and make note of any problems, using the list of heuristics to focus their thoughts.
Some studies have shown that about five reviewers can find about 75% of the problems that are found by 15 reviewers. So this method can be very economical in discovering many of the major problems with an interface, especially if it is going to be used in conjunction with other evaluation methods that would detect problems missed by the heuristic evaluation.
Refer to Preece et al, (1994) pp. 676 for further explanation.

44. 3. Focus Groups A group of users and an evaluator
Structured group interview; flexible
Allows interaction between users
Typically for requirements gathering, not system use…
… but can be used post-task

45. 4. Questionnaire Questions fixed in advance
Completed independently of the evaluator
The purpose of the questionnaire and purpose of information must be clear
General, open ended, scalar, multi-choice, ranked questions

46. 5. Interviews Asking users about their experience with a system
General questions first, followed by more detailed ones
Needs careful planning, structured around some central questions
Structured, semi-structured, unstructured
Requires consistency and flexibility

47. 6. Think Aloud User participation
User is asked to talk through what he/she is doing whilst being observed
describing what is happening,
why an action is taken,
what the user is trying to do,
what the user is thinking,
the goal
Evaluator documents actions and problems found with the interface
Actions and comments are recorded by the observer using paper notes, video or audio recording
As the name suggests, this involves someone observing a small group (usually between 3 and 6 people) of (articulate) users as they work through specified tasks or benchmark tests. The users talk aloud as they work, describing not only what they are doing but also what they are thinking while they are doing it. Their actions and their comments are recorded by the observer using paper notes, video or audio recording.
Advantages of the method are that any difficulties of the system are quickly highlighted and the users comments are a very valuable source of information for the developers, particularly about qualitative aspects of the interface. It can be used at any stage of the development and gives rapid feedback which can be quickly incorporated into the next refinement of the system.
Some disadvantages are that the very fact of being observed can affect one’s performance (up or down) so that the test subjects might not be acting like ‘normal’ users. (This is known as “The Hawthorne Effect.)
The technique can also be quite cumbersome in that many notes, or long recordings, are produced and it is quite a job to analyse these, taking perhaps quite a long time. The analysis might even involve more expensive ‘experts’, such as psychologists to analyse, say, body language observed on video tapes.
In addition, the designer is depending on the test subjects being truthful and not holding anything back for fear of embarrassing either themselves or the designer.

48. 6. Think Aloud
To increase the quality of the research, we must avoid the following effects during an observational study:
Hawthorne Effect
User increases performance to please observer
Observer Bias
Observer only sees and records what they want to see
Halo Effect
Observer’s judgement is influenced by another, separate, positive judgement

49. 7. Co-operative Think Aloud
A variation on think-aloud
User and evaluator co-operate in identifying problems
Evaluator asks questions during the session
User can ask for clarification

50. General Points
Evaluation is relevant throughout all stages of development
Different methods are most suited at different stages - rule-of-thumb:
Early design - analytical
Prototype development - observational/ experimental
Late development - survey
A mix of objective and subjective measures is desirable
Remember, we earlier said that evaluation can have two main purposes :
Formative - done throughout the development cycle in order to test ideas, find problems, with a view to improving the design.
Summative - when system more-or-less complete - th check that it is at last ready for ‘the market’.
Example of late survey method might be to include a questionnaire, or an contact address, along with a ‘beta version’ of the software and then invite comment, over a period of months, from people using the Beta version. This sort of method is often used with software such as Web browsers and other programs that can be downloaded from the Internet.