1. Human Error? No, Bad Design
Chapter 5
Human Error? No, Bad Design

2. Today’s Topics Humans, systems, and error More on data gathering
Designing for communication and coordination

3. Part 1: Humans, Systems and Error
Root cause analysis for error
Why did Air Force stop at human error?
Do not view the system being designed as one combining human and computer/machine actors (e.g. sociotechnical or cyber-human system)
The five whys method
Humans are good at creative and adaptive behavior. The more mechanistic the activity is required to be, …
Deliberate Violations
Occur in all types of contexts (e.g. driving, officework, security, …)
Sometimes because official procedures are developed for legal or accountability reasons
In many situations, work cannot get done without exceptions

4. Errors: Slips and Mistakes
Slips: the goal is correct but the actions not done correctly
Action based – inaccurate performance of planned action
Memory lapse – forgetting to take a planned action
Mistakes: the goal or plan is wrong
Rule based – wrong plan due to poor planning (“wrong rule”)
Knowledge based – wrong plan due to inaccurate/incomplete knowledge
Memory lapse – forgetting at stage of goals, plans, or evaluation
Examples from elevator
Pressing button below correct one along with correct one
Pressing the 1st floor button when going to the lobby

5. Slips Slips are more common for experts than novices Capture slips
Result of more aspects of a task being handled unconsciously
Capture slips
When similar action sequences get confused
Generally a more familiar or recent action sequence replaces another
Example might be finishing a password with more common suffix
More common for experts
Description-similarity slips
Performing an action with a similar item to the one intended
Designers should reduce similar items and reduce number to choose from
Mode errors
Occurs when design has modes where the same user action results in different system actions
Designers should try to avoid modes and make the mode visible when they are necessary
Memory lapse slips
Forgetting where one is in procedure (repeating or not performing steps)
Often caused by interruptions
Designers can show steps and indicate where one is or can create forcing functions enforcing prerequeisits

6. Mistakes Rule-based mistakes Knowledge-based mistakes
Situation is misinterpreted so wrong rule is selected
Correct rule is invoked, but it is faulty or did not account for current situation
Correct rule is invoked, but the outcome is incorrectly evaluated
Knowledge-based mistakes
Occurs in situations requiring more general problem solving
Can be due to a mistaken mental model
Designers need to provide clear conceptual models
Systems need to support users when they encounter previously unknown situations
Cooperative problem solving system
Memory-lapse mistakes
Occur when user forgets previously gathered information about situation, or forgets the plan altogether

7. Social Issues and Error
Social, time, and economic pressure lead to bad decisions
Social pressure also can lead to errors not being reported
Adding more responsible people can cause less attention to tasks
Each may assume others are being more responsible
Need to encourage reporting of errors to fix their causes
Jidoka and Poka-Yoke
NASA aviation safety reporting system

8. Detecting Errors is Non-Trivial
Errors are generally preceded by a number of uncommon events that each by itself is not strongly indicative of error
False alarm
Response to correct action interpreted as failure
May occur
When there are conflicting indicators
Due to different interpretations of plan/task
Garden Path
Incorrect action not discovered until later
Losing context of original problem
Machine can interpret action as correct for some alternative path
Users can assume they know the process without machine
Trivial breaches of understanding can become “fatal”

9. Designing for Error Warnings Computer systems can often add Checklists
Need to be clear from one another and coordinated with one another
Need to gain needed attention without being obnoxious/disabilitating
Computer systems can often add
Different visual cues and locations for different actions/features
Sensibility checks on user actions
Undo and multi-stage undo
Checklists
Common in aviation but not other industries
Design of checklists is still a difficult task
More generally, processes can focus on creating resilience
The Swiss cheese model to minimize likelihood
Notifications when features are lining up for error

10. Part 2: More on Data Gathering
Interviews
Questionnaires
Observation
Choosing and combining techniques

11. Four key issues Setting goals Relationship with participants
Decide how to analyze data once collected
Relationship with participants
Clear and professional
Informed consent when appropriate
Triangulation
Use more than one approach
Pilot studies
Small trial of main study

12. Data recording Notes, audio, video, photographs Notes plus photographs
Audio plus photographs
Video

13. Interviews
Unstructured - are not directed by a script. Rich but not replicable.
Structured - are tightly scripted, often like a questionnaire. Replicable but may lack richness.
Semi-structured - guided by a script but interesting issues can be explored in more depth. Can provide a good balance between richness and replicability.

14. Interview questions Two types: Closed questions are easier to analyze
‘closed questions’ have a predetermined answer format, e.g., ‘yes’ or ‘no’
‘open questions’ do not have a predetermined format
Closed questions are easier to analyze
Avoid:
Long questions
Compound sentences - split them into two
Jargon and language that the interviewee may not understand
Leading questions that make assumptions e.g., why do you like …?
Unconscious biases e.g., gender stereotypes

15. Running the interview
Introduction – introduce yourself, explain the goals of the interview, reassure about the ethical issues, ask to record, present any informed consent form.
Warm-up – make first questions easy and non-threatening.
Main body – present questions in a logical order
A cool-off period – include a few easy questions to defuse tension at the end
Closure – thank interviewee, signal the end, e.g, switch recorder off.

16. Enriching the interview process
Props - devices for prompting interviewee, e.g., a prototype, scenario

17. Questionnaires Questions can be closed or open
Closed questions are easier to analyze, and may be done by computer
Can be administered to large populations
Paper, and the web used for dissemination
Sampling can be a problem when the size of a population is unknown as is common online

18. Questionnaire design
The impact of a question can be influenced by question order.
Do you need different versions of the questionnaire for different populations?
Provide clear instructions on how to complete the questionnaire.
Strike a balance between using white space and keeping the questionnaire compact.
Decide on whether phrases will all be positive, all negative or mixed.

19. Question and response format
‘Yes’ and ‘No’ checkboxes
Checkboxes that offer many options
Rating scales
Likert scales
semantic scales
3, 5, 7 or more points?
Open-ended responses

20. Encouraging a good response
Make sure purpose of study is clear
Promise anonymity
Ensure questionnaire is well designed
Offer a short version for those who do not have time to complete a long questionnaire
If mailed, include a stamped addressed envelope
Follow-up with s, phone calls, letters
Provide an incentive
40% response rate is high, 20% is often acceptable

21. Advantages of online questionnaires
Responses are usually received quickly
No copying and postage costs
Data can be collected in database for analysis
Time required for data analysis is reduced
Errors can be corrected easily

22. Problems with online questionnaires
Sampling is problematic if population size is unknown
Preventing individuals from responding more than once
Individuals have also been known to change questions in questionnaires

23. Observation Direct observation in the field
Structuring frameworks
Degree of participation (insider or outsider)
Ethnography
Direct observation in controlled environments
Indirect observation: tracking users’ activities
Diaries
Interaction logging

24. Structuring frameworks to guide observation
- The person. Who? - The place. Where? - The thing. What?
The Goetz and LeCompte (1984) framework: - Who is present? - What is their role? - What is happening? - When does the activity occur? - Where is it happening? - Why is it happening? - How is the activity organized?

25. Ethnography (1)
Ethnography is a philosophy with a set of techniques that include participant observation and interviews
Debate about differences between participant observation and ethnography
Ethnographers immerse themselves in the culture that they study
A researcher’s degree of participation can vary along a scale from ‘outside’ to ‘inside’
Analyzing video and data logs can be time-consuming
Collections of comments, incidents, and artifacts are made

26. Ethnography (2) Co-operation of people being observed is required
Informants are useful
Data analysis is continuous
Interpretivist technique
Questions get refined as understanding grows
Reports usually contain examples

27. Other Observational Methods
Direct observation in a controlled setting
Think-aloud technique
Indirect observation
Diaries
Interaction logs

28. Choosing and combining techniques
Depends on
The focus of the study
The participants involved
The nature of the technique
The resources available

29. Data Gathering Summary
Three main data gathering methods: interviews, questionnaires, observation
Four key issues of data gathering: goals, triangulation, participant relationship, pilot
Interviews may be structured, semi-structured or unstructured
Questionnaires may be on paper, online or telephone
Observation may be direct or indirect, in the field or in controlled setting
Techniques can be combined depending on study focus, participants, nature of technique and available resources

30. Part 3: Designing for Conversation and Coordination
Conversational mechanisms
Coordination mechanisms
Awareness mechanisms
Examples of technologies designed to extend how people
talk and socialise
work together
play and learn together

31. Conversational mechanisms
Various mechanisms and ‘rules’ are followed when holding a conversation, e.g. mutual greetings
A: Hi there
B: Hi!
C: Hi
A: All right?
C: Good, how’s it going?
A: Fine, how are you?
C: OK
B: So-so. How’s life treating you?

32. Conversational rules
Sacks et al. (1978) work on conversation analysis describe three basic rules:
Rule 1: the current speaker chooses the next speaker by asking an opinion, question, or request
Rule 2: another person decides to start speaking
Rule 3: the current speaker continues talking

33. Conversational rules Turn-taking used to coordinate conversation
A: Shall we meet at 8?
B: Um, can we meet a bit later?
B: Wow, look at him?
A: Yes what a funny hairdo!
Back channeling to signal to continue and following
Uh-uh, umm, ahh

34. More conversational rules
Farewell rituals
Bye then, see you, yer bye, see you later….
Implicit and explicit cues
e.g., looking at watch, fidgeting with coat and bags
explicitly saying “Oh dear, must go, look at the time, I’m late…”

35. Breakdowns in conversation
When someone says something that is misunderstood:
Speaker will repeat with emphasis:
A: “this one?”
B: “no, I meant that one!”
Also use tokens:
Eh? Quoi? Huh? What?

36. What happens in technology-mediated conversations?
Do same conversational rules apply?
Are there more breakdowns?
How do people repair them for:
Phone? ?
IM?
Texting?

37. Synchronous computer-mediated communication
Conversations are supported in real-time through voice and/or typing
Examples include video conferencing, VOIP, MUDs and chat
Benefits include:
Not having to physically face people may increase shy people’s confidence
Allows people to keep abreast of the goings-on in an organization without having to move from their office
Problems:
Difficult to establish eye contact with images of others
People can behave badly when behind the mask of an avatar

38. Asynchronous computer-mediated communication
Communication takes place remotely at different times
, newsgroups, texting
Benefits include:
Read any place any time
Flexible as to how to deal with it
Can make saying things easier
Problems include:
FLAMING!!!
Message overload
False expectations as to when people will reply

39. Coordination mechanisms
When a group of people act or interact together they need to coordinate themselves
e.g., playing football, navigating a ship
They use:
verbal and non-verbal communication
schedules, rules, and conventions
shared external representations

40. Verbal and non-verbal communication
Talk is central
Non-verbal also used to emphasize and as substitute
e.g., nods, shakes, winks, glances, gestures and hand-raising
Formal meetings
explicit structures such as agendas, memos, and minutes are employed to coordinate the activity

41. Schedules, rules and conventions
Schedules used to organize regular activities in large organizations
Formal rules, like the writing of monthly reports enable organizations to maintain order and keep track
Conventions, like keeping quiet in a library, are a form of courtesy to others

42. Shared external representations
Common method used to coordinate collaborative activities,
e.g., checklists, tables, to-do lists
They can provide external information on:
who is working on what
when it is being worked on
where it is being worked on
when a piece of work is supposed to be finished
whom it goes to next

43. Collaborative technologies to support coordination
There are a variety of software tools designed to support scheduling, planning and coordinating
e.g., group calendars, electronic schedulers, project management tools, and workflow tools
Need to get balance between human and system control
too much system control and the users will rebel
too little control and the system breaks down

44. Awareness mechanisms
Involves knowing who is around, what is happening, and who is talking with whom
Peripheral awareness
Keeping an eye on things happening in the periphery of vision
Overhearing and overseeing - allows tracking of what others are doing without explicit cues

45. Lo tech awareness mechanism

46. Designing technologies to support awareness
Provide awareness of others who are in different locations
Early example was media spaces
“extend the world of desks, chairs, walls and ceilings” (Harrison et al, 1997)
Examples: Clearboard and Portholes

47. Portholes (Xerox PARC)
Regularly updated digitized images of people in their offices appeared on everyone’s desktop machines throughout day and night
Portholes, developed at Xerox PARC research labs (Dourish and Bly, 1992).The system presented regularly updated digitized video images of people in their offices from a number of different locations (in the US and UK). These were shown in a matrix display on people’s workstations. Clicking on one of the images had the effect of bringing up a dialog box, providing further information about that individual (e.g. name, phone number) together with a set of lightweight action buttons (e.g. the person, listen to a pre-recorded audio snippet). The system provided changing images of people throughout the day and night in their offices, letting others see ‘at a glance’ whether they were in their offices, what they were working on and knowing who was around (see figure 10). Informal evaluation of the set-up suggested that having access to such information led to a shared sense of community.

48. Notification systems
Users notify others as opposed to being constantly monitored
Provide information about shared objects and progress of collaborative tasks
examples: Tickertape, Babble

49. Elvin
Elvin is a distributed awareness system that provides a range of client services (Segall and Arnold, 1997)
It includes Tickertape, one of the first lightweight messaging systems

50. Conversation and Coordination Key Points
Social mechanisms, like turn-taking, conventions, etc., enable us to collaborate and coordinate our activities
Keeping aware of what others are doing and letting others know what you are doing are important aspects of collaborative working and socialising
Many collaborative technologies systems have been built to support collaboration