1. Perspectives on Sensemaking
Simon Attfield
Visual Analytics Summer School, 2010

2. Picture by Ed Sanders

3. Sensemaking is...
The integration of information and knowledge to form an interpretation (Attfield, last weekend)

5. Cognitive dissonance
'Oh, you aren't even ripe yet! I don't need any sour grapes.'
The fox and the grapes
‘adaptive preference formation’
Picture credit - Milo Winter

6. A search for coherence knowledge expectation experience information
values
aspirations
information
data

7. Key questions
Cognition - How do people reason with information? How do they find meaning, reconstruct a domain, reach conclusions, and derive new puzzles and questions?
Social interaction – How do people interact around information? How do they share, convince, question, agree, cajole and debate?
Behaviour - How do people manage information? How do they adapt and shape their environments in order to achieve cognitive and social processes?
Tools - What can all of this say about the resources and functionality that can provide the leverage to do these things more naturally, invisibly and effectively?

8. Expectation and language
Battered Child Syndrome
Up to 1940’s no concept of battered child syndrome.
Late 1940’s - a surprise - a paediatric radiologist noticed a few cases of child injuries not adequately accounted for by parents medical ‘histories’.
Discrepancy explained away.
More cases noticed.
In 1953 a comprehensive report identified 749 cases.
Within a few years laws introduced mandating reporting.
Weick, 1995

9. How do we makes sense of the world with very little information
How do we makes sense of the world with very little information? – schemata
Data structures for representing generic concepts stored in memory
Introduced into psychology by Bartlett
‘The building blocks of cognition’ (Rummelhart, 1980)
Variables that can be bound to an instantiation – e.g. Buying
A schema is a ‘theory’ – goes beyond our perception
Comprehension is a process of hypothesis testing

10. Eh?
The procedure is actually quite simple.  First you arrange things into different groups depending on their makeup.  Of course, one pile may be sufficient depending on how much there is to do.  If you have to go somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set.  It is important not to overdo any particular endeavor.  That is, it is better to do too few things at once than too many.  In the short run this may not seem important, but complications from doing too many can easily arise.  A mistake can be expensive as well.  The manipulation of the appropriate mechanisms should be self-explanatory, and we need not dwell on it here.  At first the whole procedure will seem complicated.  Soon, however, it will become just another facet of life.  It is difficult to foresee any end to the necessity for this task in the immediate future, but then one never can tell.
From Bransford and Johnson, 1973

11. The procedure is actually quite simple
The procedure is actually quite simple.  First you arrange things into different groups depending on their makeup.  Of course, one pile may be sufficient depending on how much there is to do.  If you have to go somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set.  It is important not to overdo any particular endeavor.  That is, it is better to do too few things at once than too many.  In the short run this may not seem important, but complications from doing too many can easily arise.  A mistake can be expensive as well.  The manipulation of the appropriate mechanisms should be self-explanatory, and we need not dwell on it here.  At first the whole procedure will seem complicated.  Soon, however, it will become just another facet of life.  It is difficult to foresee any end to the necessity for this task in the immediate future, but then one never can tell.
From Bransford and Johnson, 1973

12. The data-frame theory of sensemaking
Klein, Phillips, Rall, Peluso (2007)
Frame
An explanatory structure that links elements e.g. a story, a map, a script, a plan.
Accounts for data and guides the search for more data.
Inferred from a few key anchors (3 or 4 at most)
Functions as a hypothesis
Data
Information from the world.
The interpreted signals of events.
Process proceeds through fitting data to the frame while testing and improving the frame.

13. Klein, Phillips, Rall, Peluso (2007)

14. Klein, Phillips, Rall, Peluso (2007)
Depends on data, goals, repertoire and stance. Pattern matching.
Klein, Phillips, Rall, Peluso (2007)

15. Klein, Phillips, Rall, Peluso (2007)
Depends on data, goals, repertoire and stance. Pattern matching.
Add details.
Klein, Phillips, Rall, Peluso (2007)

16. Klein, Phillips, Rall, Peluso (2007)
Depends on data, goals, repertoire and stance. Pattern matching.
Add details.
Fundamental surprise (Lanir, 1991).
Navigator’s frame- breaker.
Emotional response
Experts have more differentiated frames
(Feltovich, 84)
Klein, Phillips, Rall, Peluso (2007)

17. Klein, Phillips, Rall, Peluso (2007)
Depends on data, goals, repertoire and stance. Pattern matching.
Explain away data.
Knowledge shields.
Fixation error produced by inaccurate anchor early on.
Add details.
Fundamental surprise (Lanir, 1991).
Navigator’s frame- breaker.
Emotional response
Experts have more differentiated frames
(Feltovich, 84)
Klein, Phillips, Rall, Peluso (2007)

18. Klein, Phillips, Rall, Peluso (2007)
Depends on data, goals, repertoire and stance. Pattern matching.
Explain away data.
Knowledge shields.
Fixation error produced by inaccurate anchor early on.
Add details.
Fundamental surprise (Lanir, 1991).
Navigator’s frame- breaker.
Emotional response
Experts have more differentiated frames
(Feltovich, 84)
Nurses in ICU maintaining multiple frames.
Tracking up to three frames.
Logical competitor set.
Klein, Phillips, Rall, Peluso (2007)

19. Klein, Phillips, Rall, Peluso (2007)
Depends on data, goals, repertoire and stance. Pattern matching.
Explain away data.
Knowledge shields.
Fixation error produced by inaccurate anchor early on.
Add details.
Fundamental surprise (Lanir, 1991).
Navigator’s frame- breaker.
Emotional response
Experts have more differentiated frames
(Feltovich, 84)
Nurses in ICU maintaining multiple frames.
Tracking up to three frames.
Logical competitor set.
Duncker’s radiation problem.
Klein, Phillips, Rall, Peluso (2007)

20. Klein, Phillips, Rall, Peluso (2007)
Depends on data, goals, repertoire and stance. Pattern matching.
Explain away data.
Knowledge shields.
Fixation error produced by inaccurate anchor early on.
Add details.
Fundamental surprise (Lanir, 1991).
Navigator’s frame- breaker.
Emotional response
Experts have more differentiated frames
(Feltovich, 84)
Explicit search.
Nurses in ICU maintaining multiple frames.
Tracking up to three frames.
Logical competitor set.
Duncker’s radiation problem.
Klein, Phillips, Rall, Peluso (2007)

21. Reasoning in sensemaking
Klein, Phillips, Rall, Peluso (2007)
Experts and novices reason in the same way, except novices find it difficult to distinguish signal from noise and are more reluctant to speculate.
Experts have a richer repertoire of frames – greater variety, more coverage, finer differentiation. More precise expectations.
Abductive reasoning predominates in sensemaking – ‘reasoning to the best explanation’ – based on plausibility.

22. A symmetry of inference
If it rains in the night the lawn will be wet in the morning.
It rained last night.
The lawn is wet (said in the morning). 1 1 2 1 2 2
Deduction
Induction
Abduction

23. A symmetry of inference
When someone with drugs in their pocket sees a policeman they put their hand in their pocket.
A man with drugs in his pocket saw a policeman
The man put his hand in his pocket 1 1 2 1 2 2
Deduction
Induction
Abduction

27. Will you ever look at things the same again?

28. The rain example
If it rains in the night, the lawn will be wet in the morning.
If the children get up in the night and play with the hose, the lawn will be wet in the morning.
If a bush-fire fighting helicopter on a training mission in the middle of the night accidently drops its load of water over our garden, the lawn will be wet in the morning.

29. What rule am I thinking of?
I am thinking of a rule the describes the relationship between three numbers.
The sequence satisfies the rule.
Your job is to discover the rule by suggesting other number sets - I will tell you whether each one satisfies the rule.
When you think you know the rule, tell me.

30. Confirmation bias Peter Wason
Challenged subjects to identify a rule applying to triplets of numbers
e.g
Subjects seemed to test only positive examples, e.g. each number is two greater than its predecessor (11, 13, 15)

31. Confirmation bias Karl Popper
Marxism and psychoanalysis explain everything?
Worked with Alfred Adler, co-founder of psychoanalysis
Alder analysed child in terms of theory of inferiority without seeing the child.
Picture credit:
Popper, 1963

32. Hang on. how does this relate to visual analytics
Hang on... how does this relate to visual analytics? Pirolli and Card’s model of intelligence analysis
Pirolli and Card (2005)

33. Attfield and Blandford’s model of legal investigations
Externalises evidence and reasoning
Structures thinking
Reveals gaps
Can enforce rigor
Attfield and Blandford (in press)

34. Representing the story with chronologies
P1 I think it’s a very natural way for us to think here, we always use chronologies, our great organising basis. […] I had a team of five or six people and I allocated responsibilities to each of these people saying “Right you’re going to become the master of [issue a], I’m going to do [issue b], [issue c], [issue d]. Someone else is going to do [issue e]”

35. Representing the story with chronologies
Evidence link ?
A met B at <location > on <date> …
B flew to <location > on <date> …
B met C at <location > on <date> …

36. Support recursive decomposition
Theories
Questions
Info. seeking strategies
Evidence
Line of enquiry
Knowledge reps
Investigators
Lines of enquiry

37. Support recursive decomposition
Theories
Questions
Info. seeking strategies
Evidence
Knowledge reps
Lines of enquiry
Investigators
Theories
Questions
Info. seeking strategies
Evidence
Line of enquiry
Knowledge reps
Investigators
Lines of enquiry

38. Support recursive decomposition
Theories
Questions
Info. seeking strategies
Evidence
Knowledge reps
Lines of enquiry
Investigators
Theories
Questions
Info. seeking strategies
Theories
Questions
Info. seeking strategies
Evidence
Knowledge reps
Lines of enquiry
Investigators
Evidence
Line of enquiry
Knowledge reps
Investigators
Lines of enquiry

39. Support recursive decomposition
Theories
Questions
Info. seeking strategies
Evidence
Knowledge reps
Lines of enquiry
Investigators
Theories
Questions
Info. seeking strategies
Theories
Questions
Info. seeking strategies
Evidence
Knowledge reps
Lines of enquiry
Investigators
Evidence
Line of enquiry
Knowledge reps
Investigators
Lines of enquiry
Theories
Questions
Info. seeking strategies
Evidence
Knowledge reps
Lines of enquiry
Investigators

40. Analysis of Competing Hypotheses

41. Analysis of Competing Hypotheses

42. Analysis of Competing Hypotheses
Starts with a set of possibilities rather than a most likely alternative for which the analyst seeks confirmation (avoids satisficing)
Identifies the few items of evidence or assumptions that have greatest diagnostic value.
Seeks evidence to refute hypotheses. Most probable hypothesis is one with least evidence against it (generally).

43. Analysis of Competing Hypotheses (ACH)
1. Identify hypotheses that merit consideration
Heuer’s example...
Q: Will Iraq retaliate for US bombing of its intelligence headquarters?
H1 - Iraq will not retaliate.
H2 - It will sponsor some minor terrorist actions.
H3 - Iraq is planning a major terrorist attack.
If you don’t generate the right hypothesis you wont get the right answer
Use a group of analysts

44. Analysis of Competing Hypotheses (ACH)
2. List significant evidence for and against
Heuer’s example...
E1. Saddam public statement not to retaliate.
E2. Absence of terrorist offensive during 1991 gulf war
E3. Assumption that Iraq would not want to provoke another US attack
E4. Increase in frequency/length of monitored Iraqi agent radio broadcasts
E5. Iraqi embassies instructed to take increased precautions.
E6. Assumption that failure to retaliate would be unacceptable loss of face for Saddam
Include assumptions
Ask what should I expect to see or not see?
Note the absence of evidence as well as its presence.

45. Analysis of Competing Hypotheses (ACH)
3. Prepare a matrix with hypotheses across the top and evidence down the side. Analyse how they relate. Consider diagnosticity
Hypotheses
Evidence
No retaliation
Minor actions
Major attack
E1. Saddam public statement not to retaliate. +
E2. Absence of terrorist offensive during 1991 gulf war -
E3. Assumption - Iraq would not want to provoke another US attack
E4. Increase in frequency/length of monitored Iraqi agent radio broadcasts
E5. Iraqi embassies instructed to take increased precautions.
E6. Assumption - failure to retaliate would be unacceptable loss of face for Saddam --

46. Analysis of Competing Hypotheses (ACH)
4. Refine the matrix. Reconsider hypotheses and delete evidence and arguments with no diagnostic value.
Hypotheses
Evidence
No retaliation
Minor actions
Major attack
E2. Absence of terrorist offensive during 1991 gulf war + -
E3. Assumption - Iraq would not want to provoke another US attack
E4. Increase in frequency/length of monitored Iraqi agent radio broadcasts
E5. Iraqi embassies instructed to take increased precautions.
E6. Assumption - failure to retaliate would be unacceptable loss of face for Saddam --
More hypotheses needed? Is your thinking influenced by evidence or assumptions not included?

47. Analysis of Competing Hypotheses (ACH)
5. Draw tentative conclusions about likelihood of hypotheses. Work down the matrix. Let them compete for favour.
Hypotheses
Evidence
No retaliation
Minor actions
Major attack
E2. Absence of terrorist offensive during 1991 gulf war + -
E3. Assumption - Iraq would not want to provoke another US attack
E4. Increase in frequency/length of monitored Iraqi agent radio broadcasts
E5. Iraqi embassies instructed to take increased precautions.
E6. Assumption - failure to retaliate would be unacceptable loss of face for Saddam --
Hypothesis with fewest minuses is probably best, BUT You can never prove a hypothesis true.

48. Analysis of Competing Hypotheses (ACH)
6. Consider how sensitive your analysis is to a few items of evidence. What if they were mistaken?
Hypotheses
Evidence
No retaliation
Minor actions
Major attack
E2. Absence of terrorist offensive during 1991 gulf war + -
E3. Assumption - Iraq would not want to provoke another US attack
E4. Increase in frequency/length of monitored Iraqi agent radio broadcasts
E5. Iraqi embassies instructed to take increased precautions.
E6. Assumption - failure to retaliate would be unacceptable loss of face for Saddam --
Question the linchpin assumptions – a common source of error. What about deception?

49. Analysis of Competing Hypotheses (ACH)
7. Report conclusions – discuss relative likelihood of all hypotheses.
Hypotheses
Evidence
No retaliation
Minor actions
Major attack
E2. Absence of terrorist offensive during 1991 gulf war + -
E3. Assumption - Iraq would not want to provoke another US attack
E4. Increase in frequency/length of monitored Iraqi agent radio broadcasts
E5. Iraqi embassies instructed to take increased precautions.
E6. Assumption - failure to retaliate would be unacceptable loss of face for Saddam --
There is always the possibility of being wrong. Decision makers need to evaluate.

50. Analysis of Competing Hypotheses (ACH)
8. Identify future observations that may indicate events are taking a different course than expected.
Hypotheses
Evidence
No retaliation
Minor actions
Major attack
E2. Absence of terrorist offensive during 1991 gulf war + -
E3. Assumption - Iraq would not want to provoke another US attack
E4. Increase in frequency/length of monitored Iraqi agent radio broadcasts
E5. Iraqi embassies instructed to take increased precautions.
E6. Assumption - failure to retaliate would be unacceptable loss of face for Saddam --
Analytic conclusions should always be considered as tentative.

51. Summary
Sensemaking
Integration of data and knowledge to form an interpretation.
Dependant upon knowledge, experience, values, aspirations, expectations.
Schemata and abduction can play a major role.
The process is powerful BUT fallible.
Schematisation can make sensemaking more systematic (e.g. ACH).

52. References
Aesop (c 500BC) The Fox and the Grapes. Weick K. (1995) Sensemaking in Organisations. Sage, London, England. Klein G.A., Phillips J.K., Rall E.L., Peluso D.A.: A Data-frame Theory of Sensemaking. In: Expertise Out of Context: Proc. of the Sixth International Conf. on Naturalistic Decision Making (Pensacola Beach, Florida, May 15-17, 2003). Lawrence Erlbaum Assoc. Inc, US, (2007) pp , Popper K. (1963) Science as Falsification: In Conjectures and Refutations London: Routledge and Keagan Paul, 1963, pp , also at Pirolli P & Card S (2005) The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis, Proceedings of the International Conference on Intelligence Analysis (McLean, VA) Attfield S. & Blandford A . (in press) Making Sense of Digital Footprints in Team-based Legal Investigations: The Acquisition of Focus. Human Computer Interaction Journal, Special Issue on Sensemaking. Heuer R. J. (1999) The Psychology of Intelligence Analysis: Center for the Study of Intelligence, Central Intelligence Agency, available at Jones M.D. (1995) The Thinker’s Toolkit: 14 Powerful Techniques for Problem Solving. Three Rivers, New York.