1. Data analysis, interpretation and presentation (Chapter 8 – Interaction Design Text)
The kind of analysis can be performed on the data depends on the goals of the study and the type of data that you have gathered. Typically we can classify them as qualitative, quantitative analysis or a combination of both. Most analysis whether it is quantitative or qualitative begins with an initial reaction or observation from the data. You look for patterns in the data and then follow up the initial analysis with more detailed work. A common mistake the evaluators often make is they let their personal biases and feelings creep in while trying to interpret the data. For instance, if we consider the book example, a initial look at collected data reveals that customer care calls to Sydney take a lot of time
Customer care at Sydney struggles to answer customer questions less efficient
Give more detailed responses
Poor technology support
Customers in Sydney demand a higher level of service
You have to be very careful with the usage of words when you interpret results, words like many, most and all have to be used carefully.
Questions 1, 2

2. Quantitative and qualitative
Quantitative data
Expressed as numbers
Numerical methods to ascertain size, magnitude and amount
Qualitative data
Difficult to measure sensibly as numbers, e.g. count number of words to measure dissatisfaction
Expresses the nature of elements and is represented as themes, patterns, stories
Be careful how you manipulate data and numbers!
Any qualitative data can be turned into a set of numbers, which in turn can be manipulated in a wide variety of ways and then interpreted with respect to your goals. People have a wrong impression that numbers convey a better picture, that is not necessarily true. You have to be careful when you turn qualitative data into numbers and have to make sure that it makes sense when you do that.
Be careful how you represent quantitative data. 50 % of the users found the product real bad is a lot different from saying 2/4 users found it was a real bad product. When you use percentages make sure the raw numbers are mentioned. Always present the context in which you analyze the data.

3. Simple quantitative analysis
Averages
Mean
Median
Mode
Percentages
Graphical representations give overview of data
Percentages are useful for standardizing the data, particularly when you want to compare two or more large sets of data.
Mean: add up values and divide by number of data points
Median: middle value of data when ranked
Mode: figure that appears most often in the data
The way you design a question can affect how well the data is gathered and analyzed. For instance, if you ask the question what do you think of e-bay .com and keep the question as a open-ended one, it will result in a lot of different answers and it will be very difficult to analyze the various answers you get. You need to distinguish between the various different answers you get. However, if the question was Do you like e-bay, you might have to deal with just yes, no or neither.
Graphical representations give a good overview of the data and can be used to identify outliers in the general data. Outliers are sometimes ignored because they distort a pattern that you might see in the data. However, in some cases we need to pay careful attention to see if there are special circumstances surrounding that user and that session.
Question 3

4. Simple qualitative analysis
Recurring patterns or themes
Emergent from data, dependent on observation framework if used
Categorizing data
Categorization scheme may be emergent or pre-specified
Coding
Looking for critical incidents
Helps to focus in on key events
As with qualitative analysis, the first step in qualitative analysis is to gain an overall impression of the data and to start looking for patterns.
As you become familiar with the data, recurring themes and patterns will start to emerge. An example is noticing that most senior managers interviewed expressed frustration at marketing. Patterns in quantitative data can be identified by looking at graphs of the data, however in the case of qualitative data, the evaluator has to immerse themselves in the data. In some cases these patterns will be your actual findings, but in some cases, they are just the starting point for detailed investigations of data. The goal of the study can go a long way in helping to determine them. For instance, if the study was aimed at finding how useful a new widget is and the users keep saying that they wished to see this or that info, then it is a major problem. The users might also say that the title bar is distracting, it is something to take into consideration, but it is not a major problem. You have to keep track of all themes and a good description of these themes. It will ensure that you have a set of observations that make sense with respect to the goals of the study.
Affinity diagrams are used to organized individual ideas and insights obtained from the data into a hierarchy showing common structures and themes. The groups are not pre-defined usually but emerge from the data. You put an insight you gained up and then search for other things that can be grouped with this insight.
You can use a categorization scheme(pre-defined or emergent like in affinity diagrams). Typically you need to prove that the categorization scheme you are using is reliable. This can be done by training a second person to use the categories. When the training is complete, both people analyze the same sample. If there is a large discrepancy then either the training did not work or the categorization has to be refined. You need to talk to the classifiers to determine the reason. You do this until you hit a good inter-reliability rate. The ratio of the number of items grouped similarly by the classifiers/number of items examined.
Content analysis of categorization involves categorizing the data and then studying the frequency of occurrences. Discourse analysis and conversation analysis involves focusing on what was said, what does it mean.
Critical incidents technique –introduced by air force as a way of reporting good and bad performances by the flights. Report behavior that is observable which has made a significant contribution to the activity. It is focused on identifying specific incidents that are significant and then to focus and analyze them in detail using the rest of the data as a way to provide contextual information. It is useful in video analysis, watch for critical events while looking at a video, maintain a high level narrative and label them. These critical/key events can then be analyzed.
Question 4

5. Tools to support data analysis
Spreadsheet – simple to use, basic graphs
Statistical packages, e.g. SPSS
Qualitative data analysis tools
Categorization and theme-based analysis, e.g. N6
Quantitative analysis of text-based data
CAQDAS Networking Project, based at the University of Surrey (
These typically provide facilities to associate labels with sections of data and search the data for key phrases and investigate the relationship between different themes or categories. Quantitative analysis of textual data includes providing support for content analysis.
SPSS – Statistical package for social sciences
N6 and Nvivo

6. Theoretical frameworks for qualitative analysis
Basing data analysis around theoretical frameworks provides further insight
Three such frameworks are:
Grounded Theory
Distributed Cognition
Activity Theory
These are in Ch 3 of the ID book – and we’ll get back to studying them! An intro follows…

7. Grounded Theory Aims to derive theory from systematic analysis of data
Based on categorization approach (called here ‘coding’)
Three levels of ‘coding’
Open: identify categories
Axial: flesh out and link to subcategories
Selective: form theoretical scheme
The data analysis aims to develop theory from systemic analysis and interpretation of empirical data. The idea is to develop a set of concepts and form relationships between them. The idea is to use these concepts to explain or predict phenomenon. Grounded theory, alternates data collection and data analysis. First data is collected and analyzed to identify categories and then the analysis leads to further data collection. Data gathering is guided by the categorization and this approach continues until no new insights emerge and theory is well developed.
Open coding is like categorization we discussed earlier, try and see if we can apply a categorization to the data we have gathered. Axial involves identifying sub-categories from the various categories and then selective involves organizing it around a central category.

8. Distributed Cognition
The people, environment & artefacts are regarded as one cognitive system
Used for analyzing collaborative work
Focuses on information propagation & transformation
A primary objective of distributed cognition is to describe these interactions in terms of how information is propagated through different media. Sort of think of it like a data flow which talks about how the data flows through the various people, artifacts and the environment. The system under study might be how a person reacts with a system(a context level data flow diagram) or how a group of programmers interact and coordinate various activities involved in development of systems.
For example, if the goal is to examine how a team of pilots fly a plane – with a view to improving communications that takes place, then we will focus on interactions and communications that take place between them and their instruments. A good way to begin analyzing and interpreting the data is to describe the official work practices in terms of the routine and the procedures followed and the workarounds that the teams develop when teams cope with the workload. You focus on critical incidents and then problems can be described in terms of communication path ways that are hindered or the breakdowns arising due to information not propagating effectively from one representational state to another.
The joan system.
Question 5

9. Activity Theory
Explains human behaviour in terms of our practical activity with the world
Provides a framework that focuses analysis around the concept of an ‘activity’ and helps to identify tensions between the different elements of the system
Two key models: one outlines what constitutes an ‘activity’; one models the mediating role of artifacts
Activity theory is a product of soviet psychology that explains human behavior in term of practical activity with the world.

10. Individual model
Activity theory models the activities in a hierarchical manner. At the bottom level are operations – routine behaviors that require little conscious attention – example rapid typing. Actions are characterized by some planning – example producing a glossary. The top level is the activity that provides meaningful context for understanding the individual actions – example an essay. Activities can be understood by the motives that elicit them and actions on the basis of goals that guide them and operations by the conditions that identify them. The difference between activity, action and operation is very fluid, actions become operations and activities become actions.

11. Engeström’s (1999) activity system model
Physical artifacts have physical properties that humans respond to. They also embody a set of social practices that reflect a history of particular use.
A baby trying to learn how to use a spoon , initially it is just a physical object the baby can hold, without realizing it has to be held horizontal to prevent spilling, soon with adult mediation/supervision the child learns how to use it. Without mediation however it remains an artifact. The characteristic of human development is the change from a direct model of acting to one of mediation. Activity theory emphasizes the importance of social context. Even when working alone our activities are given meaning by a wider set of practices.
Subject – who performs the activity
Object – On whom the activity is performed.
Tool – the artifact that mediates between the subject and the object
Community – people who share this object
Rules – agreed conventions and practices that cover what it means to be a member of the community
Division of labor – primary means of classifying labor in a workplace.
The biggest problem in AT is determining when something is an activity and when something is an action. Programming a software project (activity) – module(action) – statement(operation)
Question 6

12. Presenting the findings
Only make claims that your data can support
The best way to present your findings depends on the audience, the purpose, and the data gathering and analysis undertaken
Graphical representations may be appropriate for presentation
Other techniques are:
Rigorous notations, e.g. UML
Using stories, e.g. to create scenarios
Summarizing the findings
The best way to present the findings depends on the audience, the original goals of the study and the data gathering and analysis techniques used. Typically evaluations are done to either evaluate a design or to identify requirements. These tend to play a role in what we use to present the findings
Formal notations – good for communicating requirements, we can talk about the flow of data through a system , where it breaks down and explicitly identify a problem. A disadvantage is that it tends to deemphasize the rest of the findings.
Stories are particularly useful when dealing with observation.
Question 7