1. Innovative Design & Manufacturing Research Centre University of Bath World-leading research in engineering design and manufacture. 1 IdMRC Social Research Methods Autumn Lecture-Workshop Series

2. BATH September 20092 Design Research Methods: Observation and Coding Janet McDonnell

3. BATH September 20093 Note This collection of slides supports a lecture on the topic of observation and coding. The slides in this set are not intended to serve independently of the lecture as a stand-alone introduction to, or overview of, the subject. The lecture includes the use of video and other visual material excised from this sub-set of slides which is solely intended to support note taking at the lecture. Janet McDonnell

4. BATH September 20094 Outline [software tools] CASE STUDY: from the categorical to the conjectural Knowledge claims – from nature to narrative Traceability and reliability What is the data from observation Approaches to research using observation data CASE STUDY: single dataset, multiple approaches to analysis Pointers for quality

5. BATH September 20095 Software Tools - Lets get them out of the way Behavioural Research : recording behaviour, eye tracking, facial expression and other types of behaviour recognition, synchronising with transcripts and support for coding, etc. www.mangold-international.com (Interact) www.mangold-international.com www.tracksys.co.uk (Observer XT) www.tracksys.co.uk Qualitative Data Analysis – content analysis of texts – from transcripts, interview data, etc. linking in images and video clips, qualitative data base management, collaboration www.qsrinternational.com (Nvivo) www.qsrinternational.com www.atlasti.com (ATLAS-ti) www.atlasti.com Quantitative data analysis Stats packages, graphing and diagramming tools Transcription aids

6. BATH September 20096 Anatomy of a small scale enquiry An investigation into the exchange of ideas and information between an architect and building users in the early stages of a building (re)design project before the design brief or any drawings have been produced. We look at the type of information users exchange. We are interested in what influences the information exchanges - context of the meetings and - the conversational strategies of the architect. Classifications were determined in advance of coding the data.

7. BATH September 20097 Data and method Participant observation: interaction as it occurs in natural setting Data consists of transcripts acquired from audio-taped meetings Focus on manifest content Segment transcripts into units of analysis Apply predetermined codes* Inter coder issues Data analysis and data interpretation Construct an account of the topic * Based on P. Medway (2000) Writing and Designing in Architectural Education in A.Pare (ed) Transitions: writing in academic and workplace settings, Hampton Press, pp 89-129.

8. BATH September 20098 Conversation summaries CONVERSATION1234 Length mins:secs10:2025:045:2720:37 No. of turns14928543247 No. of information contributions 16431230

9. BATH September 20099 Information contributions CONVERSATION1234 Building User923129 Architect6119 Deputy Head Teacher1812

10. BATH September 200910 Codes for information contributions CODEDEFINITION 1Functional or structural naming Elements and properties of the building or space (floor, exposed brickwork, etc.) – its objective character 2Perceptual awareness Attributes of the building or space that can be perceived (visual, acoustics, etc.) 3Phenomenological experience Feelings and associations made when experiencing the space (e.g. solidity) 4Symbolic meaning Evocation of ideas unrelated to the structural form (mystery, memory, etc.)

11. BATH September 200911 shelving reachable from the floor easy to clean surfaces light and quiet study areas a place where it feels safe to think a link between our past and our future

12. BATH September 200912 Information contributions CONVERSATION1234 Building User923129 Architect6119 Deputy Head Teacher1812

13. BATH September 200913 It was noticeable that some users were competent at discussing contributions in a more informed, technical manner than others. There were differences between the users understanding of the structural properties of buildings and their range of abilities to articulate spatial preferences for the design of the workspace. Compare U3: … my wish is I suppose to take out this joint wall and incorporate this store and this area into a larger area … with U1: … if I had my wish then I would basically um take this wall down … Although conversation 3 was shorter than the others, U3s ability to converse in a designerly language made it easier … to share a common understanding of the properties of the space.

14. BATH September 200914 Types of information contribution CONVERSATION1 Ar BU 2 Ar BU 3 Ar BU 4 Ar BU Functional6 89 22 9 9 14 Perceptual 7 Phenomenological 22 4 3 Symbolic 3

15. BATH September 200915 The contribution of information with a symbolic meaning was infrequent. Symbolic references only cropped up in conversation 4 … all contributed by the deputy head teacher rather than the primary user of the space. We note also that the symbolic meaning discussed concerned what it meant to be part of the school and only once was the symbolic meaning of the space being designed mentioned strictly. The three instances are D: … this is regarded as a privilege … to give them a kind of base so that they can start acting can I say more acceptable, normal circumstances in their reaction and inter-reactions with other people and D: … thats why we still insist on them being in school uniform so theyre still part of the school to stamp identity U1: er yeah the whole idea is that theyre to be re-integrated into school and D: … we use this facility as an escape to do course work in the privacy of here as opposed to the pressures in the school

16. BATH September 200916 Claims, evidence and … truth what is provably true logico-deductive reasoning, exhaustive search what is probably true statistical reasoning what is plausibly true the evidence of arguments that are sincere and will convince a reasonable audience

17. BATH September 200917 Knowledge claims from research Provable and Probable Positivist : natural sciences : experimental reproducibility; falsifiable hypotheses Plausible Interpretivist : cultural and social sciences : plausibility – traceability; recoverability; narratives providing understanding Emancipatory Critical theorist : social sciences : emancipatory force

18. BATH September 200918 Transparency in enquiry Declared or shared framework, methodology, methods: allow process by which results are obtained to be recovered by others The process of enquiry may lead to contributions to knowledge in each aspect (framework, methodology, research area) Sharing of the meaning-making addresses subjectivity objections – a route to agreement framework of ideas methodology area of research concern

19. BATH September 200919 Inter coder reliability Context: decisions about what is required to adequately serve the claims and evidence Kappa coefficient is a measure of agreement between coders on the classification of qualitative data Measures agreement in using a defined coding scheme in a prescribed manner. Kappa takes into account the potential for chance agreement. Does not take account of degrees of disagreement. Is a good test of whether categories are sufficiently distinct from each other and sufficiently clearly defined for more than one coder to code reliably using them. So can be used to test identification of units of analysis and application of codes. [J Carletta (1996) Computational Linguistics 22(2) 249-254 Assessing agreement on classification tasks: the kappa statistic. ]

20. BATH September 200920 45 AdrianOK ++ so moving into a little bit more detail 46 [paper shuffling, 7 seconds pause] 47 Adrianerm starting with the entrance, we arrive at the car park here and we 48 move forward to the entrance and waiting area here we've upped the 49 importance of this area one of the items on the minutes last time was 50 to increase the size of the waiting area 51 Angelayes 52 Adrianand we virtually doubled in size errm we've actually extended the 53 canopy over it as well to make more of an entrance to make more of a 54 greeting zone so there's lots of space here with seats for people to wait 55 for taxis or whatever at the end of their end of their 56 Angela/hmm-mm\ 570.03.00Adrian/service\ places for people to to actually stand and mill in if they wish 58 to stand out here before they move into the waiting area then come 59 into the porte-cochere so it's an even grander entrance 60 Angela/OK yes\ 61 Adrian/than we had\ before 62 Adrianerm one of your next items on the list was to widen the access way so 63 that two vehicles perhaps two limousines or a hearse and limousine 64 could park side by side in this area so this has been widened so its 65 now possible as I say for two vehicles ++ to park here ++ before 66 moving through further on ++ one of the next points made was the size 67 of the sanctuary we've increased the diameter of this by correct me if 68 I'm wrong Toby three hundred millimeters internally 69 Tobyits three point four Extract from a transcript

21. BATH September 200921 Transcription conventions + pause of one second ( ++ two seconds, and so on) …/….\…crude indicators of simultaneous speech …/……\… NOemphatic stress GOOGLE names of people, organisations, and so on ……material omitted -incomplete or cut off utterance ( )unclear utterance (over there)unclear utterance, transcribers best guess within brackets [points at drawings]transcribers comments [laughs]paralinguistics [For a more comprehensive scheme refer to Gail Jeffersons notation described in J Atkinson and J Heritage (eds) Structures of Social Action (1984) CUP, pp ix-xvi]

22. BATH September 200922 Ways of approaching research using observation data Starting with a template e.g. a model or a theory, classifications determined in advance that are imposed on the data Starting with the data e.g. looking for emergent patterns, evolving classification to cover the phenomena observed (coding scheme not inevitable consequence) Flexible research agenda vs. rigid research agenda e.g. starting with a scheme but being open to the unclassifiable (and to new classes); developing hypotheses to test based on indications of possible patterns, causations

23. BATH September 200923 Common Dataset Multi-angle video recordings, plans, drawings, sketches, flip-charts, transcripts from meetings from two authentic design projects Architectural Design Project Data Two meetings 8 months apart between architect and clients/building users Brief to design a crematorium with chapel and related provision including offices, vestry, waiting rooms, parking, landscaping Engineering / Product Design Project Data Two meetings 3 days apart among engineers and other specialists from the same company – multi-disciplinary groups Brief to develop novel product ideas for sort of digital pen to exploit a new technology 24 different studies on themes relevant to research into design thinking Anatomy of a large collaboration: DTRS7

24. BATH September 200924 How the data was tackled Styles of approaching the data Starting with a template : e.g. a model or a theory Starting with the data Flexible agenda vs. rigid agenda Themes Understanding Process Values in Designing Aspects of Design Cognition Design Process Models Language, Discourse, and Gesture Designing Contexts Objects, References, and Representations

25. BATH September 200925 Types of analysis : imposition of a model or theory Focus of attention: uses of analogy in designing Finding:a previously unreported use of analogy, namely function-finding in creative design [Ball and Christensen, 2009] Focus of attention: object references during designing Finding:object references in meetings between co- workers are directed effectively to draw attention efficiently to features essential to the comparison [Stacey et al., 2009] Focus of attention: language of appraisals Finding:positive and negative appraisals have different effects on the focus of attention and knowledge generation during design [Dong et al., 2009]

26. BATH September 200926 Types of analysis : fine grained analysis of the data Focus of attention: gestures and individual turns at talking Finding:reveals fundamental distinctions between view of designing as goal-oriented and as goal-directed [Glock, 2009] Focus of attention: designing through talk-in-interaction Finding:design concept has a special status – less negotiable – the preserve of the designated design expert [Luck, 2009] Focus of attention: social order – the rules of interaction Finding:adherence to interactional norms interferes with the rules of brainstorming [Matthews, 2009]

27. BATH September 200927 Types of analysis : testing the limitations of prior findings or models Focus of attention: consideration of context during designing Finding:support for claims that experienced designers explore broad context before close context is considered in depth [Atman et al., 2009] Focus of attention: a design teams development of shared notions about the task and the teams operation (indicated by verbal communication); based on a prior model of the development of sharedness in teams Findings:some expected findings – predicted by the model; some unexpected findings (not predicted by prior model) – only some of these accounted for by particularities of the data [Badke-Schaub et al., 2009]

28. BATH September 200928 Quality of work For in-vivo studies err on the side of over collection of data And/or iterate over observation/data collection and analysis … and dont make excuses Coding – invest in adequate granularity (beware of findings which are artifacts of the coding or analysis) An underused question: what else could account for the findings? Consider the arguments, claims and evidence explicitly at epistemological level Choose research methods that will provide the supporting evidence you need (e.g. inter coder agreement if that is critical) Choose language with care – consider the baggage e.g. hypothesis, subjects

29. BATH September 200929 Further information The small scale enquiry was conducted with Dr Rachael Luck of Reading University and is published in Design Studies 27 (2006) pp 141-166. DTRS7 work is published as a book About:Designing J.McDonnell and P.Lloyd (eds), 2009 and parts of it in two journal special issues Design Studies 30 (2) and CoDesign 5 (1) both in March 2009. Janet McDonnell Central Saint Martins Southampton Row London WC1B 4AP j.mcdonnell@csm.arts.ac.uk www.csm.arts.ac.uk/janet-mcdonnell.htm