1. CSCW Evaluation Techniques Presented by: Christopher Edwards

2. Overview of Presentation Evaluation Techniques Understanding Ethnography Using Ethnography in CSCW Understanding Ethnomethodology Ethnomethodology and CSCW Technomethodology Conclusion

3. Evaluation Overview Olson and Olson. What are we Evaluating? Evaluations Techniques… Internal/External Validity Conclusion

4. Characteristics of Groups Individuals Differ in: Skills Ability Knowledge Personalities Motivations Agendas

5. Characteristics of Organizations System comprised of people and technology Social Technology Physical Technology Comprised of multiple actors Dependency on Communication Information Processing Entities

6. Characteristics of Task Tasks involve different types of material Physical, Digital or Ethereal Ease or Difficulty of Task Differ on Core Activity Subtasks – Tightly Coupled/Loosely coupled

7. Characteristics of Environment Physical Environment Distance between Group members CSCW technologies designed to overcome Contextual Time When in the day the interaction occurs Effects on Distant Group member

8. Characteristics of Technology Increasingly Varied Technologies to Support Conversation Auditory/Visual Back channels/gestures Technologies to Support Shared Work Objects that support work Fit of Tool to Material

9. Process Technology Deployment How and Why Process Analysis Why outcomes were affected Progress of Task Communication process Examined through time scales

10. Outcomes The initial outcome of using technology Quality of work Measure outcomes at every level Group Outcome Organizational Productivity

11. Conceptual Framework for CSCW Studies Group Organization Task Environment Technology ProcessOutcomes

12. Tools used to evaluate CSCW technologies

13. The Survey Set of questions Fixed Alternatives Statistically Analyzed Wording of questions problematic

14. Interview Structure of interview Formal and structured Unstructured Analysis can be complicated

15. Experimental Controlled Setting Specific Task Conditions Assignment of Participants Useful for making inferences about causality

16. Case Study Examines a single or small number of cases Exploratory research

17. Ethnography Method adopted from Anthropology Describing Culture Used originally to describe other cultures Misunderstood method

18. Many other Methods Diaries Analytic Field Studies Quasi Experimental Longitudinal Studies Historical Studies

19. Internal and External Validity HIGH LOW HIGH Level of Internal validity Level of External Validity Laboratory Experiments Surveys Field Experiments Ethnographies

20. Conclusion to the Overview Different factors influence use and evaluation of CSCW software Framework of CSCW studies Evaluation Techniques Validity of Techniques

21. Short Break Reconvene in 3 minutes *Upcoming – Understanding Ethnography Using Ethnography Ethnography and CSCW

22. Ethnography Understanding Ethnography Sociology Adoption Using Ethnography CSCW Ethnography in Design (Hughes)

23. Understanding Ethnography Ethnography is loosely applied to qualitative research Home is originally from Anthropology Aim to describe cultural interpretation

24. Understanding Ethnography Understanding culture from an insiders point view Three sources of data Participant Observation Interviews Collection of representative artifacts

25. Sociology Adoption Originally used to study distant cultures Chicago School of Sociology Studies focused on exploration of groups in urban settings Cultural comparisons in USA Family of Ethnographic Techniques

26. Class Participation Time In Pairs (Saul and myself included) Everyone gets a Handout For a total of 5 Minutes (2.5 Minutes each) Each member of the pair (one at a time) asks the other questions from the sheet

27. Debrief of Class participation What answers were given…. Obviously not a long term ethnography study Depending on your relationship to this lab – differing perspectives Understanding Grouplab culture (to some extent)

28. Ethnography and CSCW Prominence of Ethnography in CSCW Insufficient attention to social context New problems for design of collaborative character of work and activities Ethnography and system design Problem of scale Pressure of time Role of the ethnographer

29. Concurrent Ethnography Design is influenced by on-going ethnographic study Sequenced process Debriefing Meetings System Prototype Ethnographic Study Systems Development

30. Concurrent Ethnography in action London Air Traffic Control Centre Four week Ethnography Each stage of fieldwork was intended to target designers issues Small research team What ethnography provided

31. Quick and Dirty Ethnography Brief Ethnographic Studies Duration relative to the size of the task Selecting aspects of work setting of importance to design Outline of project Meetings Debriefing Meetings Scoping Document Short Focus Studies

32. Quick and Dirty Ethnography in Action Ethnographic investigation of software engineers Challenges of Large scale setting Working in Industrialized Environments Acceptance into the setting (*Key to Ethnographic research)

33. Evaluative Ethnography Ethnography used to verify formulated design decisions Short Ethnographic study Initial outline Design or Specification Debriefing Meetings Amended Design Or Specification

34. Evaluative Ethnography in action Fieldwork in Building Society Using research for IT developments Routine of work Finding what customers wanted Outlined limitations of model that had been proposed

35. Re-examination of previous studies Previous studies are re-examined to inform design Ethnography used for many decades Many studies related to work and occupation Can be informative

36. Re-examination in action Inform preliminary design of Shared Object Service Using previous Ethnographic studies on: Social work, police work and invoice processing in a multi-site fast food company What common service should support

37. Summary of Ethnography Understanding Ethnography Ethnography and CSCW Uses of Ethnography Concurrent Quick and Dirty Evaluative Re-examination

38. Big Break Time 5 Minutes Reconvene in 5 Minutes Upcoming – Ethnomethodology Understanding Ethnomethodology Ethnomethodology in CSCW

39. Ethnomethodology Understanding Ethnomethodology Confusing Ethnography and Ethnomethodology Ethnomethodology in system design Incorporation of Sociology and Computer Science? Technomethodology

40. Understanding Ethnomethodology Ethnomethodology literally means Peoples Methods A Shift from other Sociological Methods Social Life is potentially Chaotic Social Actors Members methods for making sense

41. Understanding Ethnomethodology Garfinkel Documentary Method Example of Documentary Method Garfinkel Indexicality Disrupt Technique Example in class

42. Understanding Ethnomethodology We can observe other members methods of construction Development of Conversation Analysis

43. Confusing Ethnography and Ethnomethodology Ethnography is a form of investigative fieldwork Ethnography focuses on the Members Point of View Ethnomethodology is a specific analytical technique

44. Confusing Ethnography and Ethnomethodology Confusion arises because: Ethnomethodologist is likely to use ethnographic techniques Analytic mentality-selection of phenomena and topics for investigation

45. Ethnomethodology in HCI and CSCW Observations of work activities and interactions help design process Understanding temporal organization of activities and interactions and implications to design

46. Learning from Ethnomethodologists Division of Labour Field Observation conducted by ethnomethodologists Act as proxy for end users Hand off requirements to computer science people

47. Ethnomethodology for Critique and Design Ethnomethodology has provided: Critique of the design Failure to support the work Technology doesnt allow people to engage in their work Outlines organization of work and communication in the real world

48. Two Paradoxes Paradox of system design Large scale activity Paradox of technomethodology Transformational nature of technology Analysis of practice not invention

49. Technomethodology Develop a stance in which ethnomethodology and computer science play equally significant roles Foundational relationships

50. Technomethodology Trying to exploit generalizations from ethnomethodology Abstractions from both disciplines Means by which such working practices arise Dialogical interfaces

51. Conclusion Understanding Ethnomethodology Confusing Ethnography and Ethnomethodology Ethnomethodology and system design Technomethodology

52. Conclusion of Presentation Overview of Evaluation Techniques used in CSCW Ethnography Ethnomethodology Technomethodology

53. My thoughts CSCW has been unfortunately caught in a battle between sociological techniques CSCW design should incorporate work context issues

54. It was this clear….