1. IS 4800 Empirical Research Methods for Information Science Class Notes March 21 and 23 Instructor: Prof. Carole Hafner, 446 WVH hafner@ccs.neu.edu Tel: 617-373-5116 Course Web site: www.ccs.neu.edu/course/is4800sp12/

2. 2 Qualitative Research : General characteristics Purpose: to study behavior patterns not well understood and/or too complex for quantitative methods Research questions can be non-specific (“panoramic view”) Research hypotheses and procedures can evolve Collection of data Only partially structured (use of open-ended observations and/or questions) Who: individuals, groups, cultures Where: natural setting

3. 3 Analysis is interpretive: importance of experimenter’s background and possible bias coding of data defined or enhanced during analysis stage Reporting of results in the form of an essay or narrative use of quotes to illustrate and back up claims use of little “stories” of what happened for the same reason

4. 4 Qualitative Research: Specific qualitative methods Ethnography – observe communities/culture Case study – observe groups/goal orientation Phenomenology – observe individuals Grounded theory – observe groups or cultures Relevance to IT field

5. 5 StrategyExample(s) or categories of IT-related research Ethnography Ethnography – Observe behavior patterns (introduce email or other apps) Ex: instant messaging among HS students. Technique: field observations/ artifacts, interviews to get background and motivations. Ex: room scheduling Case study Case study – usually IS development/deployment. Telemedicine study: introduction of IT, how it changed the patterns of service provision. Technique: interviews (facts, motivations, explanations), artifacts. Phenomenol ogy How does a new technology change employee’s work experience: study secretaries, middle managers, executives. Example: email, problem reporting, cust support. Specifics: your tasks, required skills, communication (with whom, how much, method) your perception of your job (autonomy, influence, collegiality). Technique: interviews (attitudes) Grounded theory Grounded theory: Create a model that categorizes college students in terms of studying/learning behavior. The procrastinator. The super-prepared. Etc. and use it to explain observed impact of distance learning Ex: stages of technology adoption. Technique: interviews, artifacts

6. 6 Qualitative Research (cont.) Data Collection A plan for data collection – the “heart” of the research design Who, where, what, when Techniques for data collection: field observations/participant observation interviews artifacts (documents, photos, “made objects”, etc.) Recording methods taking notes recording/taping collecting artifacts

7. 7 Qualitative ResearchData Collection (cont.) Observational protocol: Thinking out in advance what questions you want to answer through your field observations, interviews, or examination of artifacts. A “structure” for taking notes A “template” with various “slots” to fill in. (As in Homework assignment 1). For an interview: opening statement, slots for specific (demo-graphic) information, questions to ask, guidelines for follow-up

8. 8 Qualitative Research (cont.) Data Analysis Step 1. Prepare and organize collected data (notes, tapes, artifacts). Organization may be chronological, categorical, by source. Etc. Step 2. Read with open mind – take notes. Step 3. Classify into categories/taxonomies. Identify important attributes/dimensions of variation. Use a coding scheme to position elements of the data collected (answers in an interview, events observed, artifacts) within your classification scheme.

9. 9 Qualitative Research Data Analysis (cont.) Step 4. Identify larger themes and show how the categories of your classification scheme are related. Show how these themes provide explanations for the behavior observed. Step 5. Summarize. Design figures, tables or other means for presenting your analytical scheme in a summary fashion. Step 6. Conclusions. What will be the “lessons learned” or conclusions of the study.

10. 10 Validity in Qualitative Research Some approaches to increasing credibility Use data from different sources (“triangulate”) to show your analysis provides a coherent explanation Show analysis to participants to confirm accuracy Show results to other (impartial) researchers Present negative or conflicting information honestly Describe researcher’s background and possible bias

11. 11 Qualitative Research (cont.) Writing the Proposal or Report Be clear about the study’s purpose, research questions, research strategy, and limitations. Be specific about who, what, when, where and how the data will be/were collected and analyzed. Be scholarly by reviewing prior work and explaining how the most relevant prior work is related to the current study. Be honest about problems encountered, researcher’s role, conflicting data. Present complex material verbally and graphically. Follow good technical writing style.

12. Qualitative Research Data collection methods I. Observation 6 months – 1 year II. Interviews Informal III. Analysis of artifacts (documents, arts and crafts, tools)

13. 13 I. Observation: Developing Behavioral Categories, i.e. “Coding” “confused” vs. (“clicked mouse at least 5 times on inappropriate menu” OR “gazed at interface with mouth open AND no mouse clicks or keyboard presses for 5 minutes”) AND “furrowed brows”

14. 14 Developing Behavioral Categories How do you know when your definitions are good enough? Cohen’s Kappa measure of inter-rater agreement: PR(a) – PR(e) / (1 – PR(e)) a = agreement observed e = agreement expected by chance

15. 15 Refining a behavioral protocol Draft Coding manual Two or more observers code data for pilot subjects Is Kappa acceptable? Coders review differences & update coding manual Coding manual provides reliable measure, proceed with study yes no

16. 16 Coding Manual You should write your behavior identification rules down so that you could give them to someone else to follow reliably. You should also write down the sampling and coding methods you will use, as well as your recording instrument (e.g., paper form).

17. 17 Quantifying Behavior in Observational Research Frequency Method –Record the frequency with which a behavior occurs within a time period Duration Method –Record how long a behavior lasts Intervals Method –Divide the observation period into several discrete time intervals (e.g., ten 2-minute intervals), and record whether a behavior occurs within each interval

18. 18 Example: Code Posture Shifts

19. 19 Coping With Complexity in Observational Research Event Sampling –Select one behavior for observation and record all instances of that behavior –It is best if one behavior can be specified as more important than others

20. Interviews –Structured, semi-structured, informal, retrospective –Informal interviews seem to be casual conversations, reveal what people think and how one person’s perceptions compare with another’s –Difficult to conduct ethically and productively Key Actor or Informant interviewing Specific questions: open ended or closed ended

21. Techniques Grand tour techniques – show me around Study of faculty attitudes: tell me about the College Good at exploratory stage, to identify topics for further study

22. The Ethnographic Interview (paper) I. Grand Tour questions A. General Overview Ask the informant to generalize, to discuss patterns of events. Could you describe a typical day on the job? Could you show me/tell me how you usually make a box? B. Specific Tour Ask the informant about a specific incident or what he or she did on a certain day. Could you describe what happened at the management meeting yesterday, from beginning to end? Tell me about the last time you used the crane.

23. C. Guided Tour Ask the informant for a tour of the workplace or to accompany him or her while doing a job. Could you show me around the plant? Could I go on a sales call with you? D. Task-Related Grand Tour Ask the informant to perform a task to help you understand the context. Could you draw a flow chart of how the aluminum moves through the plant, from raw metal to the finished product? Could I watch you use the cutting machine and ask you questions about it afterwards?

24. II. Mini-Tour Questions "Describe what goes on when you run the coil through the annealing machine." III. Example Questions "Can you give me an example of your supervisor giving you a hard time?"

25. IV. Experience Questions "Could you tell me about some experiences you've had working on the annealing machine?" V. Native-Language Questions What do you call it when you mis- measure a piece? How do you refer to your work area?

26. 26 III. Artifacts: Content Analysis Used to analyze a written or spoken record for occurrence of specific behaviors or events Archival sources often used as sources for data Response categories must be clearly defined A method for quantifying behavior must be defined

27. 27 Example Study The CEO of Global Enterprises, Inc. is very worried about the low morale in the company, as evidenced by the amount of flame email she receives. She considers sending every office on a “ropes” course, but to do this would cost the company $10M. She asks you to do a study to tell how well her scheme might actually work in reducing her flame mail.

28. 28 Analytic Induction (Znaniecki) Nonexperimental, Qualitative analogue to scientific method 1.Phenomenon tentatively defined 2.Hypothesis is developed 3.A single instance is considered to determine if hypothesis is confirmed 4.If hypothesis fails, then phenomenon or hypothesis is redefined 5.Additional cases are examined and, if the new hypothesis is repeatedly confirmed, some degree of certainty results 6.Each negative case requires that the hypothesis be reformulated until there are no exceptions

29. 29 Typical Use of Analytic Induction Say you’re interested in employee’s impressions of WizziWord. You interview 3 people, transcribe your notes, and categorize all important statements into themes –e.g. “It’s too slow.”, “It looks cool.”, etc. You interview 3 more people, categorize their comments. Repeat until no new (significant) categories/themes emerge.

30. 30 Exercise What are your impressions of Clippy?

31. 31 Meta-Analyses Compare/Integrate “all” studies that have investigated a given phenomena –E.g., use of a particular medication for a particular disease Common in the literature (esp. medical) Very methodical –Search for articles –Eligibility criteria –Statistical analyses

32. Lecture 1 - Introduction32 Ethnography Ethnography: The art and science of describing a group or culture. The observer is a human instrument Select a place and the people or activity Observe behavior and identify patterns A great variety of techniques to gather data

33. “(Participant) observation”: in natural setting “Participant” observation occurs when you interact casually and/or form relationships with informants How much you actually “participate” depends on the goals of the study.

34. Participant observation Advantages: –Offers insights into complex behavior – Identify the “right questions” for further study –Verify/correct self-reports Disadvantage: –Time consuming –Data collection is difficult –Problem of subjectivity

35. How to operationalize Field notes –Text –Diagrams, maps –Can result in numerical data Interviews (interviewer more clueful) Focus groups (facilitator more clueful)

36. What to observe Spatial relations Activities Communication –Verbal –Other Tasks –How work is allocated

37. Ethics Do not disrupt the activity your are observing versus Do not mislead No formal rules about disclosing your role as a researcher when engaging in casual conversation – article suggests a point where you want to ask specific question Disclosure includes: right of refusal, confidentiality

38. Protecting confidentiality when data is unique Separate identify info from field notes entered into the computer People, organizations/companies, should be given fictitious names

39. How to be an effective observer Preparation Stay in the background Be factual and objective in your notes –(interpretation comes later) Taking notes: –Hand written usually –Type in to computer later EXPANDING NOTES

40. 40 Case Study Research Steps: design, conduct (data collection), analysis, write-up Exploratory research and the role of prior theory Impacts case selection, data collection Scientific method – observations should have the potential to disconfirm Case study research questions Example: Telemedicine paper

41. Case study research Design steps Define unit of analysis Case selection Creation of a protocol (plan of work) Data Collection Analysis and reporting

42. 42 Case Study Research I. Design steps: Select the unit(s) of analysis (temporal, organizational, technological) Case selection (“sampling”??) – one or several critical case theory based (confirming or disconfirming) extreme v. typical intense criterion (e.g., budget > $X) convenience

43. 43 Case Study Research – Design steps (cont.) Use of a protocol: (the “plan of work”, should be required) 1.Overview of study, including overview of data collection strategy. 2.Details of data collection (sources, procedures) 3.Interview guidelines and instruments 4.Outline of the expected project report Issues to be addressed in (2): access to the organization resources sufficient to collect the data in the field scheduling of data collection activities providing for unanticipated events

44. 44 Case Study Research – Data Collection The more different methods employed, the fuller the picture of the phenomena being studied. -- Documents (meeting minutes, project reports, newsletters, manuals) -- Archival documents (service records, system usage data) may provide quantitative information -- Interviews Typical: 95 interviews over 6 months -- Field observations (when a visit is conducted): usually meetings. (also can observe user training, etc.) -- Artifacts (problem reports – why not archival docs??)

45. 45 Case Study Research – Data Collection Selecting interviewees: a. maximum variation (preferred method) b. Homogeneous c. Snowball or chain d. Purposeful v. opportunistic Benefits of semi-structured interviews Unstructured when questions not known in advance What is “triangulation”? (paper mentions construct validity) When to STOP collecting data

46. 46 Case Study Research – Data Analysis Data analysis very different from analysis of experiment and survey data. Why? Stages of analysis: Preliminary analysis (early steps) Within-case analysis Cross-case analysis Qualitative analysis most difficult and least standardized part of empirical research.

47. 47 Case Study Research – Data Analysis Goals of qualitative data analysis: Identify themes Develop categories Explore similarities and differences Describe patterns that explain why (Propose models that predict)

48. 48 Case Study Research – Data Analysis Techniques for qualitative data analysis: Preliminary Stage 1. Coding 2. Database What is a code? a word or short phrase attached to each segment (e.g., paragraph, answer to interview question) of the collected data, indicating the “presence” of that concept. Codes can be arranged in (or derived from) a taxonomy. A good taxonomy will yield codes that reveal patterns in the data.

49. 49 Case Study Research – Data Analysis Where do codes come from: Prior work or theory Study of initial data (defined “inductively”) Iterative nature of coding Use of independent raters to validate codes The code book or code manual: desirable attributes Detailed description of each code Inclusion and exclusion criteria Examples of collected data to illustrate each code Development of higher-level “pattern codes” identify themes or relationships that are relevant to the study’s research questions

50. 50 Case Study Research – Data Analysis Case study database: uninterpreted data complete data (answers criticism of “selective quoting”) analogous to raw data collected in experiment or survey Contents of case study database Field notes (interviews, observations) Documents (including transcripts) Quantitative data (including questionnaire data if any) Contemporaneous notes (reflective remarks)

51. 51 Case Study Research – Data Analysis Techniques for qualitative data analysis: within-case stage Goal: identify larger themes, relationships and propositions Looking for larger themes and patterns: Pattern-matching compare expected elements with actual data perform cross-checking of interview transcripts and other data collected desire two or more sources for each proposition Explanation building: challenge  tactics  results approach Use of charts, table, graphs, timelines to aid understanding

52. 52 Case Study Research – Data Analysis Techniques for qualitative data analysis: cross-case stage Depends on availability of several cases Two approaches: Analyze similarities and differences among cases (e.g., factors, behaviors, results) If goal is theory-building, develop a theory using one case and systematically compare its propositions to other cases

53. 53 Case Study Research – Write-up Weakest part of the article Goals: Includes the goals of all professional writing, e.g., clarity, shows relationship to earlier work, data support conclusions For positivist case research, shows applicability (general relevance) to other examples with similar circumstances Constructive – propositions translate into “lessons learned” that offer guidance on how to make use of the results (do’s and don’ts)