1 Problems, and little victories, in transcript analysis Patrick J. Fahy Athabasca University, Centre for Distance Education CIDER Presentation 30 November 2007

2 Outline 1. Why TA? 2. Validity issues 3. Ethics principles 4. Implications for practice

3 Why transcript analysis? A conference transcript is a gold mine of information concerning the psycho-social dynamics among the participants (Henri, 1992, p. 118). A conference transcript is a gold mine of information concerning the psycho-social dynamics among the participants (Henri, 1992, p. 118). CMC is related both to achievement and attitudes (Fulford & Zhang, 1993). CMC is related both to achievement and attitudes (Fulford & Zhang, 1993). CMC is a way to establish instructor and social presence – facilitators of online community (Rourke, Anderson, Garrison, & Archer, 1999). CMC is a way to establish instructor and social presence – facilitators of online community (Rourke, Anderson, Garrison, & Archer, 1999). –Community is important to the success of online learners (Conrad, 2005).

4 Why TA? (cont) CMC is common, but how to manage CMC, and how CMC-based interaction improves learning and online experience, is not well understood. CMC is common, but how to manage CMC, and how CMC-based interaction improves learning and online experience, is not well understood. –Community constructed - somehow - through language, in CMC (Vrasidas & McIsaac,1999). Further study is needed to describe the discourse devices used in CMC to establish community (Lapadat, 2007, p. 62) Further study is needed to describe the discourse devices used in CMC to establish community (Lapadat, 2007, p. 62)

5 Validity - issues Lack of discriminant capability –Many postings in few categories (Gunawardena, Lowe, & Anderson, 1997; Kanuka & Anderson, 1998). –Postings could be coded into more than one category; category ambiguity (Zhu, 1996).

6 Validity – more issues Instruments too complex, too many codes –Gunawardena et al. (1997): 20 categories (5 phases) –Cookson & Chang, 1995: 16 codes –Rourke et al., 1999: 12 indicators –Higgins (1998): 20 –Zhu (1996): 8

7 Unreliability –inter-coder reliability is low Sometimes, reliability data not reported Sometimes, reliability data not reported

8 Questionable solutions –Collaborative (group) coding (engineered coder drift). –Mystical terminology units of meaning, discourse units, message units, etc. units of meaning, discourse units, message units, etc. –Coding of whole messages into 1 category One code fits all One code fits all

9 Considerations regarding validity 1. Principle of over-determination 2. Principle of commensurate complexity 3. Dangers of Type 1 errors

10 Principle of over-determination Social events usually have more causes than are strictly required for them to occur. Contrary to Occams Razor, simplest explanations may not be truest Contrary to Occams Razor, simplest explanations may not be truest –Seek simplicity, then distrust it. (Whitehead)... human development is the result of many overlapping spheres of influence.... (Gibson, 2000)... human development is the result of many overlapping spheres of influence.... (Gibson, 2000) Unlike rare diseases such as Huntingtons, where a single genetic variation guarantees that a carrier will be afflicted, common diseases are triggered by a complex array of factors, including multiple genes each exerting a modest effect. [Genes for several common diseases, (2007, September/October). Unlike rare diseases such as Huntingtons, where a single genetic variation guarantees that a carrier will be afflicted, common diseases are triggered by a complex array of factors, including multiple genes each exerting a modest effect. [Genes for several common diseases, (2007, September/October).

11 Principle of commensurate complexity –Social principles cannot usually be at once general, accurate, and simple. [Thorngate, W. (1976).]

12 A special problem Type 1 error: rejection of a null hypothesis that is, in fact, true.rejection of a null hypothesis that is, in fact, true.

13 Ethics In Canada, the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS, 2003), applies to all research, funded or not. In Canada, the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS, 2003), applies to all research, funded or not. –Developed and jointly issued by the Medical Research Council of Canada (MRC), the Natural Sciences and Engineering Research Council of Canada (NSERCC), and the Social Sciences and Humanities Research Council of Canada (SSHRCC).

14 Problems Research ethics boards (REBs) and researchers disagree on what is ethical. Research ethics boards (REBs) and researchers disagree on what is ethical.... an underlying current of distrust... exists within the [Humanities] community vis-à-vis the TCSP (Owen, Robert, Burgess, Golfman & Sykes, 2001, p. 4).... an underlying current of distrust... exists within the [Humanities] community vis-à-vis the TCSP (Owen, Robert, Burgess, Golfman & Sykes, 2001, p. 4). –Review boards pose threat to social scientists work. (The Wall Street Journal, 2004)

15 Where is the real problem? Ethics issues in clinical/biomedical research Ethics issues in clinical/biomedical research –Tuskegee syphilis study (1932 – 1972) –Polio vaccine studies on children (1950s) –Willowbrook hepatitis studies (1963 – 1966) –Drug trials (Olivieri vs. Apotex, 1998) –Others?

16 Key ethical principles: Getting minimal risk research approved Ethics: from ethos = custom. Ethics: from ethos = custom. –TCPS: REBs to apply ethical principles in the context of the nature of the research and of the ethical norms and practices of the relevant research discipline (TCPS, p. i.9). Emphasize maximization of benefits, minimization of risks, potential harms. Emphasize maximization of benefits, minimization of risks, potential harms.

17 Respect for human dignity Persons are ends, not means. Persons are ends, not means. –Individual rights normally take precedence over the advancement of knowledge Informed consent assures respect for individual rights... but collective rights are recognized, too. Informed consent assures respect for individual rights... but collective rights are recognized, too.

18 Justice, inclusiveness No group should bear an unfair burden in, or enjoy disproportionately the benefits of, research. No group should bear an unfair burden in, or enjoy disproportionately the benefits of, research.

19 Understand informed consent (The Gold standard ) Conditions under which informed consent may be waived (TCPS, p. 2.1) : 1.The research is minimal risk. 2.The waiver is unlikely to adversely affect the rights and welfare of the subjects. 3.The research could not practicably be carried out without it. 4.Subjects will be provided with additional information after participation, whenever possible and appropriate. 5.A therapeutic intervention is not involved [normally precluded by point 1].

20 Rationale Majorities have rights: Majorities have rights: –Individuals may refuse to participate, and subjects who express concerns about a study may have the option of removing their data from the project; however: This approach should be used only when the elimination of the subjects data will not compromise the validity of the research design, and hence diminish the ethical value of participation by other subjects. (TCPS, p. 2.3; emphasis added) This approach should be used only when the elimination of the subjects data will not compromise the validity of the research design, and hence diminish the ethical value of participation by other subjects. (TCPS, p. 2.3; emphasis added)

21 So long as the methodology is sound, all contributions are anonymous, and identities are protected in any reports, it may well be a greater ethical violation, in relation to participation, to deny the rights of the willing majority than it would be to accede to the wishes of the reluctant minority by refusing to approve a research proposal. So long as the methodology is sound, all contributions are anonymous, and identities are protected in any reports, it may well be a greater ethical violation, in relation to participation, to deny the rights of the willing majority than it would be to accede to the wishes of the reluctant minority by refusing to approve a research proposal. –Nothing in this Policy should be interpreted to mean that research subjects have the right to veto a project, though they do, of course, have the right to refuse to cooperate with the researcher(s) (TCPS, p. 1.2; emphasis added).

22 Other limitations to informed consent Preserve the integrity of the research, by avoiding colour[ing] the responses of the subjects and thus invalidat[ing] the research (TCPS, pp. 2.2 – 2.3). Preserve the integrity of the research, by avoiding colour[ing] the responses of the subjects and thus invalidat[ing] the research (TCPS, pp. 2.2 – 2.3).

23 Individuals and groups not entitled to informed consent (Remember: this is the gold standard in the ethical treatment of human subjects) Public bodies, corporations, political parties Public bodies, corporations, political parties Private organizations Private organizations –... may not interfere with or veto research about their own conduct, their participation in any research about their behavior is not required, and they need not be approached for their consent prior to research being conducted. (TCPS, p. 2-2) Public figures Public figures –(Certain types of research … may legitimately have a negative effect on public figures …. (TCPS, p. 1-6) Naturalistic observations Naturalistic observations Classroom, course evaluations Classroom, course evaluations Quality assurance, performance reviews Quality assurance, performance reviews

24 Secondary use of data Secondary use of data refers to the use in research of data contained in records collected for a purpose other than the research itself. Common examples are patient or school records or biological specimens, originally produced for therapeutic or educational purposes, but now proposed for use in research. This issue becomes of concern only when data can be linked to individuals, and becomes critical when the possibility exists that individuals can be identified in the published reports. (TCPS, p. 3.4; emphasis added) Secondary use of data refers to the use in research of data contained in records collected for a purpose other than the research itself. Common examples are patient or school records or biological specimens, originally produced for therapeutic or educational purposes, but now proposed for use in research. This issue becomes of concern only when data can be linked to individuals, and becomes critical when the possibility exists that individuals can be identified in the published reports. (TCPS, p. 3.4; emphasis added)

25 The real gold standard of ethical research ANONYMITY.

26 Suggestions for practice (praxis) 1. Notice who speaks, who doesnt. Invite the reluctant.Invite the reluctant. Respect the desire for anonymity.Respect the desire for anonymity. 2. Notice hubs, spokes. Assure hubs are positive; check on the isolated (people no one is talking or responding to).Assure hubs are positive; check on the isolated (people no one is talking or responding to). 3. Talk to everyone at least once But dont insist others do this.But dont insist others do this. 4. Link, document. Be the literature/social link.Be the literature/social link.

27 5. Be present. 80% of getting ahead is just showing up. (Woody Allen) 80% of getting ahead is just showing up. (Woody Allen) 6. Give credit for participation. What gets measured gets done. What gets measured gets done. 7. Set a standard. What gets modeled gets done. What gets modeled gets done.

28 Citations Cookson, Peter S. & Chang, Yu-bi. (1995). The Multidimensional Audioconferencing Classification System (MACS). American Journal of Distance Education, 9(3), Cookson, Peter S. & Chang, Yu-bi. (1995). The Multidimensional Audioconferencing Classification System (MACS). American Journal of Distance Education, 9(3), Fulford, C. P. & Zhang, S. (1993). Perception of interaction: The critical predictor in distance education. The American Journal of Distance Education, 7(3), pp. 8 – 21. Fulford, C. P. & Zhang, S. (1993). Perception of interaction: The critical predictor in distance education. The American Journal of Distance Education, 7(3), pp. 8 – 21. Genes for several common diseases. (2007, September/October). Technology review, 110(5), p Genes for several common diseases. (2007, September/October). Technology review, 110(5), p Gunawardena, C., Lowe, C. & Anderson, T. (1997). Analysis of a global on-line debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing. Journal of Educational Computing Research, 17(4), pp Gunawardena, C., Lowe, C. & Anderson, T. (1997). Analysis of a global on-line debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing. Journal of Educational Computing Research, 17(4), pp Henri, F. (1992). Computer conferencing and content analysis. In A. Kaye (Ed.), Collaborative learning through computer conferencing: The Najaden papers (pp ). Berlin: Springer-Verlag. Henri, F. (1992). Computer conferencing and content analysis. In A. Kaye (Ed.), Collaborative learning through computer conferencing: The Najaden papers (pp ). Berlin: Springer-Verlag. Medical Research Council of Canada. (2005). Tri-council policy statement: Ethical conduct for research involving humans. Ottawa: Government of Canada. Available: Medical Research Council of Canada. (2005). Tri-council policy statement: Ethical conduct for research involving humans. Ottawa: Government of Canada. Available:

29 Oshinsky, D. M. (2005). Polio: An American story. Toronto: Oxford University Press. Oshinsky, D. M. (2005). Polio: An American story. Toronto: Oxford University Press. Owen, M., Robert, L., Burgess, J., Golfman, N. & Sykes, S. (2001). Report to the Social Sciences and Humanities Research Council of Canada, Implementation of the Tri-Council Policy Statement on Ethics in Human Research (TCPS) (Humanities Project). Unpublished paper, Humanities and Social Sciences Federation of Canada. Owen, M., Robert, L., Burgess, J., Golfman, N. & Sykes, S. (2001). Report to the Social Sciences and Humanities Research Council of Canada, Implementation of the Tri-Council Policy Statement on Ethics in Human Research (TCPS) (Humanities Project). Unpublished paper, Humanities and Social Sciences Federation of Canada. Rourke, L., Anderson, T., Garrison, R. & Archer, W. (1999). Assessing social presence in asynchronous text-based computer conferencing. Journal of Distance Education, 14(2), pp Rourke, L., Anderson, T., Garrison, R. & Archer, W. (1999). Assessing social presence in asynchronous text-based computer conferencing. Journal of Distance Education, 14(2), pp Thorngate, W. (1976). In general vs.. it depends: Some comments on the Gergen-Schlenker debate. Personality and Social Psychology Bulletin 2, pp. 404 – 410. Thorngate, W. (1976). In general vs.. it depends: Some comments on the Gergen-Schlenker debate. Personality and Social Psychology Bulletin 2, pp. 404 – 410. Vrasidas, C. & McIsaac, M. S. (1999). Factors influencing interaction in an online course. American Journal of Distance Education, 13(3), pp. 22 – 36. Vrasidas, C. & McIsaac, M. S. (1999). Factors influencing interaction in an online course. American Journal of Distance Education, 13(3), pp. 22 – 36. Zhu, E. (1996). Meaning negotiation, knowledge construction, and mentoring in a distance learning course. In Proceedings of Selected Research and Development Presentations at the 1996 National Convention of the Association for Educational Communications and Technology (18th, Indianapolis, IN). Available from ERIC documents: ED Zhu, E. (1996). Meaning negotiation, knowledge construction, and mentoring in a distance learning course. In Proceedings of Selected Research and Development Presentations at the 1996 National Convention of the Association for Educational Communications and Technology (18th, Indianapolis, IN). Available from ERIC documents: ED

30 Pat Fahy Athabasca University, Centre for Distance Education