Comparative effectiveness of research ethics teaching methods Michael Kalichman and Dena Plemmons UC San Diego Research on Research Integrity Annual Meeting Niagara Falls, NY May 16, 2009

Goal Assess effectiveness of teaching research ethics Challenges  Different teaching objectives  Different institutions and audiences  Different instructors Assess relative effectiveness of different methods for teaching research ethics

Approach: The Course The course: Scientific Ethics (UC San Diego)  10 sessions, 1-1/2 hrs per week  3 sections each week  2 instructors (DP, MK) Teaching Methods:  Week 1: Lecture  Weeks 2-10 Lecture + Small group case discussion Lecture + Role play Case-based Lecture

Approach: The Students Students  Graduate Students, some postdocs  Biology, Neurosciences, Other (e.g., Bioengineering)  UC San Diego, Other (Salk, The Scripps Research Institute) Number  Total = 57  18 – 20 assigned per section

Approach: Randomization Week Section 1Section 2Section 3 1 MK Introductory Lecture 2 DP LCR 3 MK CRL 4 DP RLC 5 MK LCR 6 DP CRL 7 MK RLC 8 DP LCR 9 MK CRL 10 DP RLC L =Lecture + small group case discussion R =Lecture + role-play C =Case-based lecture

Approach: Outcomes 1.Pre-Test: 18 multiple choice questions (2 for each of 9 topics) 2.Weekly Quiz: 5 multiple choice questions 3 attitude questions about the topic 2 attitude questions about the survey (enjoyable? useful?) 3.Post-Test: Same as Pre-Test 4.Final Evaluation: Satisfaction, Perspectives

Overall Impact of Course: Perceptions How if at all have your awareness, knowledge, skills, or attitudes been changed by participating in this course?  “not at all”  “not much”

Overall Impact of Course: Perceptions How if at all have your awareness, knowledge, skills, or attitudes been changed by participating in this course?  “great increase in knowledge”  “definitely more aware and now know options available”  “the course stressed topics that are easy to ignore, …useful for recognizing similar situations before it's too late”

Overall Impact of Course: Perceptions How if at all have your awareness, knowledge, skills, or attitudes been changed by participating in this course?  “…I will be able to more effectively deal with many types of situations”  “learned new strategies for dealing with complex ethical issues”  “I am now better prepared to deal with problems like the ones discussed in class”

Overall Impact of Course: Perceptions How if at all have your awareness, knowledge, skills, or attitudes been changed by participating in this course?  “It has helped me to see other perspectives and change my mind on several topics. ”  “it increase[d] my moral outrage”

Sample Question #1 Research data are the property of: A.The institution which employs those who are collecting the data. B.The person(s) who collect the data. C.The head of the research group that collects the data.

Sample Question #2 The primary basis for requirements for review of human subjects research is in regulations created by the A.State government B.Federal government C.University

Overall Impact of Course: Knowledge Knowledge improved (P<0.001) However, improvement was modest  Pre-test median = 12.0  Post-test median = 13.0

Effects of Methods Knowledge: No statistically significant difference among methods for any of the 9 test weeks. Attitudes: No statistically significant difference among methods for any of the 9 test weeks.

Perceptions of Methods Useful or Enjoyable?  No statistically significant difference among methods for 5 of the 9 test weeks.  In 4 of the 9 test weeks, Lecture+Roleplay judged to be more enjoyable than Case-Based Lecture and/or Lecture+Case Discussion  In 3 of the 9 test weeks, Lecture+Roleplay was judged to be more effective than Case-Based Lecture and/or Lecture+Case Discussion However, when different methods compared across all weeks (2-factor ANOVA), no effect of method.

Student Preference - Methods Which of the methods did you find to be most useful for meeting the goals of this course?

Student Preference - Methods Which of the methods did you find to be most enjoyable for meeting the goals of this course?

Student Preference - Methods In future courses, would you recommend using:

Student Preference - Comments “I really disliked the role-playing. I didn’t think it was beneficial at all.” 13 of 51 respondents (>25%) specifically commented on not liking the role-play exercises

Summary and Conclusions Summary Student perceptions: positive impact on knowledge, skills, and attitudes Knowledge improved, but is it worth the cost? No difference in methods for knowledge or attitudes Lecture+roleplay considered more enjoyable and/or useful during several weeks of course, but least liked overall at end of course Mixed methods preferred Conclusions 1.Impact of course on attitudes needs to be assessed 2.Teachers may be more important than methods