"Evaluating Students' Evaluations of Teaching: Bias and Beyond" Excellence in Teaching and Learning Committee (ETLC) Workshop College of Liberal Arts & Social Sciences (CLASS) Chris Davoli (Psychology) & David Kinney (Sociology) November 2, 2018

Today: 1) Research regarding the sources and extent of biases in SET 2) Interdisciplinary discussion about how bias presents problems for SET at CMU 3) Possible solutions for reducing and/or eliminating biases in our SET Today:

Sources of Bias Gender Class size Race/ethnicity Students’ expected grade in course Required vs. Elective course Academic discipline Physical appearance Non-native English speaker Class size Intelligence/competency Sexual orientation Instructors perceived as “easy” Personality Entertainment Age

Gender: The most-researched characteristic as a source of bias – The larger context Women: 37.5 percent of all tenured faculty in American colleges & universities Women of color: 8 percent Full professors: Men outnumber women by more than two to one. Majority of assistant professors, instructors, and lecturers are women.  Gender gap in promotions: Sociology 85% of men and 78% of women. Computer Science: 86% of men and 81% of women. English departments: 86% of men and 80% of women (Weisshaar 2017).

Figure 2. Cumulative hazard plot of receiving tenure, by gender and field, over career time. “Years since starting” indicates the number of years spent as an assistant professor. From: Publish and Perish? An Assessment of Gender Gaps in Promotion to Tenure in Academia Soc Forces. 2017;96(2):529-560. doi:10.1093/sf/sox052 Soc Forces | © The Author 2017. Published by Oxford University Press on behalf of the University of North Carolina at Chapel Hill. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com. 5

GENDER PAY GAP Female full professors - 90% Associate professors – 93% Assistant professors – 91% Lecturers – 88% of what men make. Total full-time faculty: MALE full professors: 26% FEMALE full professors: 8.4%

Most recent and most rigorous study of gender bias Anne Boring (2017) 23,001 evaluations of 379 instructors by 4,423 students in 6 mandatory first-year courses at a French university French data: male students rank male instructors much higher than female instructors; but little difference among female students. Men viewed as being more knowledgeable even though students learn as much from men as from women.

Gender bias – boring and associates (2016) continued Four sections of an online course in a randomized, controlled, blind experiment at a U.S. university When one professor taught the exact same online course under two different names, students gave lower rankings to the professor with the female name.  Female students rated perceived male instructors higher than they rated perceived female instructors; but little difference by male students.

Mitchell & martin (2018): “gender bias in student evaluations” Content analysis of comments on official course evaluations & from Rate My Professors Students comment on female professors’ personality and appearance and are significantly more likely to call them “teachers” rather than “professors” indicating that students may have “less professional “respect” for their female professors (p. 652). Personality mentioned 4.3% of the time for male professors vs. 15.6% for female profs. Appearance: 0% for male professor and 10.6% for female professor. Entertainment: 15.2% for male professor and 32.2% for female professor. Quantitative SET findings: male received higher evaluations on all items in identical courses

KEY CONSEQUENCE OF GENDER BIAS IN S.E.T.S. IN AN EFFORT TO INCREASE THEIR S.E.T. SCORES, FEMALE PROFESSORS MAY SPEND MORE TIME AND ENERGY ON COURSE PREPARATION AND GIVING ATTENTION TO STUDENTS THIS IS AN “OPPORTUNITY” COST: LESS TIME TO CONDUCT RESEARCH & PUBLISH LESS TIME TO PROVIDE SERVICE TO THE DEPARTMENT, UNIVERSITY & COMMUNITY MAY LEAD TO LOWERED CAREER AMBITIONS AND EVEN LEAVING ACADEME

FINAL WORDS ON GENDER BIAS BIAS EXISTS IN BOTH FACE-TO-FACE & ONLINE CLASSES USE OF BIASED SETs in EMPLOYMENT DECISIONS (PROMOTIONS etc.) IS DISCRIMINATORY AGAINST WOMEN U.S. EQUAL PAY ACT OF 1963 PROHIBITS DETERMING WAGES ON THE BASIS OF SEX TITLE VII OF THE CIVIL RIGHTS ACT OF 1964 DECREES THAT DISCRIMINATING PROMOTIONAL PRACTICES ON THE BASIS OF SEX ARE FORBIDDEN

Bias due to students’ expected grade? Cho, Baek & Cho (2015): major university in South Korea Natural experiment: due to computer technical problems, some students saw final grades before completing teaching evaluations Students who know low grades in advance, rate teachers significantly lower than students who get low grades, but do not know in advance And students who see their low grades and they are lower than they expected, rate teachers even lower: “retaliatory behavior” (p.178) A discrepancy between grade received and grade expected affects SET indicating that “psychological and subjective” factors impact SET.

References Boring, Anne. 2017. “Gender Bias in Student Evaluations of Teaching.” Journal of Public Economics 145(January):27-41. Cho, D., W. Baek & J. Cho. 2015. “Why do good performing students highly rate their instructors? Evidence from a natural experiment.” Economics of Education Review 49:172-179. Mitchell, K.M.W. & J. Martin. 2018. “Gender Bias in Student Evaluations.” The Teacher: American Political Science Association. July:648-652. Weisshaar, Katherine. 2017. “Publish and Perish? An Assessment of Gender Gaps in Promotion to Tenure in Academia.” Social Forces 96(2):529–560.