1. Tailoring Course Evaluations/Student Feedback to Improve Teaching Jeffrey Lindstrom, Ph.D. Siena Heights University Webinar 6 October 2014

2. Introduction  Quality and effectiveness of teaching is a primary concern.  Siena’s Mission  Quality and effectiveness of teaching a primary concern.  Resources for faculty to assist in delivering quality, effective teaching...  Learning outcomes assessments,  Year-end faculty evaluations,  Student/Course evaluations (hereafter “course evaluations”). Siena Heights University Webinar 6 October 2014

3. Introduction  Course evaluations stand out...  Course evaluations are assessments provided by students.  Other measures originate with faculty or are product of faculty reflection, etc.  Represent one of the most common method to evaluate teaching (Stark & Freishat, 2014).  Course evaluations are a central part of faculty portfolio for promotion in rank (Greenwald, 1997; Stark & Freishat, 2014). Siena Heights University Webinar 6 October 2014

4. Introduction  Purpose of today’s presentation...  Briefly review the research literature on the validity of course evaluations and the multidimensionality of effective teaching.  Besides being a means of faculty evaluation, course evaluations can also be invaluable in improving the effectiveness of one’s teaching.  Suggest implications for course evaluations at Siena. Siena Heights University Webinar 6 October 2014

5. Introduction  Questions regarding course evaluations...  Questions whether course evaluations are valid measures (Greenwald, 1997; D’Apollonia & Abrami, 1997; McKeachie, 1997),  Questions regarding what is being assessed (i.e., “effective” teaching) (Marsh & Roche, 1997; Stark & Freishat, 2014; Young & Shaw, 1999). Siena Heights University Webinar 6 October 2014 There is strong evidence that student responses to questions of “effectiveness” do not measure teaching effectiveness (Stark & Freishat, 2014, p. 2). It is widely believed that [course evaluations] are primarily a popularity contest; that it is easy to “game” the ratings (Stark & Freishat, 2014, p. 2). What is teacher effectiveness? Important questions continue to persist: What is effective teaching? How may it be defined? How may it be measured? To date, educators and researchers have failed to reach agreement about clear-cut answers to these questions (Young & Shaw, 1999, p. 670). Too often teacher effectiveness is assessed by simply asking whether the teacher was effective (Stark & Freishat, 2014).

6. Introduction Validity of Course Evaluations  Are student evaluations of teachers/teaching effectiveness valid?  Important questions remain (e.g., Stark & Freishat, 2014) but a body of solid research suggests they are.  “Student ratings are the single most valid source of data on teaching effectiveness” (McKeachie, 1997, p. 1219).  “In general, student ratings tend to be statistically reliable, valid, and relatively free from bias... probably more so than any other data used for evaluation” (Cashin, 1995, p. 6).  “Reviews of research conducted since about 1980 indicate overwhelming evidence supporting the... validity of student ratings (Young & Shaw, 1999, p. 671). Siena Heights University Webinar 6 October 2014

7. Introduction Question of Effective Teaching  What is effective teaching and what is being assessed in course evaluations?  Many researchers agree that most definitions of teaching effectiveness are too narrow and simplistic (d’Apollonia & Abrami, 1997; Stark & Freishat, 2014; Young & Shaw, 1999).  “Researchers and practitioners agree that teaching is a complex activity consisting of multiple dimensions... and that [course evaluations] of teachers should reflect this multidimensionality (Marsh & Roche, 1997, p. 1187).  “Teacher effectiveness is a complex, multidimensional construct” (Shaw & Young, 1999, p. 672). Siena Heights University Webinar 6 October 2014

8. Introduction Understanding Effective Teaching  Young and Shaw (1999) attempted to empirically identify the principal traits of effective teaching.  Subsequent analyses narrowed list to seven items/factors... 1. Motivating students to do their best, 2. Course organization, 3. Effective communication, 4. Value of the course, 5. Comfortable learning environment, 6. Concern for student learning, and 7. Genuine respect for the students. Siena Heights University Webinar 6 October 2014

9. Introduction Siena’s Course Evaluation  Siena’s standard course evaluation form (.pdf)..pdf  1 page/10 items in length.  Desire to have a common instrument/standard for comparison resulted in a somewhat general evaluation.  Open to modification but how many faculty do and to what degree?  When was the form created? Last revised?  What model of effective teaching served as the basis for the questions and format? Siena Heights University Webinar 6 October 2014

10. Introduction JPL Evaluation Form (PSY 440 – WI 2014)  JPL customized evaluation form (Word).Word  10 pages/85 items in length.  An effort is made to assess all elements of the course.  The seven criteria of Young and Shaw’s (1999) model of effective teaching are all assessed, either as a section or as survey item(s).  Utilizes a 7-point Likert scale format.  Students’ anticipated grades are not assessed.  Course evaluation is administered during final exam period.  Includes discussion/dialogue about the course. Siena Heights University Webinar 6 October 2014

11. Introduction JPL Evaluation Form (PSY 440 – WI 2014)  Analysis (.pdf).pdf  Which items are statistically significant?  Examine the means.  Changes to course format/teaching...  Amount of lecture in course (less).  Attendance policy (not used).  Number of assignments (more smaller early, etc.).  Use of eCollege and PowerPoint handouts (more).  Original readings (keep). Siena Heights University Webinar 6 October 2014

12. Introduction Implications for Course Evaluations  Course evaluations can be an invaluable resource to assess and improve one’s teaching effectiveness.  Consider revising Siena’s course evaluation around model like Young and Shaw (1999).  Allow ease/flexibility for divisions/departments/teachers to modify to assess unique aspects of teaching.  Utilize multiple assessments to assess teaching effectiveness for promotion...  Student/Course evaluations,  Stark and Freishat (2014, p. 3)  Frequencies not averages; peer observations/evaluations, etc.  Promote culture supporting effective and innovative teaching. Siena Heights University Webinar 6 October 2014

13. Questions and Suggestions Thank you for your participation! Happy teaching. Siena Heights University Webinar 6 October 2014

14. References  Cashin, W. E. (1995). Student ratings of teaching: The research revisited (IDEA Paper No. 32). Manhattan, KS: Kansas State University, Center for Faculty Evaluation and Development.  d’Appolnia, S., & Abrami, P. C. (1997). Navigating student ratings of instruction. American Psychologist, 52, 1198 – 1208.  Greenwald, A. G. (1997). Validity concerns and usefulness of student ratings of instruction. American Psychologist, 52, 1182 – 1186.  Greenwald, A. G., & Gillmore, G. M. (1997). Grading leniency is a removable contaminant of student ratings. American Psychologist, 52, 1209 – 1217.  Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52, 1187 – 1197.  McKeachie, W. J. (1997). Student ratings: The validity of use. American Psychologist, 52, 1218 – 1225.  Stark, P. B., & Freishat, R. (2014). An evaluation of course evaluations. Retrieved from http://www.stat.berkeley.edu/~stark/Preprints/evaluations14.pdf. http://www.stat.berkeley.edu/~stark/Preprints/evaluations14.pdf  Young, S., & Shaw, D. G. (1999). Profiles of effective college and university teachers. Journal of Higher Education, 70(6), 670 – 686. Retrieved from http://www.jstor.org/stable/2649170.http://www.jstor.org/stable/2649170 Siena Heights University Webinar 6 October 2014