ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli

The Assessment Grid Consumer StudentInstructor University Beyond the university Target Program Instructional material Course Instructor Student

The Assessment Grid Consumer StudentInstructor University Beyond the university Target Programchoicex quality control accredit Instructional material ximprovexadopt? Coursechoiceimprove quality control x Instructorchoiceimprove promotion/ merit x Studentlearning improve/gra ding x employment /grad school

The Assessment Grid Implemented by StudentInstructor University Target Programxx Academic analysis Instructional materialx Targeted assessment x Course/InstructorNot at UWSurvey?? StudentSALG Exams/assignm ents x

The Assessment Grid Implemented by StudentInstructor University Target Programxx Academic analysis Instructional materialx Targeted assessment x Course/InstructorNot at UWSurvey?? StudentSALG Exams/assignm ents x

Goals Campus wide, quantitative course evaluations – Not to replace departmental evaluations Address Essential Learning Outcomes – Track student learning – Assess courses/programs – Educate students about the ELOs Post summary data on the web – Incentivize participation – Help students select courses – Be open with stakeholders

Why should instructors care? Know how we are doing Be judged on what student’s gain from the course (not charisma) Get students to reflect on learning rather than enjoyment

UW Essential Learning Outcomes

UW Essential Learning Outcomes

Approach Initial questions based on ELO language Survey Center conducted three rounds of focus groups to sequentially improve questions Also piloted data presentation

Initial question How much did this course enhance your knowledge of the human or natural world?

Final Recommendation In general, how much did this course enhance your knowledge of the world, such as knowledge of human cultures, society, or science? [Not at all, A little, Somewhat, Quite a bit, A great deal] – While students were not always sure what “..of the world” meant, with the defining words, they got the intended idea

Initial question How much did this course help you develop intellectual and practical skills, such as critical and creative thinking, written and oral communication, teamwork, and problem- solving? Combining intellectual and practical/pre- professional skills was confusing to students – split into two questions

Final Recommendations How much did this course help you develop intellectual skills, such as critical or creative thinking, quantitative reasoning, and problem solving? How much did this course help you develop professional skills, such as written and oral communication, computer literacy, and working in teams? [Not at all, A little, Somewhat, Quite a bit, A great deal]

Initial question How much did this course affect your values and sense of personal and social responsibility, for example by increasing your knowledge of policy issues, engagement in community and civic affairs, intercultural knowledge, or ability to reason ethically?

Final Recommendations How much did this course increase your sense of social responsibility, that is increased your knowledge of cultures or provided you with opportunities for civic or community involvement?” [Not at all, A little, Somewhat, Quite a bit, A great deal] There was some confusion between feeling vs. acting responsibly The question mixes an impact on a student with contents of a course We tried “reason ethically” but this confused students – More editing might be needed

Initial question How much did this course advance your ability to integrate diverse areas of knowledge?

Final Recommendations How much did this course improve your ability to combine knowledge or skills from different fields of study? [Not at all, A little, Somewhat, Quite a bit, A great deal] “I thought it just meant was the course interdisciplinary or not.” “Yeah, I thought the same thing. It didn’t have to do with my ability, but more the class itself.” If we interpret the ELO to mean interdisciplinarity within a course, then the question is working

Initial question How would you rate this course for its overall quality and educational impact? Quality and impact were found to be different.

Final Recommendation How would you rate the overall educational value of this course, that is the extent to which the course improved your all-around education or prepared you for the future?” [Very poor, Poor, Fair, Good, Very good] – Students understood: “How much you gain and take out of the course in general;” “How much you learned or how much you were challenged by the course and the subject;” “I also considered the practicality of it, for using it in the future and if it will benefit you in some way;” “Yeah, I was thinking just how happy you were that you took the course.”

Final Recommendation How would you rate the overall quality of this course, that is the extent to which it was structured and taught in order to maximize its educational value?” [Very poor, Poor, Fair, Good, Very good] – Unclear to some students if they should assess the professor or course structure – We want the focus to be on the course rather than the instructor – Maybe need to narrow, split, or drop this question

Final Recommendation Students felt it would be helpful to know whether a student was or was not “a major” Question added (at the start of the survey): At the time you enrolled, did you primarily take this course to fulfill a requirement for your major? [yes/no] – Students seemed to understand this question, as intended

General student reactions to the survey The survey made them reflect on their courses in ways they hadn’t before and thought the survey asked about aspects of courses where they did gain something One said you can appreciate classes for different reasons and “these questions reflected that.” Some students had mentally compared the answers they gave from one course to another ELOs became clearer The overall reception of the utility of the survey was very positive

Ideal implementation After each semester, students receive a link to a personalized survey covering all the courses they took in the previous semester Data are collated and made available to students and the public (CouseGuide?)

Sample Data Presentation The students liked the idea of having such data available – it would definitely help because there isn’t currently a way to know if a class is good or bad besides “anecdotes” – Better than “RateMyProfessor.com” - those ratings don’t necessarily tell you what the class is going to be like But they are realistic: – …A student shouldn’t put all his “trust into what the graph might say.”

5 Imaginary studies 101 (331 students; 105 majors) Imaginary studies 102 (183 students; 80 majors) Imaginary studies 202 (89 students; 33 majors) Imaginary studies 401 (23 students; 12 majors) Imaginary studies 402 (19 students; 17 majors) Imaginary studies 201 (113 students; 83 majors) Mean for nonmajors Mean for majors Sample data presentation Needs work!

Other things student wanted (but we don’t expect to provide) How much work How easy it was to get an A Correlation between the amount of work students put in and how much they feel they actually learned from the course Data for specific professors or TAs Nobody mentioned wanting to know how hot the professor was [Take that RmP!]

Data from pilot (8 students 3 courses each) 1 = Not at all 2 = A little 3 = Somewhat 4 = Quite a bit 5= A great deal 71% of courses received 4 or more on at least one ELO (Q2-6) 42% course received 5 on at least one ELO (Q2-6) 100% of students gave at least one 5 across their three courses (Q2-6)

Good correlation between overall impact and the other measures

But not for “overall quality”

Next steps Run a pilot survey on a large sample of students Use focus groups to improve data presentation style Consult with central administration on whether funds would be available to establish ongoing surveys and posting of data for all courses Seek faculty/staff buy-in (allowing for instructors to opt-out)

Thanks Janet Batzli TA Executive Committee and Assessment Subcommittee UW Survey Center – John Stevenson; Jennifer Dykema; Jaime Faus; Tara Piche Mo Bischoff UW Assessment Council