1. Using Sakai for Teaching Evaluations Sean DeMonner, Project Manager, University of Michigan Ellen Yu Borkowski, University of Maryland Bryan Bakotich, Pacific Lutheran University

2. Overview Evaluation System Institutional stories Q&A July 200910th Sakai Conference - Boston, MA, U.S.A.2

3. Evaluation System Provide the ability to run evaluations (course, department, institutional, group) or surveys in a flexible way which meets the needs of most Sakai using institutions July 200910th Sakai Conference - Boston, MA, U.S.A.3

4. Summary of features Basic template authoring system – Template – collection of items – Items – questions or response statements – Scales – Likert or M/C options Allow groups (instructors, departments, etc.) to create evaluations/feedback/surveys Ad-hoc groups also supported July 200910th Sakai Conference - Boston, MA, U.S.A.4

5. Summary of features Create evaluations which can be assigned to various groups at once and can be set up in advance Sends out notifications and reminders to evaluators which allow one-click access Ability to edit email templates used for notifications and reminders July 200910th Sakai Conference - Boston, MA, U.S.A.5

6. Summary of features Direct links into Sakai for all evaluations are available and login is handled automatically if needed anonymous is available Basic online reporting and the ability to export results as CSV, PDF, XLS July 200910th Sakai Conference - Boston, MA, U.S.A.6

7. Pacific Lutheran University Liberal Arts University with graduate programs in Nursing, Business, and Education Total Enrollment: 3661 Sakai in production since 2006 July 2009710th Sakai Conference - Boston, MA, U.S.A.

8. Homegrown system using Apache, PHP, and MySQL Pilot with 7 courses in 2004 Pilot with Rank and Tenure Committee in 2006 and 2007 Recommended to form another committee and study it further Early Attempts July 2009810th Sakai Conference - Boston, MA, U.S.A.

9. Administrator Evaluations Archaic process Source of frustration Good place to pilot Evaluation System New Student Safety Orientation Fulfill grant requirements Track response rates and send reminders to non respondent students Sakai EvalSys was recommended after success of Administrator Evaluation Types of Evaluations July 2009910th Sakai Conference - Boston, MA, U.S.A.

10. Administrator Evaluation Results “Many Hours” 2 WeeksFew Hours Create Lists Color Code Evals Generate Memos Receive and Process Evals Generate Results Send Out Evals July 200910th Sakai Conference - Boston, MA, U.S.A.10 15 MinutesLess Than 1 DayFew hours Create Lists Create and Populate Sakai Sites Send out Evaluations Download Results Paper Online 43% Response Rate? 53% Response Rate

11. New Student Orientation Results 1 evaluation on small focus group of transfer students 60% response rate (25 / 42) No time spent monitoring evaluation Previously took a “tremendous” amount of time July 20091110th Sakai Conference - Boston, MA, U.S.A.

12. Problems Encountered Lack of set up time Email reminders treated like spam due to lack of personalization Unable to link numeric values with comments in results All evaluators email addresses show up in to field of emails sent out by system July 20091210th Sakai Conference - Boston, MA, U.S.A.

13. Futures Modify code to hide email addresses of evaluators “Heads up” email to increase response rates Future plans to add video content to evaluation Faculty Evaluations? July 20091310th Sakai Conference - Boston, MA, U.S.A.

14. University of Michigan July 200910th Sakai Conference - Boston, MA, U.S.A.14

15. Teaching Questionnaires @ U-M History of Teaching Evaluations at U-M Collaboration between MAIS and CTools Concerns over move to online system: Research drives tenure + teaching excellence is expected - but evaluations can impact tenure decisions = significant political implications & risks Multiple pilots; adjustments Full scale rollout F08; W09 meltdown; ongoing production System operations and shifting expectations (e.g. frequency, reporting) July 20091510th Sakai Conference - Boston, MA, U.S.A.

16. Teaching Questionnaires @ U-M U-M specific features: External data loading Single email Reminder email handling Confirmation email Admin summary page controls Architectural details: PeopleSoft order entry ->XML data exchange -> Sakai -> PeopleSoft reporting July 20091610th Sakai Conference - Boston, MA, U.S.A.

17. TQs: Response Info July 20091710th Sakai Conference - Boston, MA, U.S.A. Reminders drive response but students resent spam…

18. TQs: Response Rates July 20091810th Sakai Conference - Boston, MA, U.S.A.

19. TQs: Initial Outcomes Challenges: Customized email job Load modeling and “spiky” applications Operational issues; system maturity Reporting problems; managing expectations Data exchange root of all evil (REST?); shifting requirements Fully Sakai-based system would not have this problem July 20091910th Sakai Conference - Boston, MA, U.S.A.

20. TQs: Initial Outcomes Successes: Response rates (72% students; 60+% evals) Favorability rates (no significant difference) Communications team Faculty Steering Committee in place Data warehouse coming (numeric data) July 20092010th Sakai Conference - Boston, MA, U.S.A.

21. University of Maryland Major public research university Carnegie Doctoral/Research University – Extensive Total Enrollment: 37,000 Sakai in production since Fall 2007 with only EVALSYS being used July 200910th Sakai Conference - Boston, MA, U.S.A.21

22. July 200910th Sakai Conference - Boston, MA, U.S.A.22 CourseEvalUM History of CourseEvalUM University Senate Task Force on Course and Teaching Evaluation report April 2005 Student Course Evaluation Implementation Committee charged by Provost February 2006 Final recommendations to University Senate April 2006 (approved)

23. July 200910th Sakai Conference - Boston, MA, U.S.A.23 CourseEvalUM 15 university-wide items defined 6 used for APT purposes 7 are public for student access 1 open ended comment item 1 question about fit in academic plan CORE Major/Certificate/Minor/Program Requirement Elective

24. July 200910th Sakai Conference - Boston, MA, U.S.A.24 CourseEvalUM Access to results Two-tiered incentive system for students Individual student must fill in all of their evaluations the prior semester to gain access to results of public questions for students Course must have 70% return rate for course results to be available to students who satisfy requirement above

25. July 200910th Sakai Conference - Boston, MA, U.S.A.25 CourseEvalUM Access to results (con’t) Instructors can view all items for their own course(s) Administrators can view APT items

26. July 200910th Sakai Conference - Boston, MA, U.S.A.26 CourseEvalUM Maryland-specific features External group provider Data pull from SIS on courses/enrollments Hierarchy University and college/school Multiple instructor Reminder email handling Suppress initial email; use reminder email in tool

27. CourseEvalUM Architectural details: SIS (proprietary) -> Sakai -> CourseEvalUM Reports (proprietary)

28. July 200910th Sakai Conference - Boston, MA, U.S.A.28 CourseEvalUM Pilot results University items only Summer I 2007: 40.7% Summer II: 40.8% Fall 2007: 63%

29. July 200910th Sakai Conference - Boston, MA, U.S.A.29 CourseEvalUM Production results University and college level items and multiple instructor Spring 2008:62% (3 colleges) Summer I 2008: 39% (3 colleges) Summer II 2008: 45% (4 colleges) Fall 2008: 61% (7 colleges) Spring 2009: 66% (7 colleges)

30. July 200910th Sakai Conference - Boston, MA, U.S.A.30 CourseEvalUM Challenges Email “spamming” Students complained about spamming Starting Fall 2007, initial email notification was sent once (one per student) and reminders were sent multiple times (one per evaluation)

31. CourseEvalUM Challenges (con’t) Performance issues Fall 2007 Way in which data was being parsed – patch is now in trunk Scope creep Limited resources Next Steps Waiting on prioritized functional list

32. Questions? University of Michigan http://www.umich.edu/~eande/tq/onlineeval s.htm University of Maryland https://www.irpa.umd.edu/Assessment/crs_e val.shtml Pacific Lutheran University July 200910th Sakai Conference - Boston, MA, U.S.A.32