1. 1 Quality, quantity and diversity of feedback in WisCEL courses enhances relationships and improves learning John Booske Chair, Electrical and Computer Engineering Department, Director, Wisconsin Collaboratory for Enhanced Learning (WisCEL), and Duane H. and Dorothy M. Bluemke Professor Helen C. White College Library WisCEL Wendt Commons WisCEL WisCEL Website: www.wiscel.wisc.edu

2. 2 WisCEL Feedback Strategies WisCEL facilitates multiple opportunities for student and instructor feedback: Students  Instructor-based feedback  Technology-based feedback  Student-based feedback Instructors  Technology-based feedback  Student-based feedback  Instructor-based (peer) feedback

3. 3 Instructor-based feedback For students  Instructor role as “coach”  Scaffold student thinking  Focused and timely expertise  “Just-in-time feedback”  Answer questions with questions  Probe for conceptual understanding  Expose misconceptions  Low instructor-to-student ratio For instructors  Peer learning community

4. 4 Technology-based feedback Instructional technology supports synchronous and asynchronous feedback throughout the course in formative assessment, in-class work, homework, and exams:  Provides immediate right/wrong answers and scoring  Instructors can monitor learning progress  Incentivizes student class preparation  Scaffolds progression through complicated problems  Supports student collaboration, freeing instructor to interact and assist all students

5. 5 Student-based feedback Students: Student-to-student feedback facilitates learning through pairs or small group problem-solving, inquiry, questioning, discovery, projects and discussion.  Peer-to-peer learning: Students’ learn through shared work, and iterative, reciprocal feedback  Peer-to-peer teaching: Students’ learn through explaining and guiding their peers Instructors:  Student-to-instructor: students’ questions and conversations with instructor or TA “coaches” reveal a rich and diverse set of insights on how content, practice assignments, assessment questions, etc, are interpreted through the students’ perspective.

6. 6 ECE course A Spring 2012: WisCEL Fall 2011: Traditional

7. 7 Nuclear Engineering course Pre- vs. post- WisCEL with full blended implementation (incl lecture quizzing) Lowest final course grade = C Fall 2011 exam 1 Fall 2011 exam 2 Fall 2012 exam 2 Fall 2012 exam 1

8. 8 ECE course C1 (WisCEL) vs. C2 (traditional) ECE C1 is new, but the course content difficulty is equivalent to ~ 1/3 rd of ECE C2 in the past.

9. 9 The analysis showed that by Fall 2012, among new freshmen, TM students’ performance matched that of non-TM students. Approximately 60% of each population received a grade B or better and TM students had lower rates of adverse results (grade of D, F or Drop) than non-TM students. From: Math 112 Fall Semester Grade Report, Shirin Malekpour, 2013. Math 112 Fall Semester Grade Report*

10. 10 ECE course D Final Exam Performance  Fall 2012 (WisCEL) 30 students  Mean: 78.7  Median: 79.5  Fall 2010 (conventional lecture) 18 students  Mean: 70.5  Median: 69  Fall 2009 (conventional lecture) 20 students  Mean: 71.7  Median: 74 A nonparametric Wilcoxon rank sum test was conducted on the hypothesis that F12 and [ F10; F09 ] (grouped) had equal medians. The hypothesis of equal medians was rejected at a p-value of p = 0.012. When the F08 scores were included (also a “conventional lecture” semester) the p-value dropped further to p = 0.005. This is a very strong conclusion that the students performed better with the blended learning section in WisCEL versus the previous conventional lecture mode of instruction. Interestingly, the instructor’s sense was that the F12 section (blended/flipped) had a more difficult final exam, which, if true, would further strengthen the contrast/conclusion that better learning came from the blended section in WisCEL. 1-p is the probability that the medians of the underlying distributions are truly different (and not a statistical sampling fluke). In this case the probability is almost 1, since 1-p = 0.995. Very high confidence that there was a shift in the distribution of scores.

11. 11 WisCEL Advantages Space and layout promotes interaction, collaboration, discussion, and multiple opportunities for feedback. Technology provides synchronous and asynchronous opportunities for immediate online scoring and options for students to ask questions in real-time, and instructors to provide targeted and timely guidance. Instructional design facilitates interaction and develops relationships between instructors and students that support student-centered instruction and individualized feedback. Periodic workshop participation facilitates learning community among instructors in WisCEL

12. 12 WisCEL outcomes from blended instruction Finding: The majority of WisCEL courses saw an increase in the percentage of students attaining a mastery-level grade of B or better and saw a decrease in the percentage of students with adverse outcomes (a grade of D or F or Drop).  An analysis was conducted comparing student outcomes in 13 courses held WisCEL in Fall 2012 and Spring 2013. Outcomes from each WisCEL course were compared to prior semesters of the same course taught in a traditional classroom setting for up to eight semesters (Fall 2008 – Spring 2012). Two types of outcomes were considered to determine if WisCEL courses met the goal of producing equal or better outcomes: 1) the percentage of students who received a grade of B or better in the course; and 2) the percentage of students who received a grade of D or F, or who dropped the course. The analysis produced the following outcomes:  Increases in % of students with grades B or better  9 of 13 WisCEL courses examined had 80% or more of students attaining a grade of B or better.  11 of 13 WisCEL courses offered Fall and/or Spring (2012-2013) attained increases in the percentage of students who received a grade of B or better as compared to previous semesters.  Decreases in % of students with grades D, F or Drop  12 of 13 WisCEL courses in Spring 2012, Fall 2012 and Spring 2013 had substantially lower percentages of students receiving a grade of D, F or who dropped the course compared to the same courses offered in traditional settings in the previous 4 semesters.  Engineering courses saw the biggest decreases in DF/Drops in WisCEL courses over traditional courses. The percentage of students receiving a DF/Drop declined by 9-25% in eight engineering courses taught in WisCEL in Fall and/or Spring 2012-13.