1. Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments: Workshop – Classroom Observations ASEE 2015 June 14, 2015 A Multi-Institutional Effort University of Washington – Denise Wilson, Cheryl Allendoerfer, Diane Jones Minnesota State University at Mankato – Becky Bates Seattle Pacific University – Melani Plett, Caitlin Wasilewski Simmons College – Nanette Veilleux Tuskegee University – Tamara Floyd-Smith Belonging and Engagement in STEM Education

2. Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments Complementing self-reports from students through both surveys and interviews, we also observed what students were actually doing in the classroom.

3. Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments Classroom Observation Instructor Activity Classroom Characteristics Student Engagement 20 Items 7 Items 13 Items

4. Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments Classroom Observation Instructor Activity Classroom Characteristics Student Engagement 20 Items 5 Items 13 Items Sample Items (Evaluated on a Four Point Likert Scale) Objective: Do students avoid alternative activities (e-mail, newspaper, etc.)? Do students answer questions when asked by the instructor? Do students ask questions of their own initiative? Do students ask follow-up questions/create dialogue with the instructor Are students on-time to class? Subjective: Do students maintain eye contact with the instructor or maintain attention to the instructor while he/she is speaking to the class? Do students express body language indicative of being bored or disinterested (fidgeting, wandering eyes, slouching posture, etc. Sample Items (Evaluated on a Four Point Likert Scale) Objective: Do students avoid alternative activities (e-mail, newspaper, etc.)? Do students answer questions when asked by the instructor? Do students ask questions of their own initiative? Do students ask follow-up questions/create dialogue with the instructor Are students on-time to class? Subjective: Do students maintain eye contact with the instructor or maintain attention to the instructor while he/she is speaking to the class? Do students express body language indicative of being bored or disinterested (fidgeting, wandering eyes, slouching posture, etc. Attendance was also included in this set of items

5. Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments Classroom Observation Instructor Activity Classroom Characteristics Student Engagement 20 Items 5 Items 14 Items Sample Items (Evaluated on a Four Point Likert Scale) Objective: Does the instructor use student names? Does the instructor ask questions of students and pause for responses? Does the instructor use group activities, think/pair/share, or similar active learning techniques? Subjective: Are visual aids well organized? Is the instructor interested, enthusiastic, and engaged in teaching? Does the instructor use humor appropriately? Does instructor not embarrass or belittle students in any way? Sample Items (Evaluated on a Four Point Likert Scale) Objective: Does the instructor use student names? Does the instructor ask questions of students and pause for responses? Does the instructor use group activities, think/pair/share, or similar active learning techniques? Subjective: Are visual aids well organized? Is the instructor interested, enthusiastic, and engaged in teaching? Does the instructor use humor appropriately? Does instructor not embarrass or belittle students in any way?

6. Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments Classroom Observation Instructor Activity Classroom Characteristics Student Engagement 7 Items Sample Items (Evaluated on a Four Point Likert Scale) Course Logistics Are the class web pages or other class documents well organized and easy to navigate? Do the web pages and/or syllabus provide complete logistics for the course? Does the syllabus offer “active” verb learning objectives? Classroom Environment Is the classroom environment comfortable for learning (lighting, temperature, etc.) Course Orientation Is the class oriented toward theory/basic science? Is the class oriented toward applications & examples? Sample Items (Evaluated on a Four Point Likert Scale) Course Logistics Are the class web pages or other class documents well organized and easy to navigate? Do the web pages and/or syllabus provide complete logistics for the course? Does the syllabus offer “active” verb learning objectives? Classroom Environment Is the classroom environment comfortable for learning (lighting, temperature, etc.) Course Orientation Is the class oriented toward theory/basic science? Is the class oriented toward applications & examples?

7. Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments InstitutionTarget MajorsClasses Observed HBCU Chemical Engineering, Electrical Engineering, Mechanical Engineering, Animal Science 23 sophomore; 15 junior 0 senior; 8 contrast Private (Faith-based) Computer Science, Electrical Engineering, General Engineering, Physics, Math, Chemistry 27 sophomore; 33 junior 16 senior; 14 contrast Teaching Automotive Engineering Technology Electrical Engineering, Civil Engineering, Mechanical Engineering, Chemistry 39 sophomore; 36 junior 25 senior; 16 contrast Research Electrical Engineering, Civil and Environmental Engineering, Chemical Engineering, Chemistry 55 sophomore; 26 junior 25 senior; 28 contrast Women’sComputer Science, Math, Chemistry 11 mixed 10 contrast

8. We analyzed our data, making no assumptions about what patterns lay in the data. K-Means Cluster Analysis places data into clusters or patterns so that each data point is assigned to a cluster by a minimum (e.g. Manhattan) distance to the mean of that cluster. Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments The Silhouette Values shown at left, reflect how similar each observation is to its assigned cluster; the number of clusters with maximum mean Silhouette Value is typically chosen as the optimal number of clusters. In this case, TWO = optimal number of clusters (or patterns)

9. Results of k-means clustering analysis (Single Institution) Instructor Activity: TWO patterns of activity Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments Student Engagement FOUR patterns of activity

10. Student Engagement Patterns Pattern 1: Most students are highly engaged most of the time Pattern 2: Low engagement with sporadic periods of high engagement Pattern 3: Low engagement with some students highly engaged. Pattern 4: Predominantly low engagement Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments Instructor Activity Patterns Pattern 1: Instructor interacts with students regularly and successfully Pattern 2: Instructor has few successful interactions with students

11. After k-means clustering analysis Qualitative analysis of the descriptive data in the observations worksheets was conducted to identify recurring and identifiable patterns of instructor activity and student engagement. This resulted in 8 different patterns of What Goes on in the Classroom Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments The Bottom Line Although certain instructor techniques were more successful than others, our analysis showed that no technique consistently linked to greater student engagement.

12. Each Instructor Activity Pattern Demonstrated each of four possible student engagement patterns Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments

13. Each Instructor Activity Pattern Demonstrated each of four possible student engagement patterns Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments

14. What do the Results mean? Sage on the Stage: Traditional lecture-style instruction, where the instructor presents content at the front of the classroom and students passively receive this content. Little interaction occurs between instructor and students and among individual students, which not surprisingly leads to low engagement levels. This category contains four patterns: A2, B2, B3, B4 Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments

15. What do the Results mean? Connected Sage on the Stage: The CSOS instructional patterns (A3, A4) are rooted in traditional lecture and presentation by the instructor, but also includes evidence of intentional opportunities for interaction offered from instructor to student. The use of “Real World” examples dominated this classroom observation style. Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments

16. What do the Results mean? Guide on the Side: The GOS category of activity draws heavily on student- centered learning strategies. The instructor spends little or no time delivering content through lecture, and students are actively engaged in discussions and learning activities throughout the observed class period. GOS tends to draw uniformly high engagement … through instructor design of activities or through activities that are inherently more student-driven (e.g. laboratories) Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments

17. Engaging Engineering Students is complicated The same technique can work with one group of students, but not the next. While Guide on the Side tends to lead to universally greater engagement, a hybrid style (Connected Sage on the Stage) seems to work successfully for a much broader range of classrooms. Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments

18. Lecture is not all that bad … if it’s mixed with instructor efforts to connect (develop relationship) OR finds a successful “hook” into the student experience (and interests) through application (“real world”), through active learning, or other techniques. http://upload.wikimedia.org/wikipedia/commons/8/88/University_Park_MMB_N7_Debating_Society.jpg http://commons.wikimedia.org/wiki/File:Writing_on_Board.jpg Facing the Challenge of Engaging Diverse Students in Diverse Academic Environments