1. Is it working? Assessing hybrid faculty development and the hybrid classroom Andreas Brockhaus and Carol Leppa University of Washington Bothell

2. Hybrid Course Development Institute (HCDI) u Main goal: Create a peer-reviewed, new or redeveloped course syllabus for a hybrid format class u Cohort of 12 faculty u Provide faculty incentives (currently $1000) u HCDI faculty team: Learning Technologies, Nursing, Library u Designed around Community of Inquiry model

3. HCDI structure evolution 2010 Structure u 10-week model – 3 face-to-face meetings 3 hours long – 1.5 to 2 hours online each week – Up to 4 weeks between face- to-face meetings – Ran twice – $500 stipend 2013 Structure u 6-week hybrid model – 6 face-to-face meetings 1 hour long Met weekly at the same time – 2 hours online work weekly – Ran twice – $1000 stipend

4. Moving from 10 to 6 weeks u Made HCDI shorter and more hybrid u Moved most content and lecture online u More focus on a couple specific tools u Focused mainly on social presence u More templates and peer reviews

5. Results 47% full-time participants 79% full-time participants 29% tenure- track 61% tenure- track 10 weeks6 weeks 53% completed all requirements 96% completed all requirements

6. hybrid How to assess impact u Pre- and post-HCDI participant surveys u Faculty interviews (post-HCDI) u Student evaluations

9. Feedback u Participants found the HCDI useful u Being a student was helpful u Interacting with colleagues and instructors was invigorating u Experiencing a hybrid course and defining hybrid was useful u Wanted even more technology focus u Challenges teaching hybrid included: – Technology – Course design – Time out of class

10. Feedback “There is no real substitute for an institute of this nature that asks participants to work through the many stages essential to (course) development.” “Being a student in the class taught me to think like a student.”

11. Course Evaluations u Do Standard course evaluations reflect hybrid course COI evaluations? u Completed and compared COI and standard course evaluation form X u N=371 cases – 19 class sections Sum 12 thru Spr 13 – 8 to 40 evaluations per section

12. Garrison, Vaughn; Blended Learning in Higher Education (2010)

13. Compared Factor Analysis COI confirmatory Factor Analysis u 3 factor model confirmed u 60% variance explained u Teaching Presence u Social Presence u Cognitive Presence Factor analysis Form X u 3 factor model u 71% variance explained u Effective content learning – Content – Instructor action u Effective Communication – Relative to expectations u Student reflection – Effort and engagement

14. Combined COI and Form X Factor Analysis u 4 factor model – 62% of variance u Teaching Presence/Instruction u Cognitive Presence/content u Social Presence as separate factor (COI only) u Student self reflection as separate factor (Form X only)

15. Correlations of items: Form X with COI questions u Highly skewed data for both standard and COI evaluations (any > than.15=“statistically significant”) u Teaching Presence – 7 items with r=.55 to.67 u Cognitive Presence – 5 items with r =.50 to.60 u Social Presence – 5 items with r =.25 to.39

16. Global course evaluation scores/Teaching Presence All forms N=1232 Mean(SD) 5-point scale questions HCDI hybrid n=32 Other hybrid n=21 All other n=1179 Course as a whole4.3 (.50) 4.1 (.73) 4.1 (.63) Course content4.3 (.42) 4.1 (.72) 4.1 (.60) Instructor’s contribution 4.5 (.46) 4.3 (.68) 4.3 (.63) Instructor’s effectiveness 4.4 (.48) 4.2 (.85) 4.2 (.71) Comb items 1-44.4 (.45) 4.4 (.45) 4.2 (.62) CEI (7-point scale) 5.0 (1.10) 5.2 (.52) 4.9 (1.20)

17. Social Presence proxy means N=896 course sections Forms A&X Mean(SD) 7-point scale questions HCDI alums N=31 Hybrid NON- HCDI N=20 All other courses N=845 Sessions engaging 6.4 (.49) 6.1 (.80) 6.2 (.60) Student participation encouraged 6.5 (.49) 6.4 (.74) 6.4 (.56) Help available 6.3 (.56) 6.3 (.69) 6.3 (.54) Learning concepts in course 6.4 (.51) 6.1 (.84) 6.2 (.65) Applying material in real world 6.4 (.46) 6.2 (.80) 6.3 (.60)

18. Cognitive Presence proxy means Form X only 101 Sections Mean(SD) 7-point scale questions HCDI alums N=13 Hybrid NON- HCDI N=11 All other courses N=77 Learn conceptual knowledge 6.4 (.63) 5.9 (.99) 6.2 (.71) Dev appreciation for field 6.5 (.54) 6.1 (.95) 6.4 (.70) Understand material in field 6.2 (.74) 5.9 (.94) 6.3 (.62) Solve problems in field 6.3 (.59) 5.8 (1.0) 6.2 (.60) Gen intellectual development 6.4 (.66) 6.0 (.99) 6.4 (.57)

19. Global course evaluation questions Pre- Post-HCDI N=9 Pre-HCDI Mean (SD) Post-HCDI Mean (SD) Course as a whole 5=highest 4.0 (.51) 4.3 (.60) Course content 4.0 (.58) 4.2 (.55) Instructor’s contribution 4.3 (.58) 4.4 (.61) Instructor effectiveness/content 4.1 (.54) 4.3 (.67) Combined 5=highest 4.1 (.53) 4.3 (.60) Challenge/Engagement Index 7=highest 5.4 (.58) 5.4 (.60)

20. Social Presence proxy evaluation items N=9 Pre-HCDI Mean (SD) Post-HCDI Mean (SD) Sessions engaging 7=highest 6.0 (.42) 6.3 (.70) Student participation encouraged 6.6 (.33) 6.7 (.27) Help available 6.4 (.33) 6.5 (.46) Learning concepts in course 6.2 (.46) 6.5 (.59) Applying material in real world 6.3 (.35) 6.5 (.50)

21. Challenges u Identifying and tracking number of hybrid courses u Mixed institutional interest u Student evals are outdated – Inadequate for capturing social presence – Inadequate for capturing online/hybrid info

22. Questions u Do you have hybrid-specific course evaluations on your campus? If so, how does it differ from other course evaluations. If not is it reasonable to use CoI model to design student surveys?

23. References sources for COI instrument questions u Garrison, D.R., Cleveland-Innes, M., Fung, T.S. (2010) Exploring causal relationships among teaching, cognitive and social presence: Student perceptions of the community of inquiry framework, Internet and Higher Education, 13, 31-36. u Diaz, S.R., Swan, K., Ice, P., Kupczynski, L. (2010). Student ratings of the importance of survey items, multiplicative factor analysis, and the validity of the community of inquiry survey, Internet and Higher Education, 13, 22-30. u Shea, P. & Bidjerano, T. (2012) Learning presence as a moderator in the community of inquiry model, Computers & Education, 59, 316-326. u Carlon, S., Bennett-Woods, D., Berg, L. et al (2012) The community of inquiry instrument: Validation and results in online health care disciplines. Computers & Education, 59, 215-221.

24. Appendix Teaching Presence Spearman’s rho =.50 to.65 Form X question u The course as a whole u The course content u Instructor’s effectiveness u Instructor explanations COI Questions u Facilitation questions u Facilitation u Design/facilitation

25. Appendix Cognitive Presence Spearman’s rho =.50 to.60 Form X questions u Learning content u Understanding material in field u Solving problems in field u General intellectual development COI questions u Integration u Trigger events/resolution u Integration/resolution

26. Appendix Social Presence Spearman’s rho =.20 to.40 Form X questions u Class sessions interesting and engaging u Student participation encouraged u Aware expectations u Apply real world problems COI questions u Affective experience/open communication u Affective exp/group cohesion u Affective exp/open communication/group cohesion

27. Appendix COI Evaluation questions u Teaching Presence – Design/organization 4 items – Facilitation 6 items (teacher-group support) – Direct instruction3 items

28. Appendix COI Evaluation questions u Social Presence Student-student and student-teacher – Affective expression3 items – Open communication3 items – Group cohesion3 items

29. Appendix COI Evaluation questions u Cognitive Presence – Trigger events3 items Activities/assignments – Exploration3 items Breadth of topic discussions – Integration3 items Constructing solutions – Resolution3 items Application