1. Implications of Web-based Learning for Student Evaluation of University Teaching Chuck DziubanSteve Sorg Research Initiative for Teaching Center for Distributed EffectivenessLearning Ida CookMorgan Wang Department of SociologyDepartment of Statistics Patsy Moskal Research Initiative for Teaching Effectiveness University of Central Florida

2. Distributed Learning Impact Evaluation components Reactive behavior patterns Success rates Attitudes Demographic inertia Withdrawal rates Strategies for success Students Quality assurance Online surveys Modified instructional theories Large online classes Faculty Accreditation Online programs

3. Rationale for the Study Teaching evaluation data for a 3-year period were available to allow a comparison of two different sets of items (UCF/BOR). Responding to faculty interest, the UCF Faculty Senate requested that an evaluation of the Student Evaluation of Instruction measures be performed. (FS 1995-96-11)

4. The Instrument: UCF Items Feedback on your performance in this course The instructor’s interest in your learning Use of class time The instructor’s overall organization of the course Continuity from one class meeting to the next The pace of the course The instructor’s assessment of your progress The text and supplemental learning materials used

5. The Instrument: Board of Regent Items Description of course objectives and assignments Communication of ideas and information Expression of expectations for performance Availability to assist students In or outside of class Respect and concern for students Stimulation and interest in the course Facilitation of learning Overall assessment of instructor

6. The Study Layout Approximately 450,000 student responses Five Colleges: Arts and Sciences Business Administration Education Engineering Health and Public Affairs Three Levels: Lower Undergraduate Upper Undergraduate Graduate Three Years: 1996-97 1997-98 1998-99

7. Findings Summary Correlations among the 16 items are high -- median is approximately.70 Correlation between UCF and BOR forms =.92 Reliabilities of UCF & BOR forms are high There is only one factor Overall rating of the instructor is most strongly related to other items. The items have a characteristic distribution

8. Squared multiple correlations from each item with the remaining ones ItemWF2F 1.75.69 2.77.74 3.74.69 4.72.76 5.75.71 6.64.66 7.77.75 8.50.49 ItemWF2F 9.72.71 10.80.78 11.78.76 12.70.63 13.78.72 14.79.75 15.82.76 16.88.85

9. Variance components (Generalizability Theory) for the UCF items Variance Component%Component% Students.72667.85.72865.23 Items.00500.47.00200.19 Error.33931.68.38634.58 WF2F

10. Variance components (Generalizability Theory) for the BOR items Variance Component%Component% Students.79975.66.85571.96 Items.00500.48.008.0069 Error.25223.86.32527.35 WF2F

11. Overall Rating of the Instructor (N=444,017) ExcellentVery Good FairPoor 43% 29% 19% 7% 2%

12. Student Ratings by Modality Very ModalityExcellentGoodGoodFairPoor F2F42.0029.5019.007.202.40 (N=628,623) E44.0029.1017.406.902.60 (N=6,632) M40.6028.6020.607.702.40 (N=11,450) W55.4025.2012.104.902.50 (N=5,435) ITV20.9026.2030.5016.505.90 (N=3,218)

13. A comparison of W and F2F percentage of “excellent” ratings on UCF items Item 2 W (Fully online courses)F2F (Face-to-face courses) Item 3 Item 5 Item 8 51% 40% 50% 37% 52% 41% 47% 27%

14. A Comparison of W and F2F percentage of “excellent” ratings on BOR items Item 12 W (Fully online courses)F2F (Face-to-face courses) Item 13 Item 15 Item 16 50% 38% 59% 47% 52% 34% 55% 42%

15. Overall Rating of the Instructor Binary Decision Tree SAS Enterprise Miner Remaining 15 Items Level Year College Developmental Sample 1 Developmental Sample 2 Developmental Sample 3 Validation Sample Predictors

16. Decision Tree Example 85.9% n=11,286 85.8% n=6,460 91.5% n=2,079 72.7% n=378 86.7% n=2,369 86.5% n=5,639 74.8% n=821 94.1% n=1,036 89.1% n=1,043 64.7% n=148 79.6% n=230 88.4% n=3,263 84.1% n=2,376 68.9% n=298 78.5% n=526 Arts & Sciences, Business Admin., Hospitality Mgmt. Education Engineering Health & Pub. Affairs F2F, E, MW femalesmalesA&SBA & Hosp. mgmt F2FE, M, WE, MF2F Overall

17. Rule #1: If... Facilitation of learning & Communication of ideas Excellent Very Good Good Fair Poor Then... The probability of an overall rating of Excellent =.93 & The probability of an overall rating of Fair or Poor =.00

18. A comparison of excellent ratings by college unadjusted and adjusted for instructors satisfying Rule 1 CollegeUnadjusted %Adjusted % Arts & Sciences41.692.4 Business34.990.9 Education56.894.8 Engineering36.291.3 H&PA46.193.9 (N=441,758) (N=147,544)

19. A comparison of excellent ratings by course modality--unadjusted and adjusted for instructors satisfying Rule 1 F2F42.092.2 E44.092.3 M40.692.0 W55.492.7 ITV20.986.7 Course ModalityUnadjusted % Adjusted % N=709,285 N=235,745

20. Instructor is interested in your learning Communication of ideas Excellent Very Good Good Fair Poor Then... The probability of an overall rating of Excellent =.84 & The probability of an overall rating of Fair or Poor =.00 Organization of the course Facilitation of learning Rule #2: If... Respect & concern for students

21. A comparison of excellent ratings by college unadjusted and adjusted for instructors satisfying Rule 2 CollegeUnadjusted %Adjusted % Arts & Sciences41.686.8 Business34.981.6 Education56.881.9 Engineering36.284.4 H&PA46.184.5 N=441,758N=15,207

22. A comparison of excellent ratings by course modality--unadjusted and adjusted for instructors satisfying Rule 2 F2F42.084.3 E44.084.6 M40.685.6 W55.487.5 ITV20.971.6 Course ModalityUnadjusted %Adjusted % N=709,285 N=235,745

23. Instructor is interested in your learning Communication of ideas Excellent Very Good Good Fair Poor Then... The probability of an overall rating of Excellent =.78 & The probability of an overall rating of Fair or Poor =.00 Organization of the course Facilitation of learning Expresses expectations for student performance Respect and concern for students Use of class time Rule #3: If...

24. A comparison of excellent ratings by college unadjusted and adjusted for instructors satisfying Rule 3 CollegeUnadjusted %Adjusted % Arts & Sciences41.681.6 Business34.978.0 Education56.874.5 Engineering36.280.0 H&PA46.177.6 N=441,758N=15,060

25. A comparison of excellent ratings by course modality--unadjusted and adjusted for instructors satisfying Rule 3 F2F42.083.9 E44.086.5 M40.684.0 W55.488.4 ITV20.981.1 Course ModalityUnadjusted Adjusted N=709,285 N=235,745

26. A conceptual path diagram for an instructor receiving an overall rating of Excellent at UCF Facilitate student learning Interested in student learning Well organized course Respect and concern for students Clear expectations for students Respect & concern for students Interested in student learning Effective use of class time Well-organized course Ability to communicate info. and ideas Prerequisite Excellent Rating Facilitative climate.93*.84*.78* * probability of an excellent rating Supportive environment Organization Excellent Rating