1. IDEA Student Ratings of Instruction Insight Improvement Impact ® Loyola University of Chicago April 10-11, 2013 Steve Benton, PhD Senior Research Officer

2. Individual Development Educational Assessment Teaching Improvement Faculty Evaluation Curriculum Review Program Assessment Accreditation

3. Kellogg Grant in 1975 Separate Organization and Non-profit status 2001 Mission Insight Improvement Impact ® To help colleges and universities as they seek to improve teaching, learning, and leadership A Non-Profit Organization

4. Plan for this Session What makes IDEA unique Good Conditions for the use of IDEA Student Learning Model Faculty Information Form Interpreting Reports Using Reports for Teaching Improvement Efforts Questions and Answers

5. What is teaching effectiveness? Being Organized Being Prompt Being Clear Relating course material to real life situations Requiring critical thinking Forming learning teams Introducing stimulating ideas Inspiring students to set and achieve goals

6. Most Surveys How well do the instructor’s methods resemble those of a “model” teacher? How well do students rate their progress on the types of learning the instructor targeted? Teaching Effectiveness

7. What is teaching effectiveness? Philosophy of IDEA Primary indicant of Teaching Effectiveness = Facilitating Learning

8. What makes IDEA unique? 1. Focus on Student Learning 2. Focus on Instructor’s Purpose 3. Adjustments for Extraneous Influences 4. Validity and Reliability 5. Comparison Data 6. Flexibility

9. Conditions for Good Use The instrument Targets learning Provides suggested action steps for teaching improvement

10. Conditions for Good Use The Faculty Trust the process Value student feedback Are motivated to make improvements

11. Conditions for Good Use Campus Culture Teaching excellence - high priority Resources to improve - provided Student ratings - appropriate weight

12. Conditions for Good Use The Evaluation Process 30-50% of evaluation of teaching 6-8 classes, more if small (<10) Not over-interpreted (3-5 performance categories)

13. Reflective Practice using Individual Reports Collect Feedback Interpret Results Read & Learn Reflect & Discuss Improve IDEA resources that are keyed to reports Talk with colleagues Try new ideas Online, Paper What the reports say and what they mean

14. Student Learning Model Specific teaching behaviors are associated with certain types of student progress under certain circumstances. Student Learning Teaching Behaviors Circumstances

15. Student Learning Model: Diagnostic Form Student Learning Items 21-32 Teaching Behaviors Items 1-20 Circumstances Students: Items 36-39, 43 Course: Items 33-35 Summary Items: 40-42 Research Items: 44-47 Up to 20 extra items

16. Student Learning Model: Short Form Summary Measures: Items 16-18 Experimental Questions: Items 14 20 Additional Questions Student Learning Items 1-12 Teaching Behaviors Circumstances Students: Items 13-15

17. Underlying Philosophy of IDEA Teaching effectiveness is determined primarily by students’ progress on the types of learning the instructor targets.

18. Faculty Information Form

19. FIF: Selecting Objectives 3-5 as “Essential” or “Important” Is it a significant part of the course? Do you do something specific to help students accomplish the objective? Does the student’s progress on the objective influence his or her grade? Be true to your course.

20. Curriculum Review Multi-section courses Prerequisite-subsequent courses Curriculum committee review

21. IDEA University SLOs Beyond the Major IDEA Student Learning Outcomes IDEA Objectives URI Learning Outcomes: Think critically Use methods and materials characteristic of knowledge area Lifelong learning, Self-directed learning Team skills 11 3 12 5 General Education Cognitive SLOs: Identify concepts, theories, developments Recognize issues, aesthetic and literary elements and forms Ask questions appropriate to the modes of inquiry Collect and analyze information 1 & 2 7 12 9 General Education Integrated Skills (each course 3 or more): Writing / Speaking effectively Reading complex texts Engaging in artistic activity Demonstrating information literacy Using quantitative and qualitative data Examining human differences 8 Items 33 & 35 6 9 2, FIF 3 New Item

22. IDEA Gen Ed Goals in the Curriculum IDEA University Gen Ed Courses 123456789101112 WRT 106XXX MTH 107XXXX BIO 101XXXX HIS 112XXXX ARH 120XXXXX ECN 100XXXX IDEA Learning Objectives

23. IDEA Gen Ed Goals in the Curriculum IDEA University Gen Ed Courses 123456789101112 WRT 106XXX MTH 107XXXX BIO 101XXXX HIS 112XXXX ARH 120XXXXX ECN 100XXXX IDEA Learning Objectives

24. Common Misconception #1 Students are expected to make significant progress on all 12 learning objectives in a given course.

25. Common Misconception #2 Effective instructors need to successfully employ all 20 teaching methods in a given course.

26. Relationship of Learning Objectives to Teaching Methods

27. Common Misconception #3 The 20 teaching methods items should be used to make an overall judgment about teaching effectiveness. Faculty Evaluation

28. Faculty Information Form

29. Course Description Items (FIF) Used for research Best answered toward end of term Do NOT influence your results Bottom of Page 1 Top of page 2

30. IDEA Online

31. FIF Online Delivery Reminders are delivered by email Start/end dates are determined by Institution Access is unlimited while available Questions can be added: Likert or open- ended Objectives can be copied

32. Copying Objectives

33. Student Survey Online Delivery Surveys are delivered by Email and/or the URL is embedded in a course site Blackboard Building Block is available Reminders are sent by email Start/end dates determined by Institution Submission is confidential and restricted to one

34. Online Response Rates – Best Practices Create value for feedback Monitor and Communicate Social Media Twitter Prepare Students Talk to students Put it in syllabus

35. Example: Course Syllabus Objective 3: Learning to apply course material (to improve thinking, problem solving, and decisions) Students will be able to apply the methods, processes, and principles of earth science to understanding natural phenomena Students will think more critically about the earth and environment Objective 8: Developing skill in expressing myself orally or in writing Students will be able to present scientific results in written and oral forms IDEA Center Learning Objective Course Learning Outcomes

36. Reflective Practice with IDEA Collect Feedback Interpret Results Read & Learn Reflect & Discuss Improve IDEA Student Ratings of Instruction Reports

37. Diagnostic Report Overview 1.How did students rate their learning experience? 2.What contextual factors impacted those ratings? 3.How do my scores compare to: IDEA, discipline, and institution? 4.What might I do to facilitate better learning for my students next time?

38. Your Average (5-point Scale) RawAdj. A.Progress on Relevant Objectives 1 Four objectives were selected as relevant (Important or Essential—see page 2) 4.14.3 1 If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average. 1. How did Students Rate their Learning?

39. Progress On Relevant Objectives 4 4.3 + 4.3 4.1 4.2 3.6 5

40. Summary Evaluation: Five-Point Scale Report Page 1 Your Average Score (5-point scale) RawAdj. A.Progress on Relevant Objectives Four objectives were selected as relevant (Important or Essential—see page 2) 4.14.3 Overall Ratings B. Excellent Teacher4.74.9 C. Excellent Course4.14.4 D. Average of B & C4.44.7 Summary Evaluation (Average of A & D)4.34.5 50% 25%

41. Adjusted Scores Student Work Habits (#43DF) Student Motivation (#39DF) Class Size (Enrollment, FIF) Student Effort (multiple items) Course Difficulty (multiple items)

42. 2. What contextual factors impacted those scores?

43. Work Habits (Item 43) Student Motivation (Item 39) High Avg. Avg. Low Avg. Low High 4.484.384.284.134.04 High Avg. 4.384.294.143.963.76 Average 4.284.144.013.833.64 Low Avg. 4.154.053.883.703.51 Low 4.113.963.783.583.38 Impact of Extraneous Factors Gaining Factual Knowledge – Average Progress Ratings Technical Report 12, page 40

44. Work Habits (Item 43) Student Motivation (Item 39) High Avg. Avg. Low Avg. Low High 4.484.38 High Avg. 4.384.29 Average 4.01 Low Avg. 3.703.51 Low 3.583.38 Impact of Extraneous Factors Gaining Factual Knowledge – Average Progress Ratings Technical Report 12, page 40

45. 2. What contextual factors impacted those ratings?

46. 3. How do my scores compare to: IDEA, Discipline, Institution?

47. Comparisons (Norms): Converted Averages Able to compare scores on the same scale T Scores Average = 50 Standard Deviation = 10 They are not percentiles or percentages

48. Comparisons (Norms): Converted Averages

49. Comparison Scores Distribution 40% Similar Lower 20% 10% Much Lower 10% Much Higher 20% Higher Gray Area

50. Comparison Scores

51. 4. What might I do to facilitate better learning next time?

52. Page 2: What did students learn?

53. Page 3: Suggested Action Steps #16 #18 #19

54. POD-IDEA Notes IDEA Website

55. POD-IDEA Notes Background Helpful Hints Application for online learning Assessment Issues References and Resources

56. References and Links to Helpful Resources are Provided

57. IDEA Papers Resources for Faculty Evaluation Faculty Development

58. Reflective Practice Collect Feedback Interpret Results Read & Learn Reflect & Discuss Improve POD-IDEA Notes IDEA Papers Meet with colleagues to reflect Interpret Reports Paper or Online Try something new

59. Questions ? www.theideacenter.org Visit our IDEA Help Community!IDEA Help