1. IDEA Student Ratings System Loyola University September 9 & 10, 2015 Jake Glover, PhD Senior Education Officer

2. Session Plan – Review IDEA System (brief) – Strategies for Selecting Learning Objectives – A Look at the New Reports – IDEA2 – What Else is New in IDEA? – Review – Q&A

3. What Makes IDEA Unique? 1.Focus on Student Learning 2.Focus on Instructor’s Purpose 3.Adjustments for Extraneous Influences 4.Validity and Reliability 5.Comparison Data 6.Flexibility Wouter de Bruijn

4. Reflective Practice Collect Feedback Interpret Results Read & Learn Reflect & Discuss Improve At home, in class, in lab Context, comparisons Resources keyed to reports Colleagues, IDEA users Try new ideas

5. Student Learning Model Prioritizing where to focus first by better understanding the relationships between: -Teaching Methods -Learning Objectives -Influential Factors

6. The Foundations of SRI System Teaching Methods: Learning Objectives: Influential Factors: Key Influences: Key Resources: POD IDEA Notes IDEA Papers (#50 good start) Blog Other on website Faculty/Student Collaboration Kellogg Foundation Bloom’s Taxonomy Others… Class Size, Student Motivation, Work Habits, Others 12 20

7. Conditions for Good Use Campus Culture Teaching excellence - high priority Resources to improve - provided Student ratings - appropriate weight

8. Conditions for Good Use Evaluation Process 30-50% effective teaching evaluation Sufficient N (6-8 classes, more if < 10 students per course)

9. Include Multiple forms of Assessment Student Ratings External Perspective Artifacts Balanced Plan for Summative Evaluation

10. Key Factors of the FIF Types of learning must reflect the instructor’s purpose, which will be different across courses and disciplines

11. Key Factors of the FIF Teaching effectiveness is determined by student’s learning on those objectives stressed by the instructor

12. FIF: Getting Started

13. TEACHING & LEARNING

14. For Students

15. FIF: Selecting Objectives

16. Selecting Objectives 3-5 as “Essential” or “Important” 1.Is it a significant part of the course? 2.Do you do something specific to help students accomplish the objective? 3.Does the student’s progress on the objective influence his or her grade? Be true to your course

17. Selecting Objectives How to select “important” or “essential” IDEA objectives related to the course – Use the Teaching Goals Inventory to help select objectivesTeaching Goals Inventory – Use the Three Guiding Questions – Match IDEA with course objectives – Use a Relevant Learning Theory (i.e. Bloom’s Taxonomy)

18. Selecting Objectives 1 2, 7 3, 8, 9 10, 11 6 IDEA OBJECTIVES 1. Gaining factual knowledge 2. Principles, generalizations, theories 3. Application for improved thinking 4. This profession’s points of view, skills 5. Team skills 6. Creative capacities 7. Broad, liberal education 8. Expressiveness (written and oral) 9. Find & use resources 10. Personal values 11. Analyze and critically evaluate ideas 12. Learn more by asking questions

19. Selecting Objectives: Misconception Students are expected to make significant progress on all 12 learning objectives in a given course.

20. Diagnostic Report Overview

21. Progress on Relevant Objectives What did students learn?

22. Mapping a Program Name of Course123456789101112 Senior Research ProjectXXXX Introduction to Psychology IXXX Introduction to Psychology IIXXX Developmental PsychologyXXXX Cognitive PsychologyXXXXX Neurobiological PsychologyXXX Clinical PsychologyXXXXX History of PsychologyXXXX Research MethodsXXXXX

23. Collect Feedback Interpret Results Read & Learn Reflect & Discuss Improve DATA WISDOM KNOWLEDGE INFORMATION

24. Additional Questions

26. What if you do not select any Learning Objectives?

27. Questions so far?

28. A fowl story with a happy ending… normanack

29. Page 3: What might I do differently?

30. Relationship of Learning Objectives to Teaching Methods

31. Diagnostic Report Overview

32. Formative Page What can I do differently?

33. Resources for Improvement IDEAedu.org Research and Papers – POD IDEA Notes on Learning – POD IDEA Notes on Instruction – IDEA Papers IDEA blog

35. Diagnostic Report Overview

36. Progress on Relevant Objectives What did students learn?

37. Summative View How did I do?

38. Description of Course and Students

39. Diagnostic Report Overview

40. Overall Ratings Excellent Teacher and Excellent Course 50% of Summary Evaluation Score

41. Quantitative View Statistical Detail - Any additional insights?

42. Adjusted Scores, Converted Averages, and Comparison Scores

44. Adjusted Scores Raw ScoresAdjusted Scores

45. Impact of student and course characteristics Most relevant: – Student Work Habits – Student Motivation Relevant to potentially important: – Class Size – Student Effort – Course Difficulty

46. Adjusted Scores Description of Course and Students

47. Impact of Extraneous Factors Work Habits (Item 43) Student Motivation (Item 39) High Avg. Avg. Low Avg. Low High 4.484.384.284.134.04 High Avg. 4.384.294.143.963.76 Average 4.284.144.013.833.64 Low Avg. 4.154.053.883.703.51 Low 4.113.963.783.583.38 Average Progress Ratings: Gaining factual knowledge

48. Raw or Adjusted Scores PurposeRaw or Adjusted? How much did students learn? Raw What were the instructor’s contributions to learning? Adjusted How do courses compare? Adjusted

49. Converted Averages Compare scores on the same scale T Scores Average = 50 Standard Deviation = 10 Not percentiles or percentages Similar: Middle 40% (45-55)

50. Comparison Scores IDEA DatabaseDiscipline Comparison

51. Validity To what extent do student rating items accurately measure some aspect of teaching and learning that is important to effectiveness? jkfid

52. Validity: IDEA IDEA Research Report #2 Technical Report #18 IDEA Paper # 50

53. Concurrent Validity Student ratings of instruction are positively correlated with: – Instructor self-ratings – Ratings by administrators – Ratings by peers – Ratings by alumni – Ratings by trained observers – Review of course materials

54. More Evidence of Validity There are positive correlations between student self- reports of progress on relevant objectives and performance on course exams. Benton, S. L., Duchon, D., & Pallett, W. H. (2011): Validity of student self-reported ratings of learning, Assessment & Evaluation in Higher Education. Generally performed better on course exams than those who rated their progress as moderate or less.

55. Diagnostic Feedback Form 47 questions Full assessment of student ratings of instruction using teaching methods and learning objectives Analysis is highly sophisticated, and changes based on relevant learning objectives Summative ratings Formative ratings Development materials connected Footer - Presentation Title

56. Learning Outcomes 18 questions Assessment of relevant learning objectives Assessment of teaching at summative level Assessment of course at summative level Concentrates on course outcomes, rather than faculty performance Good for clinicals, preceptorships, internships, etc., where faculty interaction is not as integral as student experience Footer - Presentation Title

57. Pilot 2015, Ultimate Objective Update learning objectives and teaching methods in our Student Ratings of Instruction tools to better reflect current trends in higher education

58. Pilot 2015, why now? IDEA research – New items were identified that align with national standards, including the AAC&U, regional accreditors, and other recognized learning organizations – Some current items were identified as less relevant or dated and were eliminated or revised Principle of continuous improvement

59. What is being added… Teaching Methods: diversity, self-reflection, and service-learning Learning Outcomes: diverse perspectives and global awareness, ethical reasoning and decision making, civic engagement, and quantitative literacy Student and Course Characteristics; self-efficacy and background and preparation

60. Pilot 2015 Process IDEA-CL Platform (courses ending May 2015) 13 items will be added to the existing student ratings Diagnostic Feedback instrument 6 new (or newly revised) Learning Outcomes will be added to the existing FIF Individual faculty reports will include statistical data for the new items; there will be no other changes to the existing reports

61. IDEA 2 Following successful pilot: Diagnostic Feedback 2.0, updated and shorter diagnostic instrument (approx. 40 items) FIF 2.0, revised Faculty Information Form Learning Outcomes 2.0, formerly referred to as the “Short” form, updated

62. NEW: Teaching Essentials  New 12-item instrument  includes specific teaching methods and external variables greatest impact on student ratings of overall course and instructor excellence  No FIF required  Application  EOT- End of term- April (tentative)  ARS-Automatic Response System- Summer (tentative)

63. IDEA 2 Instruments Items (est.)Application Diagnostic Feedback 2.040 itemsComprehensive assessment of overall teaching effectiveness Learning Outcomes 2.019 itemsBest for courses where outcomes rather than teaching methods are the focus, such as clinical, practicum, and lab Teaching Essentials12 itemsProvides a screening of fundamentals of teaching for quick review of faculty performance

64. Students View of Feedback

65. Faculty Views

66. Faculty Web Reports

67. Diagnostic Feedback Comprehensive assessment Thorough diagnostic Learning Outcomes Outcomes based Not diagnostic Teaching Essentials Short Basic screen Instant Feedback Immediate Base-level assessment

68. If you love something let it go… Yutaka Tsutano

69. From Evaluation to Engagement (Response Rates) Create Value for Students – Let them know their input is valuable and actually used to make changes Utilize In-Class Capture – Take 20 minutes in class to administer survey – Now easier online with mobile administration – Prepare students by putting it in syllabus Monitor and Communicate – Use real-time reports of response rates – Utilize effective follow-up emails

70. Selecting Objectives 3-5 as “Essential” or “Important” 1.Is it a significant part of the course? 2.Do you do something specific to help students accomplish the objective? 3.Does the student’s progress on the objective influence his or her grade? Be true to your course

71. FIF: Getting Started

72. FIF: Selecting Objectives

73. For Further Study

74. Final questions/thoughts? THANK YOU!