1. Interpreting IDEA reports Diagnostic Form Short Form www.lvc.edu/idea

2. Page 1 of report

4. Guide says: “Report Responds to These Questions” 1. Overall, how effectively was this class taught? 2. How does this compare with the ratings of other teachers? 3. Were you more successful in facilitating progress on some class objectives than on others? 4. How can instruction be made more effective?* 5. Do some salient characteristics of this class and its students have implications for instruction? * Diagnostic Form Report only.

5. Our Modest Ambitions Today Understand summative scores – Raw averages – Converted scores – Adjusted scores – How used at LVC Formative use of p.2 – relevant objectives – POD papers on objectives Formative use of p.3 for Diagnostic only – POD papers on strategies

6. Summative scores ….

7. How “Summary Evaluation” Computed 41. Overall, I rate this instructor an excellent teacher. (25%) 42. Overall, I rate this course as excellent. (25%) Objectives 1-12 (21.-32) that are ranked as Important or Essential (50%)

8. “Converted averages” Transform group’s mean to 50, s.d. to 10. Examples: – Score 57 means.7 group s.d.’s above average – That’s at 75.8 %-tile of Normal Distribution – … but these groups of IDEA scores not Normal

9. Adjusted Scores: Effects that are removed P. 2 of Diagnostic Form report:

10. Adjusted Scores: Effects that are removed P. 2 of Diagnostic Form report: Only these on Short Form

11. Is unadjusted Score on Progress toward Objectives high? (≥4.2) Are adjusted scores lower or higher than unadjusted scores? Use adjusted scores Use unadjusted (raw) scores Lower Yes Higher When to Use Adjusted Scores for Administrative Decisions: Where is there evidence of effective teaching? (old) *Expectations defined by your unit. No

12. Do unadjusted (raw) scores meet or exceed expectations?* Are adjusted scores lower or higher than unadjusted scores? Use adjusted scores Use unadjusted (raw) scores Lower Yes Higher When to Use Adjusted Scores for Administrative Decisions: Where is there evidence of effective teaching? (new) *Expectations defined by your unit. No

13. College Use for Evaluating Faculty Current: – Summative: Pay attention to extremes – Assess formative use of IDEA by instructor IDEA recommendation – Summative: Establish standards Faculty set standards Department level or College level Assess progress in meeting them

14. Page 2: Objectives

15. P. 3 of Diagnostic Form Report: Directions…. www.theideacenter.org/podidea Follow the links…

16. Teaching Methods Relevant to Target Objectives

18. Try it 1. Identify objective… 2. Find related teaching method… 3. Follow link: www.theideacenter.org/podideawww.theideacenter.org/podidea

19. Last page: Raw response frequencies … especially additional questions We can retrieve the additional questions so you know which are which.

20. Effects of Evals being On-Line? Study of on-line vs paper by IDEA published in 2010: – http://www.theideacenter.org/sites/default/files/Tech nical Report 16.pdf http://www.theideacenter.org/sites/default/files/Tech nical Report 16.pdf Online avg response rate 57% – LVC response rates 70-75% No significant change in means or s.d.’s of responses – If loss of middle group, s.d. would go up. – On 12 objectives, s.d.’s slightly lower for on-line!

21. Key questions for LVC (from IDEA) How much emphasis will you give student ratings in the evaluation of teaching? How many classes need to be rated? [LVC: all] Performance categories: How many should you use? Do you plan to authenticate objectives? Will you weight the importance of items or groups of items in the decision-making process? If so, how? Will you use averages on a five-point scale or converted averages? Will you use adjusted or unadjusted scores?

22. References www.lvc.edu/idea – Campus IDEA page www.lvc.edu/idea POD notes: – The 20 methods: www.theideacenter.org/podideawww.theideacenter.org/podidea – The 12 objectives: www.theideacenter.org/podidea/PODNotesLearning.html www.theideacenter.org/podidea/PODNotesLearning.html IDEA: www.theideacenter.orgwww.theideacenter.org Technical: http://www.theideacenter.org/sites/default/files/ techreport-12.pdf - Data about adjusting, etc. http://www.theideacenter.org/sites/default/files/ techreport-12.pdf