1. Using IDEA for Assessment, Program Review, and SACS University of Alabama Birmingham September 11, 2012 Shelley A. Chapman, PhD

2. Plan for this Session Program Evaluation & Assessment of Student Learning Group Summary Reports Aggregate Data File Benchmarking Reports Accreditation Guides

3. What makes IDEA unique? 1.Focus on Student Learning 2.Focus on Instructor’s Purpose 3.Adjustments for Extraneous Influences 4.Validity and Reliability 5.Comparison Data 6.Flexibility

4. Student Learning Model: 2 Assumptions Assumption 1: Types of learning must reflect the instructor’s purpose.

5. Student Diagnostic Form Assumption 2: Effectiveness determined by students’ progress on objectives stressed by instructor

6. Diagnostic Report Overview  Page 1 – Big Picture  How did I do? Page 3 – Diagnostic  What can I do differently? Page 2 – Learning Details  What did students learn? Page 4 – Statistical Detail  Any additional insights?

7. Your Average (5-point Scale) RawAdj. A.Progress on Relevant Objectives 1 Four objectives were selected as relevant (Important or Essential—see page 2) 4.14.3 1 If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average. The Big Picture

8. Progress On Relevant Objectives 4 4.3 + 4.3 4.1 4.2 3.6 5

9. Summary Evaluation: Five-Point Scale Report Page 1 Your Average Score (5-point scale) RawAdj. A.Progress on Relevant Objectives Four objectives were selected as relevant (Important or Essential—see page 2) 4.14.3 Overall Ratings B. Excellent Teacher4.74.9 C. Excellent Course4.14.4 D. Average of B & C4.44.7 Summary Evaluation (Average of A & D)4.34.5 50% 25%

10. Individual Reports to Group Reports

11. The Group Summary Report How did we do? How might we improve?

12. Defining Group Summary Reports (GSRs) Institutional Departmental Service/Introductory Courses Major Field Courses General Education Program

13. GSRs Help Address Questions Longitudinal Contextual Curricular Pedagogical Student Learning- focused

14. Adding Questions Up to 20 Questions can be added Institutional Departmental Course-based All of the above

15. Local Code Use this section of the FIF to code types of data.

16. Defining Group Summary Reports Local Code 8 possible fields Example: Column one – Delivery Format 1=Self-paced 2=Lecture 3=Studio 4=Lab 5=Seminar 6=Online Example from Benedictine University

17. Example Using Local code Assign Local Code 1=Day, Tenured 2=Evening, Tenured 3=Day, Tenure Track 4=Evening, Tenure Track 5=Day, Adjunct 6=Evening, Adjunct Request Reports All Day Classes Local Code=1, 3, & 5 All Evening Classes Local Code=2, 4, & 6 Courses Taught by Adjuncts Local Code=5 & 6

18. Description of Courses Included in this Report Number of Classes Included Diagnostic From 42 Short Form 27 Total 69 Number of Excluded Classes 0 Response Rate Classes below 65% Response Rate 2 Average Response Rate 85% Class Size Average Class Size 20 Page 1 of GSR

19. UAB Spring 2012 Page 1 of GSR

20. Assessment of Learning What are our faculty emphasizing? How do students rate their learning? How do our courses compare with others? How do our students compare with others (self- rated characteristics)? What efforts can we make for improvement? (How can we “close the loop”?)

21. UAB Core Competencies Shared Vision for a UAB Graduate Possible IDEA Learning Objectives Communication8, 11 Knowledge1, 2, 4, 7 Problem Solving3, 9, 11 Citizenship10, 12, extra questions

22. Are we targeting “Core Competencies” in the Core Curriculum? UAB Core Curriculum Courses 123456789101112 EH 101 English CompositionXXXX BY 101 Topics in Contemporary Biology XXXX MA 105 Pre-Calculus AlgebraXXX ARH 101 The Art ExperienceXXX HY 101 Western Civilization IXXXX PY 101 Introduction to Psychology XXXX IDEA Learning Objectives

23. Are we targeting “Core Competencies” in the Core Curriculum? UAB Core Curriculum Courses 123456789101112 EH 101 English CompositionXXXX BY 101 Topics in Contemporary Biology XXXX MA 105 Pre-Calculus AlgebraXXX ARH 101 The Art ExperienceXX HY 101 Western Civilization IXXXX PY 101 Introduction to Psychology XXX IDEA Learning Objectives

24. What are We Emphasizing? Percent of Classes Selecting Obj. as Important or Essential This GroupInstitutionIDEA System Objective 116%70%78% Objective 213%59%75% Objective 341%58%75% Objective 432%35%55% Objective 523%19%32% Objective 632%14%25% Objective 722%27% Objective 878%43%47% Objective 919%23%41% Objective 107%11%23% Objective 1168%42%49% Objective 1220%23%41% Average # of Obj. Selected3.74.25.7 Page 2

25. UAB Spring 2012

26. What are We Emphasizing? Page 9 Section B Number Rating Percent indicating amount required None or Little SomeMuch Writing662%17%82% Oral Communication666%42%52% Computer Application6650%44%6% Group Work6627%59%14% Mathematics/Quantitative Work 6597%3%0% Critical Thinking660%30%70% Creative/Artistic/Design6661%33%6%

27. How Did Students Rate their Learning on Core Competencies?

28. Do Students’ report of learning meet our expectations? Pages 5 and 6 Raw Average Adj. Average # of Classes This Report 3.9 11 Institution 4.2 3,963 IDEA System 4.0 31,991 Objective 1: Gaining factual knowledge (terminology, classifications, methods, trends )

29. How do students rate their learning? Page 3 Part 1: Distribution of Converted Scores Compared to the IDEA Database

30. Overall Progress Ratings (Courses) Page 3 Percent of Classes at or Above the IDEA database Average

31. Overall Progress Ratings (Courses) Part 3: Percent of Classes at or Above This Institution’s Average Page 4

32. Which teaching methods might we use to improve learning? Page 7 Teaching Methods and Styles Stimulating Student Interest# ClassesAv.s.d. 15. Inspired students to set and achieve goals which really challenged them 423.80.5

33. Assessing the QEP for UAB UAB’s Core Competencies IDEA Learning GoalsIDEA Teaching Methods Associated with Progress on Learning Goal Communication 8 Developing skill in expressing myself orally or in writing 11 Learning to analyze and critically evaluate ideas, arguments, and points of view 2, 7, 8, 15, 16, 18, 19 Knowledge 1 Gaining factual knowledge (terminology, classifications, methods, trends) 2 Learning fundamental principles, generalizations, or theories 4 Developing specific skills, competencies, and points of view needed by professionals in the field most closely related to this field 7 Gaining a broader understanding and appreciation of intellectual and cultural activity (music, science, literature, etc.) 1, 2, 4, 6, 7, 8, 10, 12, 13, 15, 19 Problem solving 3 Learning to apply course material (to improve thinking, problem solving, and decisions) 9 Learning how to find and use resources for answering questions or solving problems 11 Learning to analyze and critically evaluate ideas, arguments, and points of view 2, 7, 8, 9, 13, 14, 15, 18, 19 Citizenship 10 Developing a clearer understanding of, and commitment to, personal values 12 Acquiring an interest in learning more by asking my own questions and seeking answers 1, 2, 4, 6, 7, 8, 11,13, 15, 16, 18,

34. Relationship of Learning Objectives to Teaching Methods

35. How do students view course work demands? Page 8B Student Ratings of Course Characteristics Diagnostic Form Item # & ItemAverage% Classes Below 3.0 % Classes 4.0 or Above 33. Amount of Reading This Report3.421%24% Institution3.331%19% IDEA System3.233%15% 34. Amount of work in other (non-reading) assignments This Report3.324%10% Institution3.423%20% IDEA System3.421%18% 35. Difficulty of subject matter This Report3.219%0% Institution3.513%19% IDEA System3.420%18%

36. Aggregate Data File Allows you to Use Excel Spreadsheet Use with SAS or SPSS Ask other types of questions Display data in different ways

42. Instructors’ Reports on Course Emphases: Selected Pairings-Writing and Oral Communication

43. Instructors’ Reports on Course Emphases: Selected Pairings-Critical Thinking & Writing

44. Highlights for Sample U Remarkably similar profiles across terms Overall response rates ranged from 66% to 80% 1 st term in which administration was primarily online achieved a 75% response rate Transition from paper to online (fall 2009 to fall 2010) does not show major differences in profiles Sample U faculty focus on 4-5 outcomes as essential/important Over the last 3 terms, a significant increase on several objectives has been observed: application of course material, oral and written communication skills, & analysis and critical thinking skills (objectives 3, 8, & 11, respectively)

45. Benchmarking Institutional and Discipline Reports

46. Benchmarking Reports Comparison to 6-10 Peers Same Carnegie Classification IDEA database

47. Comparison Groups Your University ---------- ---------- Peer* ---------------------------- Carnegie ---------- ---------- National ---------- ---------- * Peer group is based on 6-10 institutions identified by your institution

48. Benchmarking Reports The student, rather than the class, is the unit of analysis Percentage of positive ratings is given rather than averages

49. Response Rates Your University: Student participation is similar to that of each comparison group Your University=79% Peer=77% Carnegie=79% National=75%

50. Students’ Perceptions

51. Graduate Students

52. Instructional Objectives Selected by Instructors Instructors’ Intentions/ focus Students’ Self-Reported Progress on Learning

53. IDEA Objective 3 Learning to apply course material (to improve thinking, problem solving, and decisions)

54. IDEA Objective 8 Developing skill in expressing oneself orally or in writing

55. Teaching Methods and Styles Reported by Students (Diagnostic Forms Only) Fostering Student Collaboration Encouraging Student Involvement

56. Benchmarking Look at “General Education”

58. Using Aggregate Data for Assessment UAB Core Competencies Program Learning Outcomes Group Summary Report Group Summary Report, Include Extra Questions Benchmarking: One Year or 3-5 Year Trend Report Benchmarking: Discipline Report Course Learning Outcomes Course Learning Outcomes Course Learning Outcomes Course Learning Outcomes

59. Accreditation Guides SACS NCATE CACREP

60. Questions?