1. The Basics of Student Learning Outcomes Assessment SACSCOC Summer Institute July 20, 2015 handout for this session is in the online program Timothy S. Brophy – Director and Professor, Institutional Assessment University of Florida – Office of the Provost Gainesville, Florida

2. Today’s Goals 1.To introduce, describe, and explain the basic elements of student learning outcomes, their development, and measurement 2.To share a structure for assessment planning and reporting for undergraduate, graduate, and professional programs 3.Analyze an example of an academic assessment plan, SLOs, measures, results, and use of results

3. Common Challenges Size and scope Multiple colleges/departments Undergraduate, Graduate, Professional, and certificate programs Institutional consistency Outcomes Assessment reporting Cycles Management and Tools Honoring unit autonomy, disciplinary distinctions, and institutional requirements Faculty comportment

4. SACS Standard 3.3.1.1 3.3.1 The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional effectiveness) 3.3.1.1 educational programs, to include student learning outcomes

5. Student Learning Outcomes

6. Basic Element 1: Define and Disseminate the Terms Student Learning Outcomes (SLOs) are defined generally as “what students are expected to know and be able to do by completion of their degree program” BASIC ELEMENT 1: define this for your faculty and ensure that this definition is consistent across campus and clearly posted

7. Basic Element 2: Consider a Categorical Organizing Framework Undergraduate Content/ Discipline Knowledge Critical Thinking Communication Graduate Content/ Discipline Knowledge Skills Professional Behavior

8. Basic Element 3: Recent, Relevant, and Rigorous Student Learning Outcomes reflect the curriculum the discipline, and faculty expectations; as these elements evolve, learning outcomes change. Recent – the outcome reflects current knowledge and practice in the discipline. Relevant – the outcome relates logically and significantly to the discipline and the degree. Rigorous – the degree of academic precision and thoroughness that the outcome requires to be met successfully.

9. Basic Element 4: Distinguish Outputs from Outcomes Outputs describe and count what we do and whom we reach, and represent products or services we produce. Processes deliver outputs; what is produced at the end of a process is an output. An outcome is a level of performance or achievement. It may be associated with a process or its output. Outcomes imply measurement – quantification of performance.

10. Outcomes and Outputs: What is the Difference? We seek to measure outcomes as well as their associated outputs; however, SLOs focus on outcomes. For example, while we produce a number of new graduates (the output), it is critical that we have a measure of the quality of the graduates as defined by the college or discipline (the outcome). Effective Student Learning Outcomes describe, in measurable terms, these quality characteristics by defining our expectations for students.

11. Exercise 1: Are these statements the results of outputs or outcomes?

12. Our program graduated 25 students in Spring 2011. A.Output B.Outcome

13. 80% of our students scored in the 95 th percentile on the state teacher examination. A.Output B.Outcome

14. We recruited 10 additional students in 2012-13. A.Output B.Outcome

15. 95% of our students achieved Level 4 (out of 5) on our presentation assessment rubric. A.Output B.Outcome

16. In 2011, Our doctoral students published 10 papers in The International Journal of Psychology. A.Output B.Outcome

17. Basic Element 5: Distinguish SLOs and Program Goals Student Learning Outcomes (SLOs) describe student learning – what students will know and be able to do as a result of completing an academic program (undergraduate, graduate, professional, and certificate). Program faculty set targets for their SLOs Program Goals describe the unit’s expectations for programmatic elements, such as admission criteria, acceptance and graduation rates, etc. – see p. 4 of your handout

18. Exercise 2: Student Learning Outcomes or Program Goals?

19. Students identify and describe major events in American history from 1770-1800. A.Program Goal B.Student Learning Outcome

20. We will lower the student attrition rate to 10%. A.Program Goal B.Student Learning Outcome

21. Students discriminate musical quality based on sound musical reasoning. A.Program Goal B.Student Learning Outcome

22. Students analyze experimental data and interpret results in the cellular and molecular sciences. A.Program Goal B.Student Learning Outcome

23. We will increase our 2011-12 acceptance rate of 35% to 40% in 2012-13. A.Program Goal B.Student Learning Outcome

24. Basic Element 6: Ensure the Outcome is Measurable Focus on what students will know and be able to do. All disciplines have a body of core knowledge that students must learn to be successful as well as a core set of applications of that knowledge in professional settings. Describe observable and measureable actions or behaviors. Effective SLOs present a core set of observable, measureable behaviors. Measurement tools vary from exams to complex tasks graded by rubrics. The key to measurability: an active verb that describes an observable behavior, process or product. A framework for developing SLOs: Bloom’s Taxonomy (pages 6-8 in your handout)Bloom’s Taxonomy Effective SLOs:

25. Verbs and Phrases that Complicate Measurability Understand An internal process that is indicated by demonstrated behaviors – OK for learning goals but not recommended for program or course SLOs Appreciate; Value Internal processes that are indicated by demonstrated behaviors closely tied to personal choice or preference; OK if the appreciation or valuing is supported by discipline-specific knowledge Become familiar with Focuses assessment on “becoming familiar,” not familiarity

26. Verbs and Phrases that Complicate Measurability Learn about, Think about Not observable; demonstrable through communication or other demonstration of learning Become aware of, Gain an awareness of Focuses assessment on becoming and/or gaining – not actual awareness Demonstrate the ability to Focuses assessment on ability, not achievement or demonstration of a skill

27. Developing Measurable SLOs: A Three- level Model (Ron Carriveau, Connecting the Dots, 2010, Fancy Fox Publications) This model connects course-level and program-level SLOs directly to the program learning goals Course-level – Student Learning Outcome these are determined by the faculty and specify course-level, observable products or demonstrations Program-level – Student Learning Outcome these describe what students will do to demonstrate they have met the learning goals Program Learning Goal Level – programs establish learning goals for the degree these goals require multiple actions over time to measure

28. Exercise 3: Are the Following Outcome Statements Measurable?

29. Students understand good writing style. A.Yes B.No

30. Students sight-sing a 16- measure melody with no errors. A.Yes B.No

31. Students appreciate 20 th century artworks. A.Yes B.No

32. Students apply design skills effectively to create a new office building design. A.Yes B.No

33. Students define the ethical responsibilities of business organizations and identify relevant ethical issues. A.Yes B.No

34. Basic Element 7: Balance Direct and Indirect Assessments Direct assessments of student learning are those that provide for direct examination or observation of student knowledge or skills against measurable performance indicators. Indirect assessments are those that ascertain the opinion or self-report of the extent or value of learning experiences (Rogers, 2011)

35. Exercise 4: Direct or Indirect Assessments?

36. Course midterm exam. A.Direct B.Indirect

37. Student Experience in a Research University (SERU) or National Survey of Student Engagement (NSSE) survey data. A.Direct B.Indirect

38. Final paper graded by a faculty developed rubric. A.Direct B.Indirect

39. Final presentation or performance graded by a faculty developed rubric. A.Direct B.Indirect

40. Senior exit interview. A.Direct B.Indirect

41. Planning and Reporting

42. Academic Assessment Planning Academic Assessment Plans provide a common framework for units to plan how they assess and measure student achievement of the SLOs. Plans also present the process for how the data from these assessments are used to enhance the quality of student learning.

43. Why Plan? Provides faculty a focal point for the discussion of the assessment of student learning in the degree programs. Planning discussions provide an opportunity to revisit the curriculum and its relationship to the SLOs. Provides a consistent reference resource when faculty and leadership change.

44. Basic Element 8: Develop a Planning Timeline/Cycle Develop assessment plans Submit for institutional review Implement plan Collect data and submit reports

45. Basic Element 9: Develop Templates and Rubrics Academic Assessment Plan Mission Alignment Student Learning Outcomes Curriculum/ Assessment Maps Assessment Cycle Methods and Procedures Assessment Oversight

46. Management Tools Figure 1 – Assessment Plan Review Rubrics – pp. 12-14 Figure 2 – New SLO/Academic Assessment Plan Submission form – p. 15 Figure 3 – SLO/Academic Assessment Plan Revision form – p. 16

47. Basic Element 10: Develop an Approval and Management Process Program/ Department College Academic Assessment Committee University Curriculum Committee Student Academic Support System The University of Florida SLO Approval Process

48. Basic Element 11: Develop a System or Cycle of Assessment and Reporting February – Assessment Plans submitted for the next AY October - Assessment Data, results, and use of results for previous AY reported Assessment and Institutional Effectiveness Data Reporting Establish Mission, Goals, and Outcomes Assessment Planning Implement the Plan and Gather Data Interpret and Evaluate the Data Modify and Improve

49. Reporting Assessment Results Assessment Reporting Outcome Measure(s) Data Use of Results Program Modifications

50. A Sample Reporting Template (page 11) SLOsAssessment MethodResultsUse of Results

51. Basic Element 12: Develop a Quality Assurance Process

52. Elements of Quality Assurance Multi-step, institutional review and approval process Templates and rubrics for guiding faculty through the process Review and evaluate faculty submissions Cross-reference plans with data reported annually Design and provide professional development Model the process: Modify and improve quality assurance processes based on the data you collect

53. Group activities

54. Group Activity 1: Which SLO modification is best? 1. Review the sample SLO. 2. In your group, discuss how you would modify the SLO. Use pp. 4-9 of your handout as a guide. 3. Of three choices provided, which choice do you feel is the best modification and why?

55. Students understand good writing style. A.Students write using the appropriate style. B.Students describe the appropriate writing style. C.Students create papers using the appropriate writing style.

56. Students demonstrate that they know how to solve a physics problem. A.Students calculate physics problems correctly. B.Students solve physics problems accurately. C.Students know how to solve physics problems.

57. Students describe their understanding of Napoleon’s role in French history. A.Students understand Napoleon’s role in French history. B.Students know Napoleon’s role in French history. C.Students describe Napoleon’s role in French history.

58. Students create a musical composition within given guidelines. A.Students compose a piece within given guidelines. B.Students create music within given guidelines. C.Students compose music within given guidelines.

59. Group Activity 2: Revising SLOs At your table, select one of the samples for review. Review the program’s 2011-12 Student Learning Outcomes and Program Goals. Master of Laws in Comparative Law – pp. 7-8 & top of p. 9 BS, Animal Sciences – pp. 10-12 & top of p. 13 QUESTIONS FOR DISCUSSION: Are these measureable as written? If so, why? If not, how would you revise these?

60. Group Activity 3: SLO Analysis Using the same program, review the 2012-13 and 2013-14 Academic Assessment Plans for the SLOs, measurements, and data reports. QUESTIONS FOR DISCUSSION: Have the SLOs changed? Do the assessment reports follow the guidelines provided on p. 12-14? Do the assessment reports lead to program modifications? How would you advise these program faculty?

61. A Summary of Basic Elements 1.Define the terms and disseminate them 2.Consider an institutional categorical organizing framework for SLOs 3.Recent, Relevant, and Rigorous 4.Distinguish Outputs from Outcomes 5.Distinguish SLOs from Program Goals 6.Ensure the outcome is measurable

62. A Summary of Basic Elements 7.Balance direct and indirect assessments 8.Planning Timeline/Cycle 9.Templates and Rubrics 10.Approval and Management Process 11.Develop a System or Cycle of Assessment and Reporting 12.Quality Assurance Process

63. Questions Timothy S. Brophy, Ph.D. Professor and Director, Institutional Assessment tbrophy@aa.ufl.edu 352-273-4476