Writing Measurable Student Learning Outcomes Dr. Timothy S. Brophy Director of Institutional Assessment University of Florida

Today’s Goals Describe and explain SACS accreditation expectations for academic program assessment Identify and apply steps for developing measurable student learning outcomes Develop and/or refine student learning outcomes for your degree program

What is SACS and Why is it Important to be Accredited? SACS-COC = the Southern Association of Colleges and Schools’ Commission on Colleges SACS is the Federally-approved accrediting body for southern region of the US SACS develops policies and standards that operationalize Federal Regulations Federal Student Aid is tied to our reaffirmation – without accreditation we lose this important funding source

Three Most Recently Available Years Selected Student Financial Aid Data for University of Florida Undergraduate Students for the Three Most Recently Available Years Academic Year 2008-09 2009-10 2010-11 All undergraduate students Pell Grants $ 30,894,352 $ 41,996,944 $ 46,733,613 Federal Loans $ 60,969,318 $ 59,795,534 $ 59,470,923 Full-time, first-time degree/certificate-seeking undergraduate students Federal Grants $ 6,512,300 $ 10,400,395 $ 11,826,734 $ 5,471,084 $ 8,097,580 $ 9,014,885 Other Federal Grants $ 1,041,216 $ 2,302,815 $ 2,811,849 $ 6,651,786 $ 7,457,421 $ 7,362,376 Source: IPEDS Student Financial Aid Component $106,204,536

Historical Context THE TRANSITION: NOW: THEN: 2003 Self Study Volume 1: Six required sections Volume 2: International Focus report THE TRANSITION: 2003-present Evolving Standards and Expectations Paradigm shift NOW: Data–driven Continuous Improvement Compliance Quality Enhancement Student Learning 2003: basically a series of tables – standard in left column, brief narrative in center column, links to data in the right column 2013: Online, webbased report with narratives and links to extensive data to show long term improvement efforts Introduction Letter from President Charles Young Preface Table of Contents Members of Self-Study Committees Organizational Charts Section 1: Principles and Philosophy of Accreditation Institutional Commitment and Responsibilities in the Accreditation Process Application of the Criteria Separately Accredited Units Conditions of Eligibility Initial Membership Representation of Status Section 2: Institutional Purpose Institutional Purpose Section 3: Institutional Effectiveness Planning and Evaluation: Educational Programs Planning and Evaluation: Administrative and Educational Support Services Institutional Research Section 4: Educational Program General Requirements of the Educational Program Undergraduate Program Graduate Program Publications Distance Learning Programs Continuing Education, Outreach and Service Programs Student Records Faculty Consortial Relationships and Contractual Relationships Section 5: Educational Support Services Library and Other Learning Resources/Purpose and Scope Instructional Support Information Technology Resources and Systems Student Development Services Intercollegiate Athletics Section 6: Administrative Processes Organization and Adminstration Institutional Advancement Financial Resources Physical Resources Externally Funded Grants and Contracts Related Corporate Entities

Scope of the SACS Principles 12 Core Requirements 35 subcomponents 14 Comprehensive Standards 73 subcomponents 9 Federal Requirements

SACS Standard 3.3.1.1 3.3.1 - The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional effectiveness) 3.3.1.1 educational programs, to include student learning outcomes

What SACS Expects Completion of at least one complete SLO assessment cycle Documented evidence that the “improvement cycle” is complete Evidence and documentation of compliance with all requirements and standards

The UF Assessment Process Establish Goals and Outcomes Assessment Planning Implement the Plan and Gather Data Interpret and Evaluate the Data Modify and Improve UF MISSION Institutional Effectiveness Assessment

Academic Assessment Plan Mission Alignment Student Learning Outcomes Curriculum Maps (UG)/ Assessment Timelines (Grad/Prof) Assessment Cycle Methods and Procedures Assessment Oversight Hand out the AAP and SLO guides

Student Learning Outcomes at UF March 29, 2007 – Board of Governors regulation 8.016, “Academic Learning Compacts” “Research indicates that university students are served best when students and faculty fully engage in a teaching-learning partnership, and this partnership is all the more meaningful if it is made as clear as possible to students what it is they will learn and how program faculty will assess that learning. Therefore, the Board has determined that universities must develop “Academic Learning Compacts” and related assessment processes to define and demonstrate student achievement in baccalaureate degree programs in the State University System.” BOG Regulation 8.016, revised 1-19-12, “Student Learning Outcomes Assessment”

Identifying SLOs and Program Goals

Definitions Student Learning Outcomes (SLOs)describe student learning – what students will know and be able to do as a result of completing a UF program Program Goals do not describe student learning – instead, they describe programmatic elements, such as admission criteria, acceptance and graduation rates, etc

Outputs or outcomes?

definitions Outputs describe and count what we do and whom we reach, and represent products or services we produce. Processes deliver outputs; what is produced at the end of a process is an output. An outcome is a level of performance or achievement. It may be associated with a process or its output. Outcomes imply measurement - quantification - of performance.

Outcomes and outputs: What is the difference? This distinction is important, especially in the development and review of Student Learning Outcomes. We seek to measure outcomes as well as their associated outputs; however, SLOs focus on outcomes. For example, while we produce a number of new graduates (the output), it is critical that we have a measure of the quality of the graduates as defined by the college or discipline (the outcome). Effective Student Learning Outcomes describe, in measurable terms, these quality characteristics by defining our expectations for knowledge, critical thinking, and communication for UF undergraduates, and knowledge, skills, and professional behaviors for graduate and professional students,.

characteristics of effective student learning outcomes Focus on what students will know and be able to do. All disciplines have a body of core knowledge that students must learn to be successful as well as a core set of applications of that knowledge in professional settings. Describe observable and measureable actions or behaviors. Effective SLOs present a core set of observable, measureable behaviors. Measurement tools vary from quizzes and tests to complex rubrics. The key to measurability: an active verb that describes a observable behavior, process, or product A framework for developing SLOs: Bloom’s Taxonomy (see Table 3 in your Student Learning Outcomes handout)

Verbs and phrases that complicate measurability Understand An internal process that is indicated by demonstrated behaviors – OK for ALCs but not recommended for program or course SLOs Appreciate; value Internal processes that are indicated by demonstrated behaviors closely tied to personal choice Become familiar with Focuses assessment on “becoming familiar,” not familiarity Learn about, think about Not observable; demonstrable through communication or other demonstration of learning Become aware of, gain an awareness of Focuses assessment on becoming and/or gaining – not actual awareness Demonstrate the ability to Focuses assessment on ability, not achievement or demonstration of a skill

Direct or Indirect assessment?

Definitions Direct assessments of student learning are those that provide for direct examination or observation of student knowledge or skills against measurable performance indicators. Indirect assessments are those that ascertain the opinion or self-report of the extent or value of learning experiences

Developing Measurable SLOs: A Three-level Model (carriveau, 2010) This model allows you to develop assessments that measure the outcomes, and that then connect directly to the program learning goals Course-level Student Learning Outcome these are determined by the faculty and specify course-level, observable products or demonstrations Program-level Student Learning Outcome these describe what students will do to demonstrate they have met the learning goals Program Learning Goal Level – programs establish learning goals for the degree these are described in the Academic Learning Compact, Program mission, or Catalog entry

Level 1: Establishing Learning Goals for the Degree Learning Goals – these are found in the Academic Learning Compact in the description of the major Example: Materials Science and Engineering The major enables you to develop an understanding of materials systems and their role in engineering. Emphasis is placed on the ability to apply knowledge of mathematics, science and engineering principles to materials science and engineering; to design and conduct experiments, as well as to analyze and interpret data; and to design a system, component or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability and sustainability. Source: 2012-13 UF Undergraduate Catalog, https://catalog.ufl.edu/ugrad/current/engineering/alc/materials-science-and-engineering.aspx

Level 1: Learning Goals based on the MSE ALC Students who complete the MSE degree will: Understand materials systems and their role in engineering Apply knowledge of mathematics, science and engineering principles to materials science and engineering to design and conduct experiments, as well as to analyze and interpret data Design a system, component or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability and sustainability

Level 2 – Program Student Learning Outcomes for MSE Content Knowledge Apply knowledge of mathematics, science and engineering principles to materials science and engineering. Design and conduct materials science and engineering experiments and analyze and interpret the data. Critical Thinking Design a materials science and engineering system, component or process to meet desired needs within realistic economic, environmental, social, political, ethical, health and safety, manufacturability and sustainability constraints. Communication Communicate technical data and design information effectively in speech and in writing to other materials engineers.

MSE: Connecting Goals to Outcomes ALC Learning Goals: Understand materials systems and their role in engineering Design a system, component or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability and sustainability Student Learning Outcomes: Design a materials science and engineering system, component or process to meet desired needs within realistic economic, environmental, social, political, ethical, health and safety, manufacturability and sustainability constraints. Communicate technical data and design information effectively in speech and in writing to other materials engineers Goal SLO

MSE: Connecting Goals to Outcomes ALC Learning Goals: Understand materials systems and their role in engineering Apply knowledge of mathematics, science and engineering principles to materials science and engineering to design and conduct experiments, as well as to analyze and interpret data Student Learning Outcomes: Apply knowledge of mathematics, science and engineering principles to materials science and engineering. Design and conduct materials science and engineering experiments and analyze and interpret the data Level 1 Level 2

Connecting Program SLOs to Courses MSE Curriculum Map   Additional Assess-ments Content Knowledge EMA3050 EMA3066 EMA4714 EMA3080C EMA3513C #1 I R A Senior exit survey #2 Critical Thinking EMA4223 #3 Communi-cation EMA3013C #4 Assessments in the boxes marked A are conducted using specific homework, exam, or assignment questions aligned with that SLO. Source: 2011-12 MSE Academic Assessment Plan

Level 3 – Course level SLOs These are determined by the faculty to teach the course However, these should directly relate to the program SLOs

Writing student learning outcomes (use with the UF Student Learning Outcome Guide) Review the Academic Learning Compact. List the learning goals for the program that are in the ALC. Review the current SLOs for your area with your program faculty for recency, relevance, and rigor. Examine the SLOs for the Knowledge Type (see Table 1) and Cognitive Processes level (see Table 2) they engage. The majority of the SLOs should be in the upper three levels of the Cognitive Processes Dimension – Analyze, Evaluate, and Create. The Taxonomy template in Figure 1 may help with this process. Cross-reference your SLOs with the list of verbs/actions associated with their corresponding cognitive dimension levels (see Table 3), and replace any “verbs and phrases to avoid” with appropriate verbs from Table 3. Write the SLO concisely and clearly.

Additional Resources Carriveau, R. (2010). Connecting the dots – Developing student learning outcomes and outcomes-based assessments. Denton, TX: Fancy Fox Publications Suskie, L. (2009). Assessing student learning: A common sense guide (2nd ed.) San Francisco, CA: Jossey-Bass. Walvoord, B. (2010). Assessment clear and simple: A practical guide for institutions, departments, and general education (2nd ed.). San Francisco, CA: Jossey-Bass. You can always find our UF-specific Institutional Assessment resources at our website, http://assessment.aa.ufl.edu/