A Paradigm Shift in the Assessment of Learning Outcomes Tara Rose Director of University Assessment University of Kentucky Jeanne Mullaney Assessment Coordinator and Professor Community College of Rhode Island Michael Ben-Avie Director of the Office of Assessment and Planning Southern Connecticut State University Moderated by: Teddi Safman Assistant Commissioner for Academic Affairs Utah System of Higher Education

Multi-State Collaborative (MSC) to Advance Learning Outcomes Assessment Pilot Study Year (2015) & Demonstration Year (2016) Course Embedded Authentic Student Work Faculty Engagement Model for Assessing Student Learning Data without reliance on standardized tests Across Institutions and States Critical Thinking, Quantitative Literacy, Written Communication

Taking the Vision to Scale in Thirteen States CT, HI, IN, KY, MA, ME, MO, MN, OR, RI, TX, UT, VA Participants: Steering Committee: Point person from each state and reps from SHEEO & AAC&U Institution Point Persons: From each campus in each state

Pilot Study Year (2015)Demonstration Year (2016) States Involved 9 Connecticut, Indiana, Kentucky, Massachusetts, Minnesota, Missouri, Oregon, Rhode Island, Utah 12 Connecticut, Indiana, Kentucky, Massachusetts, Minnesota, Missouri, Oregon, Rhode Island, Utah + Hawaii, Maine, Texas Public Institutions four-year (8 research) 31 two-year four-year (11 research) 34 two-year Rubrics Critical Thinking, Quantitative Literacy, Written Communication Critical Thinking, Quantitative Literacy, Written Communication + Civic Engagement (optional)

MSC by the Numbers Comparison Pilot Study Year (2015) Demonstration Year (2016) Pieces of student work submitted (approximates # of students) 7,21510,948 Students had to be 75% of the way to completion of institutional degree requirements Pieces of work scored twice in order to measure inter-rater reliability 2,642 (36%) 3,031 (28%) Number of assignments (approximates # of faculty participants) 1,1661,573 Number of faculty scorers > 150> 190

Institutional Perspectives

The University of Kentucky Large, Public 4-Year Institution Doctoral University 17 Colleges & Professional Schools 360+ Programs 13,000+ full time employees 2,400+ full time faculty 30,000+ students Accredited by SACSCOC

The UK Perspective: MSC Goals Determine if student demonstration of learning meet UK faculty expectations Measure WC and QL at UK as a whole and at course level, and also in comparison to MSC 4 & 2 year institutions. Measure growth in performance from freshman to senior Develop reliable and cost effective sustainable scoring protocols for student learning assessment.

The UK Perspective: Benefits Benefits Benefits Gauge the level of student learning occurring at UK by outcome area Comparison data by using authentic student work Encourages faculty engagement at multiple levels Participant, External Scorer, Internal Scorer

The UK Perspective: Challenges Challenges Challenges Recruiting Faculty with no ‘real’ benefit IRB Paperwork Collection of Artifacts Trusting the Process/Transparency

UK’s Approach to the MSC Closing the Loop Implementation Planning UK Team IRB Paperwork Faculty Recruiting Collection of Artifacts Pulling Demographic Data Redacting/Coding Artifacts University Reporting (External/Internal) Faculty Reporting and Consultations

The UK Perspective: Reflections Faculty Flexibility was key Levels of Engagement Reporting and Feedback Connection to Internal Assessment Processes Institutional Support Upper Administration Support Campus Awareness Marketing Process and Technology Oversample Simple Scoring Raw Data Files Use of Aqua for multiple initiatives: MSC, UK Core, QEP, UAC

The Community College of Rhode Island New England’s Largest, Public 2-Year Institution 17,000 students on 4 campuses and 2 satellites Associate’s College 100 degree and certificate programs 330 Full-time Unionized Faculty Accredited by NEASC

The CCRI Perspective: MSC Goals MSC GOALS Assess the CCRI Definition of An Educated Person Respond to NEASC standards about assessing general education Promote the faculty role in assessing student learning Develop and augment faculty knowledge of assessment

The CCRI Perspective: Benefits Benefits Benefits Prestige of national project Grant incentive for faculty Collaboration with colleagues Faculty engagement focus Authentic faculty and student work

The CCRI Perspective: Challenges Challenges Challenges Sampling Process Students with 45 credits found in dozens of classes Faculty engagement focus Collecting artifacts per outcome Only specific kinds of artifacts accepted

Faculty Learning Community 9 faculty from different disciplines Worked with a total of 36 colleagues Collected artifacts and checked files Reassured colleagues regarding anonymity CCRI’s Approach to the MSC

The CCRI Perspective: Reflections Project overall more effective than anticipated Faculty cooperation and engagement Eagerness to learn results and participate in session to brainstorm how to respond to them Enthusiasm around learning how to score and participating in scoring process Representative sample Taskstream system Easy to store and retrieve artifacts Convenience for scorers Reports available Streamline the file preparation

Southern Connecticut State University

Impact of MSC After the pilot, faculty members approached me and asked if we could collect the papers of sophomores and score them in a process similar to MSC. This was the first time that faculty initiated an assessment project.

Talking about MSC Distributed exemplary papers MSC day at the Teaching Academy Small group “parlor meetings” Predictive models of graduation outcomes

MSC SWAT Team Connecticut is the only state that established an “MSC SWAT Team” that went to each campus to provide technical assistance. The team became familiar with the experiences of both two-year community colleges and four-year universities as they engaged in the initiative. The team was led by an undergraduate math student who now works at SHEEO.

A project and not a strategy After the first year, the directors and coordinators of assessment met with the Provost of Connecticut’s Board of Regents. We told her that MSC felt like a project and not a strategy. What could we do to turn it into a strategy?

SLO Assessment Advisory Working Group In response, the Provost established the Student Learning Outcomes Assessment Advisory Working Group. The goal of this group is to promote the replication of best practices of student learning outcomes assessment by Connecticut State Colleges & Universities, primarily to: (1) demonstratively improve the quality of teaching and learning and, (2) document the educational effectiveness of academic programs.

Assessment Summit To set the foundation, the Student Learning Outcomes Assessment Advisory Working Group convened an assessment summit on May 6 th with a keynote address by Dr. Barbara Brittingham, President of the Commission on Institution of Higher Education. The planning team for the assessment summit originally expected 50 participants; 200 showed up.

Assessment Council The newly-formed Assessment Council will meet on a regular basis. Working in tandem with faculty, the Assessment Council will work towards common definitions of criteria for student learning outcomes among the campuses. The Assessment Council will also provide a forum for sharing ideas and concerns. The Assessment Council will meet at least quarterly, and more frequently if needed.

NEASC Accreditation We are currently engaged in writing our accreditation report in line with the new standards. MSC figures prominently.