1. VOLUNTARY SYSTEM OF ACCOUNTABILITY AND LEARNING OUTCOMES: AN UPDATE Teri Hinds Voluntary System of Accountability Natasha Jankowski National Institute for Learning Outcomes Assessment October 30, 2012 Assessment Institute

2. Overview  NILOA’s work with Transparency  Internal and External Communication  Voluntary System of Accountability  Evaluation of VSA College Portrait Pilot  Future of Student Learning Outcomes on College Portrait  VSA as a communication tool

3. The Heat is On  Unprecedented public challenges for higher education to be more accountable for results.  Accompanying external demands for information about student and institutional performance are growing calls for institutions and accreditors to become more transparent about what they do and the results they achieve. (Ewell, 2010)

4. Overview of NILOA NILOA’s mission is to document student learning outcomes assessment work, identify and disseminate best practices, and support institutions in their assessment efforts. S URVEYS ● W EB S CANS ● C ASE S TUDIES ● F OCUS G ROUPS ● O CCASIONAL P APERS ● W EBSITE ● R ESOURCES ● N EWSLETTER ● L ISTSERV ● P RESENTATIONS ● T RANSPARENCY F RAMEWORK ● F EATURED W EBSITES ● A CCREDITATION R ESOURCES ● A SSESSMENT E VENT C ALENDAR ● A SSESSMENT N EWS ● M EASURING Q UALITY I NVENTORY ● P OLICY A NALYSIS ● E NVIRONMENTAL S CAN ● Degree Qualifications Profile www.learningoutcomesassessment.org

5. www.learningoutcomeassessment.org

6. Reports on Transparency

7. Question  What information does your institution share publically on its website?  How are results of student learning shared internally?

8. Transparency Framework http://www.learningoutcomesassessment.org/TransparencyFramework.htm

10. Transparent Communication  Internal communication  External communication  Telling an institutional story about student learning

11. Questions  What types of stories does your institution tell about its students?  What types of stories does your institution tell about how it assures quality?

12. Background  VSA launched in 2007  Partnership between APLU and AASCU  320 participating institutions  Three original objectives  Demonstrate accountability and transparency  College search tool  Support innovation in the measurement of student learning outcomes

13. Student Learning Outcomes Pilot  Student learning outcomes page includes link to institution specific learning outcomes data;  Selection of the CLA, CAAP or ETS Proficiency Profile to measure value-added learning gains  Report results no more than four years after joining  Link to information on test administration, sample and response rate  Pilot period ends 12/31/12  What was learned from the pilot?

14. Evaluate the Effectiveness of SLO Pilot  Focus groups  Interviews  Surveys  Google Analytics  College Portrait statistics

15. Findings related to Transparency  Both VSA-participating and nonparticipating institutions agree that VSA in 2007 was a wise, timely, useful, and necessary response to the accountability and transparency demands of the time  Yet, nearly half of the participants have not yet met expectations of College Portrait student learning outcomes pilot  45% of participating institutions have yet to post student learning outcomes information  Information that was provided on student learning outcomes lacked explanatory or contextual material

16.  82% of participating institutions had traffic on their student learning pages but almost 1 in 5 did not have a single visitor to their student learning outcomes page since 2009.

17.  Information posted on the College Portrait may not reflect the needs of prospective students and families or provide the information they actually seek to make decisions about where to attend college.  The authorized standardized test measures of student learning outcomes lack broad credibility and acceptance in the higher education community – it is thought the tests do not accurately reflect student performance within the institution.

18.  Concerns about displaying results include:  Inability to understand what scores represent  Questions regarding what tests measure  Issues of reliability and validity  Difficulty in using scores to improve student learning  Motivation of students & difficulty to obtain a sample  Questions regarding quality of results  81% of survey respondents stated that the pilot requirements did not align with their assessment practices and that the results were not usable for campus improvement efforts

19.  Participating institutions agree - expanding the number & nature of student learning measures will:  produce more accurate portrayals of student attainment,  provide more useful information for campuses,  make information more meaningful for all audiences.

20. Recommendations  Focus on specific audiences and communicating meaningful information to them  Contextualize information  Expand range of assessment tools and approaches  Increase traffic to site  Include examples of use of evidence to improve learning  Consider creating template which could be used by all postsecondary institutions

21. Working Group Deliberations  Technical work group considered alternative measures of learning outcomes to recommend to VSA Board  Confirmed importance of student learning outcomes reporting within the VSA  No perfect of measure of students learning exists for all audiences  External accountability  Institutional improvement  College selection

22. Working Group Deliberations  Make value-add measurement using existing measures (CAP, CLA, ETS Proficiency Profiler) optional  Introduce option to use VALUE Rubrics based on AAC&U essential learning outcomes (written communication and critical thinking)  Introduce option to use aggregate scores from professional and graduate admissions exams such as the GRE, GMAT, LSAT, and MCAT

23. Work Group Ratings of Recommended VSA Instruments in Key Areas Instrument 1= lowest rating 10=highest rating CLACAAP ETS PP (short form) ETS PP (long form) VALUE Rubrics GRE Allow comparisons, benchmarking across institutions? 68 9.5 67 Offer a representative sample? 77777 2 Usefulness to faculty/students for learning improvement? inst- 2 class-7 5 229 3 Transparency of method, results 6888 9 8 External accountability 8 77755.5 Ease of administration on campus 5676 49.9 Reasonable costs (time, resources) 5776 startup - 3 marginal- 8 9 Motivation for students to do well? 333399.9 Likely interest to consumers? 2222 56 Valid learning outcomes instrument for average student? 5.5666 portfolios - 9 assign -7 3

24. VSA Board Decisions  College Portrait re-focus from “college selection tool for prospective students” to “consumer information tool for institutions to demonstrate effectiveness of educational programs.”  Primary audiences: state officials, policy makers, consumers  Instrument options:  CAAP, CLA, ETS Proficiency Profile  AAC&U VALUE rubrics – critical thinking, written communication  GRE General Test  Reporting options: value-added, benchmarking

25. Implementation/Next Steps  All VSA institutions report results of pilot by end of 2012 to fulfill reporting requirement  3 part report with tabbed display Did you participate (or try to)? If so, how’d it go? If not, why not? Give an example of another institutional-level evaluation Which of the new options are you considering for the future?  Reporting Format Preview: http://www.collegeportraits.org/slo_preview.html http://www.collegeportraits.org/slo_preview.html

26. Implementation/Next Steps  Communications Task Force working with College Portrait Support Team on display of new options  Emphasis on more graphical displays  Additional guidance on what the scores and results mean  Click-through links for additional details, e.g., subscores  New options released for 2012-13 data cycle (Jan 2013)  Develop outreach strategy to recruit new participants and connect with key audiences – state officials, accreditors

27. Websites www.voluntarysystem.org www.collegeportraits.org www.learningoutcomesassessment.org Teri Hinds: thinds@aplu.org Natasha Jankowski: njankow2@illinois.edu

28. Let’s Discuss.