1. WHAT DO WE KNOW ABOUT THE QUALITY OF HIGHER LEARNING IN THE UNITED STATES? Marilee J. Bresciani, Ph.D. Professor of Postsecondary Education Leadership at San Diego State University October 4, 2010

2. WEBINAR OVERVIEW BRESCIANI, M.J., 2010  Context for Accountability of the Quality of General Learning within the United States  Exploring Comparability of Quality of General Learning  Positing Questions for Consideration

3. FRAMING QUESTIONS BRESCIANI, M.J., 2010  What do you know about the quality of general learning at your institution?  How does the quality of general learning at your institution compare to the quality of learning at other institutions?

4. Bresciani, M.J. CONTEXT FOR ACCOUNTABILITY (BRESCIANI, ET AL, 2009)  To improve the underperforming student  Competency Movement in Business and Industry  International Trade Agreements  Bologna Declaration of 1999

5. Bresciani, M.J. CONTEXT FOR ACCOUNTABILITY, CONT. (BRESCIANI, ET AL, 2009)  Government Conversation, 1985  The Higher Education Re-authorization Act Testimonies in USA, 2002 and 2006  Response to NCLB Legislation  Regional Accreditation – flexibility  CRAC – 2003, 2004  Both documents focus on student learning

6. Bresciani, M.J. CONTEXT FOR ACCOUNTABILITY, CONT. (BRESCIANI, ET AL, 2009) National Commission on the Future of Higher Education Demand for Public Information about Performance Transparency of outcomes and results Comparable measures of quality Demonstration of value-added of the entire educational experience

7. Bresciani, M.J. CONTEXT FOR ACCOUNTABILITY, CONT. (BRESCIANI, ET AL, 2009)  Accountability requirements handed back to states  Performance indicators  Increased desire for transparency, comparability, and value-added  Discipline Standards still expected to be addressed by disciplines

8. IN OTHER WORDS… (BRESCIANI, ET AL, 2009)  We are being held accountable for general learning expectations by the public.  The disciplines are “taking care” of accountability via professional accreditation/certification. Bresciani, M.J.

9. SO, THE QUESTIONS REMAIN… (BRESCIANI, ET AL, 2009)  So, how effective is our design and delivery of general learning?  How well equipped are we to evaluate that learning?  How well can we compare our quality of general learning with another institution? Bresciani, M.J.

10. REPORT OUT Bresciani, M.J.

11. TIPS FROM GOOD PRACTICE INSTITUTIONS (BRESCIANI, 2007)  They clearly communicate the purpose of the programs that deliver their general learning  There is cross institutional buy-in to the purpose  There is cross institutional buy-in to the outcomes and in some cases assessment methods, if appropriate

12. Bresciani, M.J. TIPS FROM GOOD PRACTICE INSTITUTIONS (BRESCIANI, 2007)  They clearly define their expected learning through learning outcomes  They align the process for creating the expected learning with the manner in which they evaluate the learning  They “advertise” the variances in their processes  They advertise the “success” of their general learning with evidence; they improve where they are not successful

13. Bresciani, M.J. TIPS, CONT. (BRESCIANI, 2007)  They invite peer critiques, internal to their organization and external to their organization  They use testimonials to demonstrate their effectiveness  Some articulate expectations of and investments in to students and faculty  Some are mindful of their ”inputs”

14. SECOND SET OF QUESTIONS (BRESCIANI, ET AL, 2009)  Who claims ownership for the design, deliver, and evaluation of your general learning?  How well does the assessment data inform improvements?  How well coordinated is the evaluation and decision-making process? Bresciani, M.J.

15. REPORT OUT Bresciani, M.J.

16. SUMMARY OF PROS AND CONS OF COMPARABLE DATA BRESCIANI, 2010 Standardized Tests  Commercially designed  Ability to update is limited  May be able to align with learning outcomes  May be able to align with delivery of learning  Generates validity and reliability statistics  Less subjective Rubrics  Faculty designed  Ability to update is dynamic  May be able to align with learning outcomes  May be able to align with delivery of learning  Generates inter-rater reliability statistics  Subjective

17. CONSIDERATIONS FOR USING RUBRICS FOR COMPARABILITY BRESCIANI, 2010  Can you answer the previously posited questions?  Can your faculty collaborate on the design of rubrics or can your faculty adapt those developed by the AAC&U VALUES project?  Can you collaborate with other institutions to share rubric results and run inter-rater reliability data?

18. SESSION CITATION  Bresciani, M.J. (October 4, 2010). What do We know About the Quality of Higher Learning in the United States? EDUCAUSE Learning Initiative Webinar

19. REFERENCES  Bresciani, M. J. (Ed.). (2007). Good practice case studies for assessing student learning in general education. San Francisco, CA: Jossey Bass.  Bresciani, M. J., Gardner, M. M., & Hickmott, J. (2009). Demonstrating student success in student affairs. Sterling, VA: Stylus Publishing.

20. RESOURCES  AAC&U Leap Report - http://www.aacu.org/leap/documents/GlobalCentury_Exec Sum_3.pdf  Raising the Bar: Employers' Views on College Learning in the Wake of the Economic Downturn" (January, 2010) - http://www.aacu.org/leap/documents/2009_EmployerSurve y.pdf http://www.aacu.org/leap/documents/2009_EmployerSurve y.pdf  "How Should Colleges Assess And Improve Student Learning? Employers' Views on the Accountability Challenge" (January 2008): http://www.aacu.org/leap/documents/2008_Business_Lead er_Poll.pdf http://www.aacu.org/leap/documents/2008_Business_Lead er_Poll.pdf

21. RESOURCES, CONT.  AAC&U VALUE Rubrics - http://www.aacu.org/value/rubrics/index_p.cfm?CFID=2506 8677&CFTOKEN=29196266 http://www.aacu.org/value/rubrics/index_p.cfm?CFID=2506 8677&CFTOKEN=29196266  New Leadership Alliance for Student Learning and Accountability - http://www.newleadershipalliance.org/http://www.newleadershipalliance.org/  Bresciani, M. J., Oakleaf, M., Kolkhorst, F., Nebeker, C., Duncan, K., Barlow, J., & Hickmott, J. (2009). Examining Inter-Rater Reliability for a Research Methodology Rubric. [Electronic version] Practical Assessment, Research, & Evaluation, 14(13A),1-7.

22. QUESTIONS? Marilee.Bresciani@mail.sdsu.edu