1. Outcomes Assessment and Assurance of Learning James M. Kraushaar, Ph.D., Associate Professor of MIS Nicole Chittenden, IT Manager Robert Rohr, IT Professional Sr. The School of Business Administration at the University of Vermont © James Kraushaar et al., 2005. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors. Presentation for NERCOMP 2006, Mar. 20-22, Worcester, MA

2. © 2005 The School of Business Administration at the University of Vermont Agenda Tools Tools Model Model Data Collection Data Collection Sample - Data Analysis Sample - Data Analysis Conclusion Conclusion

3. © 2005 The School of Business Administration at the University of Vermont School of Business Assessment Systems Satisfaction: Satisfaction: Educational Benchmarking (EBI) Educational Benchmarking (EBI) On-line course evaluations On-line course evaluations Advisor survey Advisor survey AACSB accreditation AACSB accreditation UVM Office of Institutional Studies (OIC) year out survey UVM Office of Institutional Studies (OIC) year out survey Competency: Competency: Educational Testing Services (ETS) Major Field Test in Business Grades Faculty Activity Reports Curriculum: Curriculum: Faculty Curriculum Coverage Inventory

4. © 2005 The School of Business Administration at the University of Vermont Curriculum: Product delivery Mission Curriculum SatisfactionCompetency

5. © 2005 The School of Business Administration at the University of Vermont Are we delivering the academic program we think we are? Curricular Programs Course Content/Delivery Extracurricular Programs Academic Advising Career & Professional Dev. Leadership Opportunities OUTCOMES ASSESSMENT COMPETENCYGrades ETS Major Field Test SATISFACTION Course Evaluations EBI Exit Survey Advising Survey GOAL: The Best Small School of Business

6. © 2005 The School of Business Administration at the University of Vermont The Predictable Questions Are our students satisfied with education? Are our students satisfied with education? Course evaluations Course evaluations Exit interviews Exit interviews Benchmarking Benchmarking Are our students prepared? Are our students prepared? Grades Grades Standardized tests Standardized tests Employer surveys Employer surveys Mission Curriculum SatisfactionCompetency Mission Curriculum SatisfactionCompetency

7. © 2005 The School of Business Administration at the University of Vermont Less Obvious Answers Curricular choices drive results: Curricular choices drive results: Where should emphasis be placed to address competency shortfalls? Where should emphasis be placed to address competency shortfalls? Which choices are working? Which choices are working? What changes will have greatest effect? What changes will have greatest effect? Are key concepts reinforced? Are key concepts reinforced? Can curricular focus on mission critical aims be identified? Can curricular focus on mission critical aims be identified?

8. © 2005 The School of Business Administration at the University of Vermont The plan… gather Curricular Coverage Inventory Evaluate aggregate curricular focus Evaluate aggregate curricular focus Quantify the total experience of program Quantify the total experience of program Target specific areas for review and/or adjustment Target specific areas for review and/or adjustment Mission Curriculum SatisfactionCompetency

9. © 2005 The School of Business Administration at the University of Vermont Design for alignment Align faculty survey of curricular coverage Align faculty survey of curricular coverage with mission statement with mission statement with student competency with student competency Can now compare/contrast subjective and objective quality measures Can now compare/contrast subjective and objective quality measures Student satisfaction vs. competency Student satisfaction vs. competency

10. © 2005 The School of Business Administration at the University of Vermont Curricular Coverage Inventory Collected from faculty Collected from faculty Info across every course Info across every course Three areas: Three areas: Mission driven Mission driven Competencies driven Competencies driven Learning Assessment methods Learning Assessment methods Demo  Demo  Demo  Demo 

11. © 2005 The School of Business Administration at the University of Vermont Sample Analysis: Topics bigger than any one course Mission Statement Excerpt: … prepares students for … dynamic global economy … ethical, social, and environmental responsibility…

12. © 2005 The School of Business Administration at the University of Vermont Results: Curricular focus on Mission Global Global Ethical Ethical Environ- mental Environ- mental Social Social

13. © 2005 The School of Business Administration at the University of Vermont Sample Report: How are students assessed in an average course

14. © 2005 The School of Business Administration at the University of Vermont Recap Talked about curriculum coverage data gathering system Talked about curriculum coverage data gathering system Nicole will discuss external satisfaction and competency measurement systems Nicole will discuss external satisfaction and competency measurement systems

15. © 2005 The School of Business Administration at the University of Vermont Incorporating Selected Outside Tools Student Satisfaction Student Satisfaction EBI – (http://www.webebi.com/wess) EBI – (http://www.webebi.com/wess)http://www.webebi.com/wess Offers “Complete Assessment Solutions” Offers “Complete Assessment Solutions” Focus on annual comparison with Carnegie class institutions Focus on annual comparison with Carnegie class institutions Select 6 - comparisons have limited value given inconsistent participation by peers Select 6 - comparisons have limited value given inconsistent participation by peers Student Competency Student Competency ETS – (http://www.ets.org) ETS – (http://www.ets.org)http://www.ets.org Major field test Major field test Measures learning outcomes and benchmarks against other institutions Measures learning outcomes and benchmarks against other institutions Required for completion of capstone class Required for completion of capstone class

16. © 2005 The School of Business Administration at the University of Vermont EBI – Measure of Student Satisfaction Incorporates measures of school initiatives Incorporates measures of school initiatives Advising, Instruction in Ethics, Career & Professional Development Advising, Instruction in Ethics, Career & Professional Development Summary format for sharing results with faculty, staff, Board of Advisors, Provost Summary format for sharing results with faculty, staff, Board of Advisors, Provost Priority Matrix – identifies items that are highly correlated with student satisfaction Priority Matrix – identifies items that are highly correlated with student satisfaction Home grown reports Home grown reports

17. © 2005 The School of Business Administration at the University of Vermont EBI – Priority Matrix 2005

18. © 2005 The School of Business Administration at the University of Vermont Example: Analyzing Satisfaction      

19. © 2005 The School of Business Administration at the University of Vermont ETS - Student competency What has it told us? Significant Fall/Spring variation. Why? Significant Fall/Spring variation. Why?

20. © 2005 The School of Business Administration at the University of Vermont Student competency differences across semesters?

21. © 2005 The School of Business Administration at the University of Vermont Multiple measures Moved from individual measurements to composite view Moved from individual measurements to composite view Neither satisfaction nor competency tells the whole story Neither satisfaction nor competency tells the whole story Combine them for more complete picture Combine them for more complete picture

22. © 2005 The School of Business Administration at the University of Vermont Curriculum Change: Student Satisfaction (EBI) vs. Student Competency (ETS) Human Resource Management LowLow AccountingLowHigh Legal and Social Environment HighLow Satisfaction Competency

23. © 2005 The School of Business Administration at the University of Vermont Case Study Legal environment Legal environment Courts & legal systems Courts & legal systems Constitution and business Constitution and business Administrative law Administrative law Regulatory Environment Regulatory Environment Employment & labor law, Antitrust law, Employment & labor law, Antitrust law, Consumer protection Consumer protection Tort law Tort law Crimes Crimes Environmental and international law Environmental and international law Business relationships Contract & commercial law (including UCC) Business organizations Law of agency Intellectual property Ethics & Social responsibility Ethics Social responsibility 10% of ETS questions center around Legal & Social Environment 10% of ETS questions center around Legal & Social Environment Legal & Social Environment Content Includes: Legal & Social Environment Content Includes:

24. © 2005 The School of Business Administration at the University of Vermont Curricular Coverage Inventory = Limited Review of Business Relationships 5 4 3 2 1 Avg

25. © 2005 The School of Business Administration at the University of Vermont Curricular Coverage Inventory: Faculty focus within course Section II: Skills/Topics Mentioned in Syllabus 132 - Legal and Political Environment of Business QuestionValue Legal and Social Environment Legal environment Courts & legal systems3.67 Averyt, William5.00 Meier, Frederic2.00 Monte, Peter4.00 Constitution & business4.33 Averyt, William5.00 Meier, Frederic4.00 Monte, Peter4.00 Administrative law2.67 Averyt, William3.00 Meier, Frederic2.00 Monte, Peter3.00 Business relationships Contracts & commercial law (incl. UCC)1.33 Averyt, William1.00 Meier, Frederic1.00 Monte, Peter2.00 Business organizations1.33 Averyt, William1.00 Meier, Frederic1.00 Monte, Peter2.00 Law of agency1.33 Averyt, William1.00 Meier, Frederic1.00 Monte, Peter2.00 Intellectual property2.33 Averyt, William4.00 Meier, Frederic1.00 Monte, Peter2.00 Ethics & social responsibility Ethics3.33 Averyt, William2.00 Meier, Frederic4.00 Monte, Peter4.00 Social responsibility3.33 Averyt, William2.00 Meier, Frederic3.00 Monte, Peter5.00 (continued, next column…)

26. © 2005 The School of Business Administration at the University of Vermont Conclusion Based on data (NOT just intuition) Based on data (NOT just intuition) Both satisfaction and competency Both satisfaction and competency Better information about delivery of topics across entire program Better information about delivery of topics across entire program Better target curricular changes Better target curricular changes Mission Curriculum SatisfactionCompetency

27. © 2005 The School of Business Administration at the University of Vermont Acknowledgements AACSB Accreditation and Training Process AACSB Accreditation and Training Process University of Alaska, Fairbanks University of Alaska, Fairbanks Western Ontario – Outcomes Assessment Conference Western Ontario – Outcomes Assessment Conference Our staff and faculty Our staff and faculty

28. © 2005 The School of Business Administration at the University of Vermont Questions

29. Contact information web: http://www.bsad.uvm.edu/assess web: http://www.bsad.uvm.edu/assesshttp://www.bsad.uvm.edu/assess James M. Kraushaar, Ph.D., Associate Professor of MIS James M. Kraushaar, Ph.D., Associate Professor of MIS Kraushaa@bsad.uvm.edu Kraushaa@bsad.uvm.edu Kraushaa@bsad.uvm.edu Nicole Chittenden, IT Manager Nicole Chittenden, IT Manager nicole.chittenden@uvm.edu nicole.chittenden@uvm.edu nicole.chittenden@uvm.edu Robert Rohr, IT Professional Sr. Robert Rohr, IT Professional Sr. robert.rohr@uvm.edu robert.rohr@uvm.edu robert.rohr@uvm.edu