1. M ODELS & S TRATEGIES FOR A SSESSING I NSTITUTIONAL LEARNING OUTCOMES Ashley Finley, Ph.D Senior Director of Assessment & Research, AAC&U National Evaluator, Bringing Theory to Practice Kutztown University January 19, 2012 L EARNING -C ENTERED A SSESSMENT :

2. W HY ARE WE HERE ? What are you doing? Why are you doing it? How do you know you’re doing it well? What does it mean to be learning-centered at KU? What practices promote and advance student learning? How pervasive are these? What is the change you want to see in your students when they leave the institution? What are the foundational skills and competencies students should acquire throughout their learning experiences? What are the effective approaches for implementing a cycle of inquiry and improvement?

3. T HE N ATIONAL W ORK ON L EARNING -C ENTERED A SSESSMENT, L IBERAL E DUCATION O UTCOMES, & G ENERAL E DUCATION Liberal Education: A philosophy of learning that empowers & prepares individuals to deal with complexity, diversity, & change. = Broad knowledge combined w/ in-depth study  To help students develop a sense of social responsibility, strong & transferable intellectual and practical skills & a demonstrated ability to apply knowledge. Vast majority of institutions have articulated learning outcomes ANY institution can achieve the goals of a liberal education – 3 Major National Initiatives Compass Project (State University Systems – CA, OR, WI) Quality Collaboratives VALUE Rubric Project

4. W HAT E MPLOYERS S AY A BOUT L IBERAL E DUCATION S KILLS : % of Employers who agree with each statement Our company is asking employees to take on more responsibilities and to use a broader set of skills than in the past Employees are expected to work harder to coordinate with other departments than in the past The challenges employees face within our company are more complex today than they were in the past To succeed in our company, employees need higher levels of learning and knowledge today than they did in the past Source: Raising the Bar (AAC&U, 2010)

5. % of Employers who agree that two- and four-year colleges should place MORE emphasis on helping students develop the following: Effective oral/written communication Critical thinking/ analytical reasoning Knowledge/skills applied to real world settings Analyze/solve complex problems Connect choices and actions to ethical decisions Teamwork skills/ ability to collaborate Ability to innovate and be creative Concepts/developments in science/technology Source: “Raising the Bar: Employers’ Views on College Learning in the Wake of the Economic Downturn” (AAC&U and Hart Research Assoc. 2010)

6. A ND IN THESE AREAS … Locate/organize/evaluate information Understand global context of situations/ decisions Global issues’ implications for future Understand & work with numbers/statistics Understand role of U.S. in the world Knowledge of cultural diversity in US/world Civic knowledge, community engagement “Raising the Bar: Employers’ Views on College Learning in the Wake of the Economic Downturn” (AAC&U and Hart Research Assoc. 2010)

7. Source: Georgetown University, Center for Education and the Workforce (Anthony Carnavale) H OW M UCH D OES A L IBERAL E DUCATION P AY ? : “I MAGINATION IS MORE IMPORTANT THAN KNOWLEDGE ” -A LBERT E INSTEIN

8. Source: Georgetown University, Center for Education and the Workforce

11. Content Outcomes Learning Outcomes Theoretical traditions Methodology Topical areas How do we assess these? Course Exams Course Papers Individual/ Group course projects In-Class work/ participation ???? Articulating Student Learning and Developmen t Kutztown Mission: “ provide a high quality education at the undergraduate and graduate levels in order to prepare students to meet lifelong intellectual, ethical, social, and career challenges.” How do we assess these?

12. Knowledge of Human Cultures & the Physical & Natural World Content areas Intellectual & Practical Skills Inquiry & Analysis Critical & Creative Thinking Written & Oral Comm. Reading Quantitative Literacy Information Literacy Teamwork & Problem-solving Personal & Social Resp. Civic Knowledge Intercultural Knowledge & Competence Ethical Reasoning Lifelong Learning Integrative & Applied Learning Gen.Ed./ Major Course Institutional Outcomes Goal 1: Cultivate intell. & pract. skills Inquiry & Analysis Decision making Written & Oral Communication Quantitative Literacy Information Literacy Teamwork & problem solving Wellness Practiced throughout curr, w/ progressively more challenging probs, projects & standards Goal 2: Develop understanding of human cultures & physical and natural world Sciences, mathematics, social sciences, humanities, histories, languages, arts Engage w/ big questions, contemp. & enduring Goal 3: Sense of personal & social resp. Local/Global civic knowledge Intercultural knowledge & competence Ethical reasoning & action Personal qualities & attitudes (passion, curiosity, self-confidence, imagination, cooperation, commitment, support) Active inv. w/diverse comm. & real world chall.

13. Learning Outcomes Assess Learning Outcomes What is the story you want to tell about student learning?

14. Outcomes Assessment Curriculum

15. Learning Outcomes Assess Learning Outcomes What is the story you want to tell about student learning?

16. Basic Logic Model INPUTS Resources are needed for activities – What is needed to start or keep something going? Human Financial Technological Curricular/ Program ACTIVITIES Actions or processes necessary to produce outputs – What activities need to occur to produce evidence? OUTPUTS Products used to assess outcomes - What can be counted as evidence of change? OUTCOMES Expected Changes and Benefits – What are the goals you want to reach? Short-term Intermediate Long-term Assumptions

17. OUTCOMES Goals & Domains of Learning Understanding of human cultures… Intellectual and practice skills Critical thinking Personal & Social Responsibility Civic engagement Sample Logic Model INPUTS GE Depts. Advising Writing Center Stud. Aff. Health & Wellness Center Alumni Engagement Multicult. Services Athletics Distance Ed Career Services Inst. Res. ACTIVITIES Experiential learning Civic learning Interaction OUTPUTS Signature assignments Reflection papers Group projects Community- based projects Engagement in life’s big questions Practiced extensively & progressively Active involve. In diverse comm. & real-world challenges

18. VALUE P ROJECT ( WWW. AACU. ORG / VALUE ) Project Goals  1) Create dialogue and develop shared understanding of common learning outcomes  2) Create template for direct assessment of student work (in text and non-text formats)  3) Create student-friendly format to engage students in self- evaluation Rubric Development  Teams of faculty/scholars nationwide  Reviewed existing rubrics to identify commonalities, clarify language, and develop broad agreement on outcomes criteria ( openedpractices.org )  To date accessed by over 3000 institutions/organizations, 11,000 individuals  Domestic & international, K-12, state systems  Reliability study

19. L IST OF VALUE R UBRICS Knowledge of Human Cultures & the Physical & Natural Worlds Content Areas  No Rubrics Intellectual and Practical Skills Inquiry & Analysis Critical Thinking Creative Thinking Written Communication Oral Communication Reading Quantitative Literacy Information Literacy Teamwork Problem-solving Personal & Social Responsibility Civic Knowledge & Engagement Intercultural Knowledge & Competence Ethical Reasoning Foundations & Skills for Lifelong Learning Integrative & Applied Learning

20. Located at: http://www.aacu.org/value/index.cfm

22. Criteria The Anatomy of a VALUE Rubric Levels Performance Descriptors

23. T HE C ALIBRATION T RAINING P ROCESS Scoring Steps: Review rubric to familiarize yourself with structure, language, performance levels Ask questions about the rubric for clarification or to get input from others regarding interpretation Read student work sample Connect specific points of evidence in work sample with each criterion at the appropriate performance level (if applicable) Calibration Steps: Review scores Determine common score(s) Hear from outliers Discuss Determine final score

24. T HE G ROUND R ULES This is not grading. We are not changing the rubric (today). Our work is time sensitive. Go with your instinct. Think globally about student work and about the learning skill. Think beyond specific disciplinary lenses or content. Start with 4 and work backwards. Pick one performance benchmark per criterion. Avoid “.5”. Zero does exist. Assign “0” if work does not meet benchmark (cell one) performance level. N/A exists. Assign “not applicable” if the student work is not intended to meet a particular criterion.

25. S IGNATURE A SSIGNMENTS Assignment should enable attainment of criteria Break down criteria to determine key components for assignment What should students do with content to meet criteria? E.g. What are the pieces to be analyzed, compared, integrated? Will the assignment be used for more than one outcome? What are the types of assignments that will be most helpful for allowing students to demonstrate competency?