Learning Outcomes: Critical to Assessment Prepared for Office of Dean of Students Assessment Training January, 2011 Gail S. Rooney, Ph.D. Director, The Career Center University of Illinois at Urbana-Champaign

Session Purposes Introduce a learning framework for student affairs Gain experience writing learning outcomes tailored to your unique offices Share sample ideas for engaging in the process of defining outcomes Share sample evaluation project using learning outcomes Engage in hands-on learning that you can take back to your offices Julia

Session Outline Learning as a framework for student development programs and services Share a process for writing effective learning outcomes Defining context, brainstorming outcomes, writing outcomes statements, and considering assessment processes Considerations of a learning outcomes approach for your office Julia

Warm Up Activity Write a learning outcome for a program or service that you might offer. Rotate, discuss, and rate outcomes Scale of 1 (not good) to 8 (excellent) Add the rating scores for a total score

A Learning Framework for Student Affairs Part 1: A Learning Framework for Student Affairs

Learning Framework To be a part of the core functions of our institution, we need to think of ourselves as learning partners In an educational environment, learning is the name of the game Student development programming is education As student affairs practitioners, we are facilitating the development of lifelong and decision-making skills Learning can be a paradigm for describing the value of our work

What is Learning? * Learning is a complex, holistic, multicentric activity that occurs throughout the lifespan Learning is more than information transfer – it is identity development (transformation), giving meaning, reflection and application in the context of the individual’s life * Learning Reconsidered, 2004

What are Learning Outcomes? * Learning outcomes focus on intentional student experiences (activities, courses, programs, services, resources) What clients should be able to know, do, demonstrate or feel as a result of engaging in a learning experience Outcomes help us define how clients have changed; they tell us how we are making a difference * Keeling & Associates, 2007

Why Learning Outcomes? * Outcomes define impact – how the client has changed Focusing on learning moves beyond “bodies in the door” and “smiles on the way out” The achievement of learning outcomes (individual success) measures institutional or intervention effectiveness * Keeling & Associates, 2007

How do Learning Outcomes Shift Our Focus? * Process Planning activities and things to “do” “What are we going to do?” Counting and asking Outcome How the client will change as a result of what we do How will clients be different; how will they learn, change, grow? Documenting change; showing results * Keeling & Associates, 2007

TCC’s Office-Wide Student Learning Outcomes Posed two questions: What do we want undergraduate students to know and be able to do upon graduation regarding personal and professional career development? When a student graduates from the University of Illinois, what difference should The Career Center have made in their lives? Gathered input from: All office staff – counselors, support staff, graduate assistants, undergraduate outreach presenters, undergraduate office assistants TCC Advisory Committee members from around campus Questions back in Spring 2005

Application of Theories, Models, & Standards – TCC’s Office-Wide Student Learning Outcomes Brainstormed responses, which you see in your handouts

TCC’s Office-Wide Student Learning Outcomes There are no “right” answers… but you may be wondering: What did TCC come up with? Employed a counseling model Affective, Behavioral, and Cognitive outcomes Followed learning domains from professional standards – Explore self and options, Manage education, Use resources, Gain experience, Communicate accomplishments, Conduct a search Recognized career development theory Self, options, decision-making, life-long process Affective components of encouragement, hopefulness, etc (See “Samples” Packet, page 2)

TCC’s Office-Wide Outcomes: Practice (See “Samples” Packet, pages 3 - 4)

A Process for Writing Effective Learning Outcomes Part 2: A Process for Writing Effective Learning Outcomes

Learning Outcomes Cycle Defining Context Reconsidering Outcomes, Evaluations, Interventions Brainstorming Outcomes Evaluating Interventions Writing Outcome Statements Julia Connecting Theories / Models / Standards Prioritizing Focus Areas

Demonstrating a Process Steps addressed in learning outcomes activities: Defining context Brainstorming outcomes Writing outcomes statements Connecting theories, models, & professional standards Additional steps: Prioritizing focus areas Evaluating interventions Re-considering outcomes, evaluations, and interventions Julia

Defining Context What is your program / service? This could be… Small – a single workshop or activity Medium – multi-session interactions Large – all interactions between students and your office Who are your intended learners? This could be… Specific – first-year students in a particular minority group Broad – all community members who may seek services Bob

Defining Context – Example 1 Resume Review Services Intervention: Daily resume review services provided on a drop-in basis to clients who are conducting job and internship searches, as well as by students who need to prepare a resume as a part of an academic class. Intended Learners: Current students and alumni at the University of Illinois (See handouts for more detail) Bob

Defining Context – Example 2 Exploring Pathways in Career Success Intervention: EPICS is an interactive, online, modular program that engages students in thinking about their career decision-making early in their college experience. Intended Learners: Current undergraduate students at the University of Illinois at Urbana-Champaign (See handouts for more detail) Bob

Defining Context – Example 3 Health and Graduate School Fair Intervention: Health and Grad School Fair is held each fall in a large ballroom. It is a time of information networking and sharing regarding educational opportunities for college graduates. Intended Learners: Undergraduate students who may or may not have decided to attend grad school. (See handouts for more detail) Bob

Defining Context – Your Turn! Review the tips, examples, and prompts on pages 2 and 3 of your worksheet packet Share with your partner: the program or service that you will work with today the intended learners for that program or service Record your responses on page 4 in your packet Bob

Brainstorming Outcomes What should (intended learners) know, be able to do, be able to demonstrate, value, or feel when they have completed (program / service)? Or, stated another way… What difference should (program / service) have made for (intended learners) who participate? Gail

Brainstorming Outcomes – Resume Review Services Example What should undergraduate students know, be able to do, be able to demonstrate, or feel when they have completed a drop-in resume review? Know the characteristics of a well formatted resume (Know) Demonstrate accomplishments in their descriptions of past work experiences (Do / Demonstrate) Feel confident in their ability to create a resume that highlights their strengths and abilities (Feel) (See handouts for more detail)

Brainstorming Outcomes – EPICS Example What should undergraduate students know, be able to do, be able to demonstrate, or feel when they have completed online modules in the EPICS program? Understand how exploring their own personal interests is helpful for narrowing down potential major choices (Know) Identify skills that they would like to use in their major (Do / Demonstrate) Feel encouraged – that they can choose a major that will be a good fit for them (Feel) (See handouts for more detail)

Brainstorming Outcomes – Health and Grad Fair Example What should undergraduate students know, be able to do, be able to demonstrate, or feel when they have attended the Health and Graduate School fair? Understand the requirements for health and graduate school programs (Know) Identify and implement next steps to prepare for graduate school (Do / Demonstrate) Feel confident talking with health and graduate school representatives and exploring options (Feel) (See handouts for more detail)

Brainstorming Outcomes – Your Turn! Return to working with your partner Review the tips, examples, and prompts on page 5 of your worksheet packet Try brainstorming potential learning outcomes for the intervention and intended learners that you selected Record your responses on page 6 in your packet Gail

Writing Outcomes Statements – First Round (Intended learners) who (intervention) will be able to (intended outcome)? { } Engage in Participate in Complete { } Identify Summarize List Discuss Describe Explain Demonstrate Julia * Keeling & Associates, 2007

Writing Outcomes Statements – Resume Review Services Example Undergraduate students who engage in drop-in resume review services will be able to articulate a clear target or purpose for their resume.

Writing Outcomes Statements – EPICS Example Undergraduate students who complete EPICS Competency Builder will be able to use resources to strengthen communication skills needed in their potential work environments.

Writing Outcomes Statements – Health and Grad Fair Example Juniors and Seniors who attend the Health and Graduate School fair will be able to identify at least 3 educational options beyond their undergraduate degree.

SMART Approach to Writing Outcomes S Specific M Measurable A Achievable R Relevant T Time-based

Writing Outcome Statements – Your Turn! Return to working with your partner Review the tips, examples, and prompts on pages 7 and 8 of your worksheet packet Try writing some outcome statements that connect intervention, intended learners, and intended outcomes that you have brainstormed Record your responses on page 9 in your packet Julia

Writing Outcomes Statements – Second Round Bloom’s Taxonomy* Different Levels of Student Understanding & Performance Evaluation Synthesis Higher Cognitive Levels Analysis Julia Application Comprehension Lower Cognitive Levels Knowledge * Bloom, 1956

Writing Outcomes Statements with Bloom – Example Undergraduate students who engage in drop-in resume review services will be able to: list appropriate content for the education section of a resume (Knowledge) use strong, varied action verbs to describe their experiences (Application) integrate accomplishments / results into descriptions of their experiences (Synthesis)

Writing Outcome Statements with Bloom – Your Turn! Return to working with your partner Review the tips, examples, and prompts on pages 10 and 11 of your worksheet packet Try writing some outcome statements that connect activity, intended learners, and intended outcomes that you have brainstormed Record your responses on pages 12 and 13 in your packet

Connecting Theories, Models, & Standards Cognitive Information Processing Krumboltz’s Learning Theory Person-Environment correspondence (e.g., Holland’s typology) Social Cognitive Career Theory Super’s life-span, life-space Tiedeman’s decision making model and many more… Professional Standards National Career Development Guidelines (NCDG; 1996, 2004) National Association of Colleges and Employers (NACE) Principles (2006) Council for the Advancement of Standards in Higher Education (CAS; 2003) American School Counselor Association’s National Standards for Students (ASCA; 2003) and many more…

Theories & Models – Benefits & Drawbacks Helps us understand our assumptions Provides recognized support for program and service decisions Provides insights into: important program and service components target learners potential learning outcomes evaluation strategies Drawbacks Can create “blind spots” May not adequately translate to our unique populations and contexts At times, challenging to figure out how to translate theory into practice

Professional Standards – Benefits & Drawbacks Provides recognized practices for program and service decisions Provides a framework, each through a different lens CAS → system / program level NCDG & ASCA → individuals NACE → partnerships between career practitioners and employers Drawbacks Standards speak broadly Need to tailor outcomes to our unique learners / environments Can be overwhelming Must prioritize to make it manageable

Connecting Theories, Models, & Standards– Homework! Review the tips, examples, and prompts on page 14 of your worksheet packet Brainstorm the theories, models, and standards that influence your practice. Record your responses on page 15 in your packet Julia

Connecting Theories, Models, & Standards – Homework A “take-home” exercise! How do your learning outcomes currently speak to your theory/model/standards? Would you change or re-word any of your learning outcomes based on your theory/etc.? What learning outcomes might you add based on your theory/etc.? How might your preferred theory/etc. help justify or support your choice of learning outcomes? We’ll still share examples

A Sample Learning Outcomes Evaluation Part 3: A Sample Learning Outcomes Evaluation

Learning Outcomes Cycle Defining Context Reconsidering Outcomes, Evaluations, Interventions Brainstorming Outcomes Evaluating Interventions Writing Outcome Statements Julia Connecting Theories / Models / Standards Prioritizing Focus Areas

Setting Evaluation Priorities Learning Reconsidered Framework, with activities to engage staff in thoughtful discussion Write learning outcomes for specific programs and services Discuss program evaluation possibilities for the next year (See “Samples” Packet, pages 5 - 9)

Setting Evaluation Priorities, Con’t Selected three programs for “critical incident” evaluations Exploring Pathways in Career Success (EPICS) Resume Review Services Mock Interview Program Imagine building a “house of evidence” and each program evaluation is a brick

Evaluation Example: Resume Review Services Investigates: Student learning outcome of effectively communicating accomplishments, as demonstrated through resume writing abilities Three experimental groups Participated ONLY in resume review services Participated in resume review services AND other programs/services Have NOT used programs/services (control group)

Evaluation Example: Resume Review Services Sample consisted of Juniors and Seniors in the College of Liberal Arts & Sciences who had posted resumes on our online job posting system Blind review of “self-selected, final product” resumes with pre-determined rubric (See handout) Compared with ANOVA analysis and Dunnett’s Test of Multiple Comparisons Total rubric score Eight individual components of effective resume writing

Evaluation Example: Resume Review Sample Findings Overall resume quality In comparison to “No TCC” group: “Resume services and more” group found to have significantly higher quality resumes (α<.05) “Resume services only” group found to have significantly higher quality resumes (α<.10)

Evaluation Example: Resume Review Sample Findings Resume components Significant differences found between control and experimental groups on three components: education, additional resume sections, clear purpose/target.

Evaluation Example: Resume Review Discussion Items Overall highest ratings found on resume sections with “fairly standard” content Name / contact information (3.82/4.00) Spelling / grammar (3.47/4.00) Education (3.22/4.00) Modest overall means on areas that have general tips and guidelines, yet leave ultimate decision-making up to the resume writer. General format (2.90/4.00) Additional resume sections (2.88/4.00) Positive / professional tone (2.87/4.00) Primary experience (2.82/4.00)

Evaluation Example: Resume Review Discussion Items Lowest overall mean on component that requires themes to be threaded through various experiences, requiring high-level analysis and synthesis skills Clear purpose / target (2.43/4.00)

Evaluation Example: Resume Review Discussion Items Overall significantly higher quality for both experimental groups, in comparison to our control Significantly higher quality for both experimental groups on the education section of the resume Content was more complete, organized, clear, and well-defined than control group

Evaluation Example: Resume Review Discussion Items Significantly higher quality for “resume services and more” group on the additional resume sections and clear purpose/target section These sections demonstrate a “global sense” of career development and focus Possible that participation in a wider array of career development programs and services is beneficial to assist students in developing skills to effectively communicate a cohesive set of career aspirations through connections among the wide varieties of experiences that make up the content of their resumes

Evaluation Example: Resume Review Limitations Small sample (20 resumes in each groups = 60 total) Possible pre-existing differences in student populations Who uses the online job posting service that resumes were drawn from? How might they differ from other students? Students may be receiving resume and career assistance from other sources Private counseling Family members Professional networks Etc.

Evaluation Example: Resume Review Next Steps Recognize the good work that the resume review and counseling teams have done in working with students (Celebrate successes!) Consider areas for improvement in staff training, resume workshops, resume reviews, and (where appropriate) counseling appointments Focus on room for growth in several resume component areas, such as: general formatting, experience sections, tone, clear purpose/target. Training staff to make referrals to counseling appointments, particularly for students who are having difficulty: Defining a clear purpose or target Developing a cohesive, logical flow to the variety of experiences on their resumes

YOUR Evaluation How will you evaluate your learning outcomes? Examples? Brainstorm assessment approaches Surveys Documents Experimental – Pre & Post Qualitative Others?

Value of Learning Outcomes What are learning outcomes? Learning outcomes define impact; they measure institutional or intervention effectiveness Learning outcomes tell us how the client has changed and how we are making a difference Learning Reconsidered, 2004

Connections Between Learning Outcomes Evaluation and Advocacy Bringing learning outcomes and advocacy together Learning outcomes tell us much more about the impact of our programs and services than traditional strategies of participant counts and satisfaction surveys They show the difference that we make, giving us powerful evidence of the value of our work

Learning Outcomes Evaluations: “The Challenges” Pioneering work that may not be fully understood Requires sustained effort, energy, resources, and leadership over time Crosses many evaluation fields leading to confusion regarding terms and strategies Requires developing new creative thinking and technical evaluation skills Other ideas?

Thank You for Participating! For more information, contact me at: Gail S. Rooney grooney@uiuc.edu