1. Learning Outcomes: Critical to Assessment
Prepared for Office of Dean of Students
Assessment Training
January, 2011
Gail S. Rooney, Ph.D.
Director, The Career Center
University of Illinois at Urbana-Champaign

2. Session Purposes Introduce a learning framework for student affairs
Gain experience writing learning outcomes tailored to your unique offices
Share sample ideas for engaging in the process of defining outcomes
Share sample evaluation project using learning outcomes
Engage in hands-on learning that you can take back to your offices
Julia

3. Session Outline
Learning as a framework for student development programs and services
Share a process for writing effective learning outcomes
Defining context, brainstorming outcomes, writing outcomes statements, and considering assessment processes
Considerations of a learning outcomes approach for your office
Julia

4. Warm Up Activity
Write a learning outcome for a program or service that you might offer.
Rotate, discuss, and rate outcomes
Scale of 1 (not good) to 8 (excellent)
Add the rating scores for a total score

5. A Learning Framework for Student Affairs
Part 1:
A Learning Framework for Student Affairs

6. Learning Framework
To be a part of the core functions of our institution, we need to think of ourselves as learning partners
In an educational environment, learning is the name of the game
Student development programming is education
As student affairs practitioners, we are facilitating the development of lifelong and decision-making skills
Learning can be a paradigm for describing the value of our work

7. What is Learning? *
Learning is a complex, holistic, multicentric activity that occurs throughout the lifespan
Learning is more than information transfer – it is identity development (transformation), giving meaning, reflection and application in the context of the individual’s life
* Learning Reconsidered, 2004

8. What are Learning Outcomes? *
Learning outcomes focus on intentional student experiences (activities, courses, programs, services, resources)
What clients should be able to know, do, demonstrate or feel as a result of engaging in a learning experience
Outcomes help us define how clients have changed; they tell us how we are making a difference
* Keeling & Associates, 2007

9. Why Learning Outcomes? *
Outcomes define impact – how the client has changed
Focusing on learning moves beyond “bodies in the door” and “smiles on the way out”
The achievement of learning outcomes (individual success) measures institutional or intervention effectiveness
* Keeling & Associates, 2007

10. How do Learning Outcomes Shift Our Focus? *
Process
Planning activities and things to “do”
“What are we going to do?”
Counting and asking
Outcome
How the client will change as a result of what we do
How will clients be different; how will they learn, change, grow?
Documenting change; showing results
* Keeling & Associates, 2007

11. TCC’s Office-Wide Student Learning Outcomes
Posed two questions:
What do we want undergraduate students to know and be able to do upon graduation regarding personal and professional career development?
When a student graduates from the University of Illinois, what difference should The Career Center have made in their lives?
Gathered input from:
All office staff – counselors, support staff, graduate assistants, undergraduate outreach presenters, undergraduate office assistants
TCC Advisory Committee members from around campus
Questions back in Spring 2005

12. Application of Theories, Models, & Standards – TCC’s Office-Wide Student Learning Outcomes
Brainstormed responses, which you see in your handouts

13. TCC’s Office-Wide Student Learning Outcomes
There are no “right” answers… but you may be wondering: What did TCC come up with?
Employed a counseling model
Affective, Behavioral, and Cognitive outcomes
Followed learning domains from professional standards –
Explore self and options, Manage education, Use resources, Gain experience, Communicate accomplishments, Conduct a search
Recognized career development theory
Self, options, decision-making, life-long process
Affective components of encouragement, hopefulness, etc
(See “Samples” Packet, page 2)

14. TCC’s Office-Wide Outcomes: Practice
(See “Samples” Packet, pages 3 - 4)

15. A Process for Writing Effective Learning Outcomes
Part 2:
A Process for Writing Effective Learning Outcomes

16. Learning Outcomes Cycle
Defining Context
Reconsidering Outcomes, Evaluations, Interventions
Brainstorming Outcomes
Evaluating Interventions
Writing Outcome Statements
Julia
Connecting Theories / Models / Standards
Prioritizing Focus Areas

17. Demonstrating a Process
Steps addressed in learning outcomes activities:
Defining context
Brainstorming outcomes
Writing outcomes statements
Connecting theories, models, & professional standards
Additional steps:
Prioritizing focus areas
Evaluating interventions
Re-considering outcomes, evaluations, and interventions
Julia

18. Defining Context What is your program / service? This could be…
Small – a single workshop or activity
Medium – multi-session interactions
Large – all interactions between students and your office
Who are your intended learners? This could be…
Specific – first-year students in a particular minority group
Broad – all community members who may seek services
Bob

19. Defining Context – Example 1
Resume Review Services
Intervention: Daily resume review services provided on a drop-in basis to clients who are conducting job and internship searches, as well as by students who need to prepare a resume as a part of an academic class.
Intended Learners: Current students and alumni at the University of Illinois
(See handouts for more detail)
Bob

20. Defining Context – Example 2
Exploring Pathways in Career Success
Intervention: EPICS is an interactive, online, modular program that engages students in thinking about their career decision-making early in their college experience.
Intended Learners: Current undergraduate students at the University of Illinois at Urbana-Champaign
(See handouts for more detail)
Bob

21. Defining Context – Example 3
Health and Graduate School Fair
Intervention: Health and Grad School Fair is held each fall in a large ballroom. It is a time of information networking and sharing regarding educational opportunities for college graduates.
Intended Learners: Undergraduate students who may or may not have decided to attend grad school.
(See handouts for more detail)
Bob

22. Defining Context – Your Turn!
Review the tips, examples, and prompts on pages 2 and 3 of your worksheet packet
Share with your partner:
the program or service that you will work with today
the intended learners for that program or service
Record your responses on page 4 in your packet
Bob

23. Brainstorming Outcomes
What should (intended learners) know, be able to do, be able to demonstrate, value, or feel when they have completed (program / service)? Or, stated another way…
What difference should (program / service) have made for (intended learners) who participate?
Gail

24. Brainstorming Outcomes – Resume Review Services Example
What should undergraduate students know, be able to do, be able to demonstrate, or feel when they have completed a drop-in resume review?
Know the characteristics of a well formatted resume (Know)
Demonstrate accomplishments in their descriptions of past work experiences (Do / Demonstrate)
Feel confident in their ability to create a resume that highlights their strengths and abilities (Feel)
(See handouts for more detail)

25. Brainstorming Outcomes – EPICS Example
What should undergraduate students know, be able to do, be able to demonstrate, or feel when they have completed online modules in the EPICS program?
Understand how exploring their own personal interests is helpful for narrowing down potential major choices (Know)
Identify skills that they would like to use in their major (Do / Demonstrate)
Feel encouraged – that they can choose a major that will be a good fit for them (Feel)
(See handouts for more detail)

26. Brainstorming Outcomes – Health and Grad Fair Example
What should undergraduate students know, be able to do, be able to demonstrate, or feel when they have attended the Health and Graduate School fair?
Understand the requirements for health and graduate school programs (Know)
Identify and implement next steps to prepare for graduate school (Do / Demonstrate)
Feel confident talking with health and graduate school representatives and exploring options (Feel)
(See handouts for more detail)

27. Brainstorming Outcomes – Your Turn!
Return to working with your partner
Review the tips, examples, and prompts on page 5 of your worksheet packet
Try brainstorming potential learning outcomes for the intervention and intended learners that you selected
Record your responses on page 6 in your packet
Gail

28. Writing Outcomes Statements – First Round
(Intended learners) who (intervention) will be able to (intended outcome)? { }
Engage in
Participate in
Complete { }
Identify Summarize
List Discuss
Describe Explain
Demonstrate
Julia
* Keeling & Associates, 2007

29. Writing Outcomes Statements – Resume Review Services Example
Undergraduate students who engage in drop-in resume review services will be able to articulate a clear target or purpose for their resume.

30. Writing Outcomes Statements – EPICS Example
Undergraduate students who complete EPICS Competency Builder will be able to use resources to strengthen communication skills needed in their potential work environments.

31. Writing Outcomes Statements – Health and Grad Fair Example
Juniors and Seniors who attend the Health and Graduate School fair will be able to identify at least 3 educational options beyond their undergraduate degree.

32. SMART Approach to Writing Outcomes
S Specific
M Measurable
A Achievable
R Relevant
T Time-based

33. Writing Outcome Statements – Your Turn!
Return to working with your partner
Review the tips, examples, and prompts on pages 7 and 8 of your worksheet packet
Try writing some outcome statements that connect intervention, intended learners, and intended outcomes that you have brainstormed
Record your responses on page 9 in your packet
Julia

34. Writing Outcomes Statements – Second Round
Bloom’s Taxonomy* Different Levels of Student Understanding & Performance
Evaluation
Synthesis
Higher Cognitive Levels
Analysis
Julia
Application
Comprehension
Lower Cognitive Levels
Knowledge
* Bloom, 1956

35. Writing Outcomes Statements with Bloom – Example
Undergraduate students who engage in drop-in resume review services will be able to:
list appropriate content for the education section of a resume (Knowledge)
use strong, varied action verbs to describe their experiences (Application)
integrate accomplishments / results into descriptions of their experiences (Synthesis)

36. Writing Outcome Statements with Bloom – Your Turn!
Return to working with your partner
Review the tips, examples, and prompts on pages 10 and 11 of your worksheet packet
Try writing some outcome statements that connect activity, intended learners, and intended outcomes that you have brainstormed
Record your responses on pages 12 and 13 in your packet

37. Connecting Theories, Models, & Standards
Cognitive Information Processing
Krumboltz’s Learning Theory
Person-Environment correspondence (e.g., Holland’s typology)
Social Cognitive Career Theory
Super’s life-span, life-space
Tiedeman’s decision making model
and many more…
Professional Standards
National Career Development Guidelines (NCDG; 1996, 2004)
National Association of Colleges and Employers (NACE) Principles (2006)
Council for the Advancement of Standards in Higher Education (CAS; 2003)
American School Counselor Association’s National Standards for Students (ASCA; 2003)
and many more…

38. Theories & Models – Benefits & Drawbacks
Helps us understand our assumptions
Provides recognized support for program and service decisions
Provides insights into:
important program and service components
target learners
potential learning outcomes
evaluation strategies
Drawbacks
Can create “blind spots”
May not adequately translate to our unique populations and contexts
At times, challenging to figure out how to translate theory into practice

39. Professional Standards – Benefits & Drawbacks
Provides recognized practices for program and service decisions
Provides a framework, each through a different lens
CAS → system / program level
NCDG & ASCA → individuals
NACE → partnerships between career practitioners and employers
Drawbacks
Standards speak broadly
Need to tailor outcomes to our unique learners / environments
Can be overwhelming
Must prioritize to make it manageable

40. Connecting Theories, Models, & Standards– Homework!
Review the tips, examples, and prompts on page 14 of your worksheet packet
Brainstorm the theories, models, and standards that influence your practice.
Record your responses on page 15 in your packet
Julia

41. Connecting Theories, Models, & Standards – Homework
A “take-home” exercise!
How do your learning outcomes currently speak to your theory/model/standards?
Would you change or re-word any of your learning outcomes based on your theory/etc.?
What learning outcomes might you add based on your theory/etc.?
How might your preferred theory/etc. help justify or support your choice of learning outcomes?
We’ll still share examples

42. A Sample Learning Outcomes Evaluation
Part 3:
A Sample Learning Outcomes Evaluation

43. Learning Outcomes Cycle
Defining Context
Reconsidering Outcomes, Evaluations, Interventions
Brainstorming Outcomes
Evaluating Interventions
Writing Outcome Statements
Julia
Connecting Theories / Models / Standards
Prioritizing Focus Areas

44. Setting Evaluation Priorities
Learning Reconsidered Framework, with activities to engage staff in thoughtful discussion
Write learning outcomes for specific programs and services
Discuss program evaluation possibilities for the next year
(See “Samples” Packet, pages 5 - 9)

45. Setting Evaluation Priorities, Con’t
Selected three programs for “critical incident” evaluations
Exploring Pathways in Career Success (EPICS)
Resume Review Services
Mock Interview Program
Imagine building a “house of evidence” and each program evaluation is a brick

46. Evaluation Example: Resume Review Services
Investigates:
Student learning outcome of effectively communicating accomplishments, as demonstrated through resume writing abilities
Three experimental groups
Participated ONLY in resume review services
Participated in resume review services AND other programs/services
Have NOT used programs/services (control group)

47. Evaluation Example: Resume Review Services
Sample consisted of Juniors and Seniors in the College of Liberal Arts & Sciences who had posted resumes on our online job posting system
Blind review of “self-selected, final product” resumes with pre-determined rubric (See handout)
Compared with ANOVA analysis and Dunnett’s Test of Multiple Comparisons
Total rubric score
Eight individual components of effective resume writing

48. Evaluation Example: Resume Review Sample Findings
Overall resume quality
In comparison to “No TCC” group:
“Resume services and more” group found to have significantly higher quality resumes (α<.05)
“Resume services only” group found to have significantly higher quality resumes (α<.10)

49. Evaluation Example: Resume Review Sample Findings
Resume components
Significant differences found between control and experimental groups on three components: education, additional resume sections, clear purpose/target.

50. Evaluation Example: Resume Review Discussion Items
Overall highest ratings found on resume sections with “fairly standard” content
Name / contact information (3.82/4.00)
Spelling / grammar (3.47/4.00)
Education (3.22/4.00)
Modest overall means on areas that have general tips and guidelines, yet leave ultimate decision-making up to the resume writer.
General format (2.90/4.00)
Additional resume sections (2.88/4.00)
Positive / professional tone (2.87/4.00)
Primary experience (2.82/4.00)

51. Evaluation Example: Resume Review Discussion Items
Lowest overall mean on component that requires themes to be threaded through various experiences, requiring high-level analysis and synthesis skills
Clear purpose / target (2.43/4.00)

52. Evaluation Example: Resume Review Discussion Items
Overall significantly higher quality for both experimental groups, in comparison to our control
Significantly higher quality for both experimental groups on the education section of the resume
Content was more complete, organized, clear, and well-defined than control group

53. Evaluation Example: Resume Review Discussion Items
Significantly higher quality for “resume services and more” group on the additional resume sections and clear purpose/target section
These sections demonstrate a “global sense” of career development and focus
Possible that participation in a wider array of career development programs and services is beneficial to assist students in developing skills to effectively communicate a cohesive set of career aspirations through connections among the wide varieties of experiences that make up the content of their resumes

54. Evaluation Example: Resume Review Limitations
Small sample (20 resumes in each groups = 60 total)
Possible pre-existing differences in student populations
Who uses the online job posting service that resumes were drawn from? How might they differ from other students?
Students may be receiving resume and career assistance from other sources
Private counseling
Family members
Professional networks
Etc.

55. Evaluation Example: Resume Review Next Steps
Recognize the good work that the resume review and counseling teams have done in working with students (Celebrate successes!)
Consider areas for improvement in staff training, resume workshops, resume reviews, and (where appropriate) counseling appointments
Focus on room for growth in several resume component areas, such as: general formatting, experience sections, tone, clear purpose/target.
Training staff to make referrals to counseling appointments, particularly for students who are having difficulty:
Defining a clear purpose or target
Developing a cohesive, logical flow to the variety of experiences on their resumes

56. YOUR Evaluation How will you evaluate your learning outcomes?
Examples?
Brainstorm assessment approaches
Surveys
Documents
Experimental – Pre & Post
Qualitative
Others?

57. Value of Learning Outcomes
What are learning outcomes?
Learning outcomes define impact; they measure institutional or intervention effectiveness
Learning outcomes tell us how the client has changed and how we are making a difference Learning Reconsidered, 2004

58. Connections Between Learning Outcomes Evaluation and Advocacy
Bringing learning outcomes and advocacy together
Learning outcomes tell us much more about the impact of our programs and services than traditional strategies of participant counts and satisfaction surveys
They show the difference that we make, giving us powerful evidence of the value of our work

59. Learning Outcomes Evaluations: “The Challenges”
Pioneering work that may not be fully understood
Requires sustained effort, energy, resources, and leadership over time
Crosses many evaluation fields leading to confusion regarding terms and strategies
Requires developing new creative thinking and technical evaluation skills
Other ideas?

60. Thank You for Participating!
For more information, contact me at:
Gail S. Rooney