1. The Nuts and Bolts of Outcome Assessment
Terri Manning, Ed.D.
Center for Applied Research
Central Piedmont Community College

2. Stages of Grief for Outcome Measurement
Acceptance & adaptation
Challenge & competition
Catalyst - Proactive
Stage 4
Depression
Compliance - Passive reactive
Stage 3
Bargaining - no time/no money
Seek outside sources
Stage 2
Anger and antagonism
Resistant & Reactive
Disbelief & Denial
Paralysis - Passive resistance
Stage 1

3. In Education… We’ve learned that things come and they go
Most of these trends are purely academic exercises
The national educational climate is skeptical about accountability!
They want us to prove that students are learning, that their lives are improving and that we are good stewards of funds!

4. The Spellings Commission
Some things the draft report called for:
The creation of an overall measurement of an institution's "bottom line," including measures of institutional costs and performance that let parents and policy makers view institutional results;
A mandate that institutions measure student learning outcomes, disseminate the results to students, and report them publicly in the aggregate;
The development of a national student unit-record database to follow the progress of each student;
The establishment a national accreditation framework that includes comparable performance measures, and making the findings of reviews easily accessible to the public;

5. According to SACS, an Institution Needs to Have in Place:
An ongoing, integrated, and institution-wide research-based planning and evaluation processes which includes:
a systematic review of institutional mission, goals and outcomes
results in continuing improvement in institutional quality
demonstrates that the institution is effectively accomplishing its mission
A list of expected outcomes, the assessment of those outcomes and evidence of improvement based on analysis of those outcome results in each of the following areas:
educational programs (student learning outcomes at the program and individual level)
administrative support services
educational support services
Identified college-level general education competencies (based on best practices in assessment) and provide evidence that graduates have attained them

6. But Why?
SACS Core Requirements and Comprehensive Standards are based on best practices of effective institutions (a peer developed and peer reviewed process).
SACS staff members enforce the standards.
But shouldn’t we periodically take a serious look at our students?
Are they learning?
Who is learning best?
Are they achieving the outcomes we expected?
Should we make changes in programs and services?
Do we need more in-depth services?
Do we need a new curriculum or a change in methodology?

7. The Great Fallacy Grades
In this day of social promotion, grade inflation and different teaching/learning philosophies, grades tell you virtually nothing.
They are not a measure of outcome achievement.
Two teachers will grade a student differently for the exact same work.
They cannot be used!

8. Why are We Moving from Goals to Outcomes?
Outcomes are program-specific
They measure the effect of classroom activities and services provided.
Outcomes represent a new way of thinking
Outcomes have become widely accepted by our various publics
They are here to stay
We used to measure ourselves by our activities

9. Program Outcome Model INPUTS ACTIVITIES OUTPUTS
Resources Services Products or Results of
Activities
Staff Education (classes) Numbers served
Buildings Services FTE (input next year)
Facilities Counseling # Classes taught
State funds Student activities # Students recruited
FTE
Constraints
Laws
State regulations
United Way model

10. Program Outcomes Model
INPUTS
> > >
Benefits for People
*New knowledge
*Increased skills
*Changes in values
*Modified behavior
*Improved condition
*Altered status
*New opportunities
ACTIVITIES
OUTPUTS
OUTCOMES
(Outcomes answer the “so what” question)

11. Which Is It?
An input
An activity
An output
An outcome

12. GED Preparation College provides well trained faculty.
200 students complete their GED.
Students move from public housing.
30 courses are offered each semester.
150 FTE are generated.
Students’ reading level improves.
Students are gainfully employed.

13. Student Services Four counselors are hired.
Students successfully transfer courses and enter the university system.
Students receive financial aid at the university.
Students attend campus activities.
Students are able to pay back their student loans.

14. General Education Courses
Students receive creative classroom experiences.
Faculty members receive a grant to offer multimedia opportunities to students.
Students become active participants in County civic activities.
700 FTE are generated in core courses.
Students’ math skills improve.

15. Outcomes are ……... Driven by the mission.
Related to overall program goals.
Specific to the teachings/activities of your program/course.
Determined by faculty and front-line staff.
Measured carefully and specifically.

16. Inputs through Outcomes: The Conceptual Chain
Long-range
Intermediate
OUTCOMES
Initial
Outputs
Activities
Inputs

17. Different Types of Outcomes
Learning Outcomes (can be at course, program or institutional level)
Program Outcomes
Administrative Outcomes

18. Definitions and Examples
Learning Outcomes:
What changes in knowledge, skills, attitude, awareness, condition, position (etc.) occur as a result of the learning that takes place in the classroom. These are direct benefits to students.
Examples: general learning skills (e.g. improved writing and speaking abilities), ability to apply learning to the work environment (e.g. demonstrate skills in co-op), program-specific skills developed or enhanced (e.g. take blood pressure.)

19. Definitions and Examples
Program Outcomes:
The benefits that results from the completion of an entire program or series of courses. Are there benefits for students who get the AAS in welding versus those who take a few courses? If so what are they?
Typical examples are: licensure pass rates, employment rates, acceptance into 4-year schools, lifelong learning issues, contributions to society, the profession, etc.)

20. Definitions and Examples
Administrative Outcomes
Units/programs want to improve services or approach an old problem in a new way.
They want to become more efficient and effective.
Typical examples are:
All faculty will attend one professional meeting annually so they can stay up-to-date in their field, or:
Counseling wants to recruit a new counselor with expertise in working with first-generation students, or:
Facilities services wants all college units to feel that they respond quickly to security issues.

21. What is an Outcome Objective?
A short-term, measurable, specific activity having a time limit or timeline for completion around a specific outcome
They measure outcomes and are used to show progress toward goals
They specify who, will do what, under what condition, by what standard and within what time period

22. How to Set Outcome Objectives
There’s no magic number
e.g. 80% or 90%
What is reasonable?
What can you afford?
What realistically can your staff accomplish?
What percent shows you’re not committed and what percent shows you’re naïve?

23. How to Set Outcome Objectives
Examples:
Fifty percent of students will be able to communicate effectively in writing (complete the writing exam with a grade of 60 [D] or better)
By the end of the spring term, 95% of faculty and staff will have completed 20 contact hours of professional development (workshops, college courses, conferences, onsite trainings, etc.)

24. More Realistic
Seventy percent of students will be able to communicate effectively in writing (complete the writing exam with a grade of 75 [C+] or better)
By the end of the spring term, the professional development office will increase their offerings for faculty and staff by 10% over what was offered last year (workshops, college courses, conferences, onsite trainings, etc.)

25. How to Set Objectives
The first time you set objectives, be conservative
Allow yourself a pilot semester or year to determine the appropriate levels of change that can be expected
Don’t pull a rabbit out of your hat (e.g. let’s grow enrollment by 10%)
May need to benchmark (what does it mean)

26. Fall Curriculum Enrollment at ABC College
(1.2%)
(1.5%)
(2.6%)
(1.6%)
(1.7%)
(1.5%)
(1.8%)
(1.6%)
Should they set a goal for a 5% enrollment growth for next fall?

27. Why is This Hard? Because it is education
Because the best results may not happen for years
Because we are so busy doing what we are doing…. we forget why we are doing it

28. Let’s Look at Healthcare
When you have strep throat and go to the doctor for your antibiotics (your intervention)
What are your intended outcomes?
Would the doctor ever tell you:
We believe curing disease is a developmental process
We believe there is value in the activity of taking pills and receiving shots
We’re not sure if you’ll get better, it is how all doctors have treated the disease since we learned about it
What would you say to that?

29. How to Measure Program/Student Outcomes

30. Identifying Outcomes
Faculty/staff in an area are often the worst in defining outcomes because they are too close to the subject.
New Teachers
1st Year
2nd- 3rd
Year Teachers
Middle Years of Teaching
Older
Teachers
Unconsciously Unskilled
Consciously Unskilled
Consciously Skilled
Unconsciously Skilled
Don’t realize they don’t know what they are doing
Realize they don’t know what they are doing – begin to ask for help
Know what works and why
Can’t remember why they do what they do – just know it works

31. Sources of Ideas for Outcomes
* program documents
* program faculty and staff
* national associations/credentialing boards
* key volunteers
* former students
* parents of students
* records of complaints
* programs/agencies/employers that are the next
step for your students
* other colleges with similar programs, services
and students as yours
* outside observers of your program in action

32. How Often
Should we measure objectives or student learning outcomes every year?
When does measurement become too time consuming?
Units need time to put into effect the changes made as a result of outcome assessment before they are thrown back into another cycle. They need time to reflect on changes and results.

33. Disappointing Outcome Findings: Why Didn’t We Meet Our Objectives?
Internal Factors:
* Sudden staff turnover
* New teaching philosophy/strategy
* Curricular change (campus move)
* Unrealistic outcome targets
* Measurement problems (lack of follow through, no effective tracking)

34. Disappointing Outcome Findings
External Factors:
* Community unemployment increases
* State funding changes
* Related programs (BS or MS programs) close
* Public transportation increases fares or shuts
down some routes serving your campus or
time slot
* Employment trends change

35. Use Your Findings

36. Internal Uses for Outcome Findings
Provide direction for curricular changes
Improve educational and support programs
Identify training needs for staff and students
Support annual and long-range planning
Guide budgets and justify resource allocations
Suggest outcome targets (expected change)
Focus board members’ attention on programmatic issues
Help the college expand its most effective services
Facilitates an atmosphere of change within the institution

37. An Example from Instruction
Workplace Basic Skills
This program is a literacy initiative that goes directly into the worksite and teaches ESL classes, GED prep and GED classes.
During their review, they surveyed both employers and students.
This was the first time they had ever done this.

38. What They Learned Employers said:
43.8% of employers reported increases in employee performance as a result of participation in the program.
31.3% reported a reduction in absenteeism by participants.
87.5% said classes improved the morale of their employees
37.5% said participants received raises
50% said communication had improved.

39. What Students Said
70.2% reported being able to fill out job forms better
35.5% said they could now help their children with their homework
91.1% said they felt better about themselves
44.4% said they had received a raise, promotion or opportunity as a result of the courses
86.3% said their ability to communicate in the workplace had improved

40. What Has Happened Since
Their assessment data has shown up in their marketing brochures to employers.
Their enrollment has grown dramatically.
They have received funding and marketing support from Charlotte Reads (considered a model adult literacy program).

41. External Uses of Outcome Findings
Recruit talented faculty and staff
Promote college programs to potential students
Identify partners for collaboration (hospitals, businesses, etc....)
Enhance the college’s public image
Retain and increase funding
Garner support for innovative efforts
Win designation as a model or demonstration site

42. So will someone help me do this… help me select and measure outcomes for my program?

43. Ideas for Outcomes
Typical general education goals:
Students will demonstrate the ability to obtain meaning from printed, electronic, and graphical resources
Students will effectively communicate both orally and in writing. Students will demonstrate the ability to locate, critically evaluate, and present information.
Students will apply mathematical concepts and skills to analyze, manipulate, and interpret quantitative data.
Students will demonstrate the basic computer skills necessary to function in a technological world.
Students will demonstrate the ability to identify, analyze, question, and evaluate content as a guide to understanding and action.
Students will demonstrate knowledge of cultural differences.
Students will demonstrate an understanding of the influence of the individual on group behavior and conversely, the influence of the group on the individual.
Students will demonstrate comprehension of the major steps of the scientific method.
Students will demonstrate knowledge of the humanities and critical skills in assessing cultural/artistic merit and significance.

44. Let’s work through a few.
How does the welding program improve math skills in students?
Welding students will be proficient at weights and measures, calculating angles and predicting poundage of resistance.
How does the culinary arts program teach students to write effectively?
Culinary students will be able to successfully write menu descriptions, restaurant advertisements and job descriptions.
How does the Nursing program teach students to think critically?
Through case scenarios, nursing students will identify the correct course of treatment for diabetic patients based on descriptions of symptoms, behaviors and patient knowledge.

45. Top Ten Skills for the Future
Work ethic, including self-motivation and time management.
Physical skills, e.g., maintaining one's health and good appearance.
Verbal (oral) communication, including one-on-one and in a group
Written communication, including editing and proofing one's work.
Working directly with people, relationship building, and team work.
Influencing people, including effective salesmanship and leadership.
Gathering information through various media and keeping it organized.
Using quantitative tools, e.g., statistics, graphs, or spreadsheets.
Asking and answering the right questions, evaluating information, and applying knowledge.
Solving problems, including identifying problems, developing possible solutions, and launching solutions.
The Futurist Update (Vol. 5, No. 2), an e-newsletter from the World Future Society, quotes Bill Coplin on the “ten things employers want [young people] to learn in college”

46. Let’s work through a few?
How does the early childhood program improve the work ethic of childcare workers or the children in childcare centers?
How do IT programs improve teamwork skills?
How does Engineering improve students’ ability to use spreadsheets and read tables/graphs?
How do you improve a student’s ability to:
Influence people
Have time management skills
Ask the right questions
Solve problems in the work environment

47. Learning Outcomes for the 21st Century
Students in the 21st Century will need to be proficient in:
Reading, writing, speaking and listening
Applying concepts and reasoning
Analyzing and using numerical data
Citizenship, diversity/pluralism
Local, community, global, environmental awareness
Analysis, synthesis, evaluation, decision-making, creative thinking
Collecting, analyzing and organizing information
Teamwork, relationship management, conflict resolution and workplace skills
Learning to learn, understand and manage self, management of change, personal responsibility, aesthetic responsiveness and wellness
Computer literacy, internet skills, information retrieval and information management
(The League for Innovation’s 21st Century Learning Outcomes Project.)

48. Let’s work through a few.
How does any given program or course improve:
Listening skills
Environmental awareness
Creative thinking
Relationship management
Conflict resolution
Self-management
Wellness
Information management
How can we measure it and use the results.

49. Challenges Identifying and defining outcomes is the easy part.
The devil is in the details.
How do we track it, where does it all go, how do we score it, compile it, turn it into a comprehensive report.
How do we “demonstrate improvement in institutional quality.”

50. Things to Remember
Outcome measurement must be initiated from the unit/department level (promotes ownership of process).
Measure only what you are teaching or facilitating.
Measure what is “important” to you or your program.
Be selective (2-3 outcomes only for a course, a select list for programs and institutional outcomes).
Put as much time in to “thinking through” the tracking process as you do into the definition of outcomes.
Spend the time up front in planning and the process will flow smoothly.
It will prove to be energy well spent.

51. Look at Early Childhood Examples

52. Remember To Improve Programs and Services
We do not do outcome evaluation so we can say we did it.
We do it only for one reason:
To Improve Programs and Services

53. Where Colleges Get In Trouble
Overkill – they evaluate everything that walks and breaths every semester in every area.
No time to “reflect” before they enter back into another assessment cycle.
No focus on “use of results.”
No ability to track results and tally them across the College.

54. The Insanity Principle
Doing the same thing we have always done but expecting different results.

55. Contact
Terri M. Manning
(704)