Prove It! New Tools To Measure Camp Outcomes Deb Bialeschki, ACA Jim Sibthorp, University of Utah

Overview of Session Why do we need to measure outcomes? Why do we need to measure outcomes? What do the outcomes surveys look like? What do the outcomes surveys look like? How do I know which surveys to use? How do I know which surveys to use? How do I know what they tell me? How do I know what they tell me? How do these tools fit with the new outcomes workbook? How do these tools fit with the new outcomes workbook?

Why Measure Outcomes Anyway? Helps you evaluate your program goals Helps you evaluate your program goals Documents the changes in your campers so that information can be shared with key stakeholders (parents, funders, staff, etc) Documents the changes in your campers so that information can be shared with key stakeholders (parents, funders, staff, etc) Demonstrates your commitment to quality programs that make a difference in people’s lives Demonstrates your commitment to quality programs that make a difference in people’s lives

What the Camp Folks wanted…. A tool that was: A tool that was:  Camp-specific  Easy to administer  Age- appropriate  Evidence of reliability and validity  Modular (able to select as you want) to reflect intentional focus of your program

The Chosen Outcomes Friendship Skills Friendship Skills Responsibility Responsibility Independence Independence Family Citizenship Family Citizenship Teamwork Teamwork Competence Competence Exploration Exploration

Outcomes’ Descriptions Have Camp experiences helped campers... Have Camp experiences helped campers... –Friendship Skills: develop skills in making friends and maintaining friendships? –Independence: learn to depend less on adults and other people for solving problems and for their day-to-day activities? –Teamwork: learn to be more effective when working in groups of their peers? –Family Citizenship: become better citizens when they are with their families?

Outcomes’ Descriptions… –Perceived Competence: believe that they can be successful in the things they do? –Interest in Exploration: be more curious, inquisitive, eager to learn new things? –Responsibility: learn to be better at taking responsibility for their own actions and mistakes?

What the surveys look like… Young Camper Outcomes Interview Questions Young Camper Outcomes Interview Questions 14 questions, α = questions, α =.85 Measures camp learning Measures camp learning Based on 7 domains Based on 7 domains Examples: Examples: –1.At camp, did you learn how to be better at making friends? –2.At camp, did you learn to feel good about things that you do well? –3.At camp, did you learn that you can be good at some things that you didn’t know you were good at? 4 point scale: 4 point scale: I didn’t learn anything about this I’m not sure I learned a little about this I learned a lot about this

What the surveys look like… Increase Format, α >.90 Increase Format, α >.90 Available for 7 outcomes Available for 7 outcomes 6-14 items/outcome 6-14 items/outcome 5 point scales 5 point scales How much, if any, has your experience as a camper in this camp changed you in each of the following ways? Decreased Did not increase or decrease Increased a little bit, maybe Increased some, I am sure Increased a lot, I am sure 1. Placing group goals above the things that I want. O O O O O 2. Working well with others. O O O O O

What the surveys look like… Status + change format, α generally >.84 Status + change format, α generally >.84 Available for 7 outcomes Available for 7 outcomes 6-13 items/outcome (status and change) 6-13 items/outcome (status and change) 6 point scales 6 point scales 1. I accept responsibility for my actionsFalse Somewhat False A little FalseA little True Somewhat True True Is the above statement more or less true today than before camp? A lot less Somewhat less A little lessA little more Somewhat more A lot more 2. I own up to my mistakesFalse Somewhat False A little FalseA little True Somewhat True True Is the above statement more or less true today than before camp? A lot less Somewhat less A little lessA little more Somewhat more A lot more

How do I know which one(s) to use?? Young Camper Young Camper –Best for campers 7-9, possibly up to 10 or 11 –Shortest & fastest –Unidimensional (camp learning) Increase Format Increase Format –Campers are 10+ –Interested in gains through camp –Less burdensome than status + change –Multidimensional (7 outcomes) Status + Change Status + Change –Campers are 10+ –Interested in gains through camp –Interested in status after camp (correlations with other indices) –Most burdensome –Multidimensional (7 outcomes)

How might I use these? What do you want to know? What do you want to know? –Scores for camp? –Scores by camper age? –Scores by camper gender? Make sure you are collecting the data to answer your questions… Make sure you are collecting the data to answer your questions… Give the “surveys” at the end of camp in a “quite” space. Give the “surveys” at the end of camp in a “quite” space. Data can be entered as numbers (e.g., 1= false, 2 = somewhat false…) Data can be entered as numbers (e.g., 1= false, 2 = somewhat false…) All of the scales are summative (add items up) All of the scales are summative (add items up) Examination of means and/or percentages is appropriate Examination of means and/or percentages is appropriate

What can I say? Externally: Externally: –“90% of our campers reported gains in Independence because of their time at camp” –“80% of our day campers and 90% of our residential campers reported gains in Friendship Skills because of their time at camp” Internally: Internally: –“Campers who participated in program X (e.g., our challenge course) learned more about Teamwork than campers who did not participate.” –“The older campers (> 12 year old) reported learning more about Responsibility at camp than our younger campers”

What about the Outcomes Workbook? Provides a process to get to outcomes Provides a process to get to outcomes  “Readiness” to focus on outcomes  Steps to identify goals, program elements, short & long term outcomes, and evaluation tools  Facilitator’s guide and resource CD  Useful as a staff training tool to focus on your camp’s goals and outcomes

Where do I find the outcomes surveys? PDFs at ACA Research Website ( PDFs at ACA Research Website ( Incorporated into the Outcomes Workbook Incorporated into the Outcomes Workbook Coming this year as a web-based tool Coming this year as a web-based tool  Select & print the surveys  Online data entry  Individualized report summaries

Questions?