Presentation is loading. Please wait.

Presentation is loading. Please wait.

HOW TO EXAMINE AND USE FAMILY SURVEY DATA TO PLAN FOR PROGRAM IMPROVEMENT Levels of Representativeness: SIOBHAN COLGAN, ECO AT FPG BATYA ELBAUM, DAC -

Similar presentations


Presentation on theme: "HOW TO EXAMINE AND USE FAMILY SURVEY DATA TO PLAN FOR PROGRAM IMPROVEMENT Levels of Representativeness: SIOBHAN COLGAN, ECO AT FPG BATYA ELBAUM, DAC -"— Presentation transcript:

1 HOW TO EXAMINE AND USE FAMILY SURVEY DATA TO PLAN FOR PROGRAM IMPROVEMENT Levels of Representativeness: SIOBHAN COLGAN, ECO AT FPG BATYA ELBAUM, DAC - FL MELISSA RASPA, ECO AT RTI

2 Purpose of the Session Overview of APR requirements regarding data quality issues Review of different types of representativeness  Response rates  Proportional representation  Within subgroup representation How to use data for program improvement

3 Family Outcomes: Part C Indicator 4 Percent of families participating in Part C who report that early intervention services have helped the family  Know their rights  Effectively communicate their children’s needs  Help their children develop and learn

4 APR Requirements: Data Quality Response rates  Address any problems with response rates, missing data, and selection [or response] bias Representativeness  Data must be representative of each program sampled, considering such variables as eligibility definition (e.g., diagnosed condition, developmental delay), age, race, and gender Part C State Performance Plan (SPP) and Annual Performance Report (APR) Instruction Sheet, available at

5 APR Report Data: Representativeness Data used for comparison  Thirty-nine states (70%) reported a source of data used for comparison Data sources included  Part C population/ 618 data: 31 states  Program population data: 3 states  Target population: 3 states  State data (not specified): 2 states

6 Criteria Used for Evaluating Representativeness Forty-six states (89%) reported the criteria they used for determining representativeness  Race/ ethnicity: 73% (41 states)  Geography (district, county, region): 50% (28 states)  Sex: 21% (12 states)  Child’s age: 20% (11 states)  Disability/ eligibility category: 9% (5 states)  Length of time in services: 9% (5 states)  Program size : 9% (5 states )

7 Were Data Representative? Forty-four states reported whether their data were representative (79%)  Yes, some data provided: 36% (20 states)  Yes, no data provided: 14% (8 states)  No: 11% (6 states)  Varied results: 18% (10 states) No statement of representativeness reported among the remaining 12 states (21%)

8 Challenges Related to Representativeness Challenges identified through APR analysis  What criteria should be used in determining representativeness  What comparison data should be used in the analyses  How to analyze the data and make conclusions  What improvement activities to develop to address the issue of representativeness

9 Levels of Representativeness Level 1: Response rates  Did everyone who was supposed to respond to the survey actually respond? Level 2: Proportional representativeness  How close do the response rates match the comparison data? Level 3: Subgroup representativeness  Within a subgroup of interest (e.g., race/ethnicity), are respondents representative of their subgroup?

10 Level 1: Response Rates Possible categories to use for comparison  Race/ethnicity  Region or program  Age  Disability type  Length of time receiving services Suggestions for others?  Income  Medicaid status

11 Example 1: Response Rates Compare families who received the survey to those who responded to the survey  Race/ethnicity  Program/region  Age group

12 What is the overall response rate? Were any groups over or under sampled? How do the response rates compare across groups? Are the data representative? Families who Received Survey Survey Respondents Response Rate Race/ethnicity#%#% American Indian/Alaska Native753%87%11% Asian/Pacific Islander1004%736%73% Black (not Hispanic)32513%645%20% Hispanic25010%212%8% White (not Hispanic)175070%104086%59% TOTAL % %48%

13 Level 2: Proportional Representativeness Possible data used for comparison  Part C population (i.e., child count)  618 data  Target population (e.g., all those exiting program)  State data Suggestions for others?  Sampling plan

14 Example 2: Proportional Representativeness Compare families who responded to the survey to a pre-specified comparison population  Race/ethnicity  Program/region  Age group

15 What data are being used for comparison? Where any groups over or under sampled? How do the percentages compare across groups? Are the data representative? Families who Received Survey Survey Respondents Comparison Data Age group#%#%#% Birth – 1 yr35014%151%45011% 1 – 2 yrs85034%18916%121031% 2 -3 yrs130052%100283%226458% TOTAL % % %

16 Level 3: Within Group Representativeness Within subgroups, are families who completed the survey representative of the whole subgroup  Race/ethnicity  Income  Disability category

17

18 Region differences: Compare row percentages to total Ethnicity differences: Compare column percentages to total Program / Region Race/ethnicityABCDTOTAL American Indian/Alaska Native0%1% 0%2% Asian/Pacific Islander1%0%2%0%3% Black (not Hispanic)4%2%5%0%11% Hispanic2%1% 5% White (not Hispanic)21%15%33%10%79% TOTAL28%19%42%11%100%

19 Program Improvement Improving data quality  Examine response rates and representativeness to determine if there is a need to get better quality data  If so, make changes to data collection methodology Improving outcomes  Once there are no differences in response rate or representativeness can examine individual outcome/indicator data  Only then can we determine next steps for program improvement

20 Contact Information Siobhan Colgan, ECO at FPG   Batya Elbaum, DAC, University of Miami   Melissa Raspa, ECO at RTI  


Download ppt "HOW TO EXAMINE AND USE FAMILY SURVEY DATA TO PLAN FOR PROGRAM IMPROVEMENT Levels of Representativeness: SIOBHAN COLGAN, ECO AT FPG BATYA ELBAUM, DAC -"

Similar presentations


Ads by Google