2. Making Family Outcome Data Representative of the State Monday, August 27, 2007 Rosanne Griff-Cabelli rosanne.griff-cabelli@state.de.us rosanne.griff-cabelli@state.de.us Sue Campbell susan.campbell@state.de.us susan.campbell@state.de.us Delaware Birth to Three Early Intervention System

3. Purpose of the Original Family Survey The survey was conducted in order to obtain information about families perceptions of and satisfaction with services they received from the Child Development Watch Program (CDW) determine families satisfaction with the CDW office facilities and staff interactions determine the impact of CDW and Birth to Three services on families quality of life

4. Topics Addressed in the Original Family Survey Did parents think their ability to parent their child with a disability improved? Did parents think their child(ren)s development improved? Did parents feel that CDW was responsive to their needs? Did parents feel that they had input into the services their children received? Were families satisfied with CDW and staff interactions? Did CDW services have a positive impact on families quality of life?

5. 2006 Family Outcomes to be Assessed Outcome 1: Families know their rights and advocate effectively for their child Outcome 2: Families Understand their childs abilities and special needs Outcome 3: Families help their child develop and learn

6. Reasons for a Pilot OSEP Family Outcomes Develop and integrate OSEP outcomes questions into existing survey Expand responses for existing survey items From a three point response to a six point response Test strategy of telephone interview for surveying families

7. Developing the New Family Survey Delaware Ongoing Program Evaluation Committee (OPEC) made recommendations: Added questions to increase depth and range of response for the three family outcomes Responses reported as single questions and by clusters– look at trends since 1999 Expand responses of the existing survey from a range of three to a range of six BEFORE:Yes; Less than Id Like; No AFTER:Very strongly agree; Strongly agree; Agree; Disagree; Strongly disagree; Very strongly disagree

8. Piloting the New Family Survey Conducted in Fall 2005 Telephone contact families randomly selected letter sent to families before calls made Calls made during day and evening

9. Demographics of 2005 Telephone Pilot Family Survey Respondents (N=49) Gender (childs) 26 male(53%) 22 female (45%) Race/Ethnicity 38 Caucasian (79%) 7 AfricanAmerican (15%) 1 Hispanic (2%) 1 Asian (2%) 1 Not Reported (2%) Family Income 11>$100,000 (23%) 19$50-100,000 (41%) 9$20-49,999 (19%) 6<$20,000 (13%) 2 Not Reported Location 24 New Castle County (49%) 25 Kent or Sussex County (51%)

10. 2005 Federal Outcomes Measures Internal Consistency Federal Outcome Topic Number of Questions Chronbach Alpha Coefficient % of Families indicating yes Families Know their Rights 4.84691.1% Families Effectively Communicate their Childrens Needs 5.71595.1% Families help their Children Develop and Learn 4.80893.4%

11. 1.Questions from pilot found appropriate for use 2.Better to use telephone survey than mail distribution method 3.Stratified sample rather than random sample Pilot resulted in the following:

12. 2006 Telephone Interviews with Families Representative sample is predetermined by information in data system (ISIS) Geographic region Race/ethnicity Length of time in CDW (state focus) Sample matrix currently includes 12 data cells Each cell contains 30 families to be sampled Total of 360 families to be sampled Can always add specific demographics if need to focus on specific populations Small cell size

13. Sampling Matrix Geographic RegionEthnicityLength of Time in Program North Caucasian< 18 months (30 families) > 18 months (30 families) African American < 18 months (30 families) > 18 months (30 families) Other< 18 months (30 families) > 18 months (30 families) South Caucasian< 18 months (30 families) > 18 months (30 families) African American < 18 months (30 families) > 18 months (30 families) Other< 18 months (30 families) > 18 months (30 families)

14. Number of Families by Region, Ethnicity, and Months in CDW Region African AmericanCaucasianOther Total Less than 18 Months More than 18 Months Less than 18 Months More than 18 Months Less than 18 Months More than 18 Months North1105722114711237684 South5226104603515292 Total1628332520714752976 Received list of 976 families in CDW 684 in New Castle County (CDW North) 292 in Kent and Sussex Counties (CDW South)

15. Before the Telephone Interviews… 850 of these families were mailed: 1.a cover letter explaining the purpose of the interview 2.an information sheet describing the usefulness of family feedback assurances of confidentiality examples of some information that would be asked additional contact numbers if families had questions about the interview

16. Number of Letters Mailed by Region, Ethnicity, and Months in CDW (N=850) Ethnicity African American CaucasianOther Length of Time in Program Less than 18 Months Greater than 18 Months Less than 18 Months Greater than 18 Months Less than 18 Months Greater than 18 Months State 1 st Mailing 11283120 14752 2 nd Mailing 606000 3 rd Mailing 4401006000 Total1628322618014752

17. Letters Mailed and Return Rate Family Survey. Telephone calls were made to all of the families from the original list, unless letters were returned due to incorrect addresses, until 224 families had completed the Family Survey. The families were contacted between one and four times. Calls would be made during the day, in the evenings, and on the weekends Reasons families were not contacted via telephone: Telephone was disconnected Number was wrong Families were found not to meet the criteria for completing the telephone interview

18. Reasons why Families were not Contacted Reasons families were not contacted via telephone: 97 Telephone was disconnected (35.4%) 63 Number was wrong (23.0%) 59 Mail returned (21.5%) 28 No phone number listed (10.2%) 9 Less than six months in CDW (3.3%) 5 Phone not in service (1.8%) 14 Other reasons (5.1%) 275 families

19. How did Delaware Determine Family Outcome Data is Representative? 1.Contract with the Center for Disabilities Studies, University of Delaware to moderate biased response 2. Sample according to demographics of Annual Child Count

20. Cost Data Preliminary Analysis Delivery MethodMailed SurveyTelephone Interview Year of Implementation20042006 Sampling MethodRandom SampleStratified Random Sample Number Contacted203 surveys576 candidates Number Completed100 surveys224 interviews Return Rate49.3% of sample38.9% of sample % of the Population10.80%22.90% Cost/Person Contacted$3.23/family$2.34/family Cost/Completed Survey$6.57/survey$8.91/survey Methodology Mailed survey up to 4 times Mailed letter Telephone call reminder Telephone Interviews Costs IncludedMailed letter and surveyMailed letter Return envelopeTelephone calls Return mailing Survey reproduction costs Phone call reminders

21. Conclusions 1.The expanded response possibilities allows for deeper analysis of families responses (move to 6 point scale) 2.The telephone survey allows for individual responses and clarification of questions 3.The telephone survey allows for the immediate identification of the sample demographics, and allows for targeting of underrepresented categories. Data collection will continue until all categories are appropriately represented 4. The telephone survey also allows for immediate feedback as to why families chose not to participate and the demographics of who chose not to participate

22. Lessons Learned for Efficient Family Surveying and Adequate Representation 1.Contact families at various times of the day within a shorter timeframe (1-2 weeks) rather than making multiple attempts at a larger timeframe. 2.Request updated information from CDW offices to contact families who have disconnected phone numbers, no telephone numbers or wrong telephone numbers. If phone number still not available, mail survey. 3.Send a follow-up letter to families asking families to call back to schedule an interview