1. Collecting Useful Data from Diverse Stakeholder Groups: Challenges and Solutions David J. Bernstein, Ph.D. Senior Study Director, Westat 2011 American Evaluation Association Conference An Inspired Design for Collecting Useful Data From Diverse Stakeholder Groups November 4, 2011

2. Presentation Outline 1. The Players 2. About the Stakeholders: Data Sources 3. Challenges and Solutions 4. Conclusions 5. Contact Information 2

3. The Players The Client: U.S. Department of Education Rehabilitation Services Administration. Data Collection Approval: The Office of Management and Budget. The Consultant: Westat. HKNC Staff: Central office point of contact, 11 regional representatives, 5 site visits, 3 interview sites. Interview Subjects:  50+ Former HKNC Trainees who worked with deaf- blind individuals.  36 deaf-blind former HKNC participants.  Other stakeholder groups. 3

4. About the Stakeholders 1. RSA identified specific stakeholder groups. 2. Little was known about the populations for most stakeholder groups from which purposive samples were to be drawn. 3. One data collection involved a census of the population (State Vocational Rehabilitation Agency Email Survey). 4 In-Person Interviews/Discussions Telephone Interviews Email Survey Site Visits D-B Clients HKNC Regional Reps. HKNC Department Heads/Key Staff Family Members Voc. Rehab. Agency Staff Service Providers Stake- holder Group Reps. VR Agencies HKNC Regional Offices and HKNC HQ 36 9992052 7765

5. Challenges and Solutions 5 Data Collection Challenge #1: Look at the outcomes, but don’t draw conclusions about effectiveness. Solutions: 1. Shift emphasis to evaluating operations and stakeholder perceptions. 2. Adjust research questions to shift focus from effectiveness to operations. 3.Identify performance metrics, but avoid conclusions about effectiveness. 4.Use methods that were as objective as possible.

6. Challenges and Strategies 6 Data Collection Challenge #2: The instrument was designed to get feedback about training for service providers, but two-thirds of the contacts may not have taken the training? Solutions: 1. Figured out that field training contacts may have coordinated but not participated in trainings. 2. Contacted coordinators to confirm if they took the workshop, and if they did not, asked for names of individuals still with the organization who did. 3. Sample was purposive not random. Selection criteria included requirement that interviewee took the workshop and were still with the agency. If not, they did not meet eligibility criteria, so response rate goals were not affected.

7. Challenges and Strategies 7 Data Collection Challenge #3: No Privacy Information Act clearance means no “system of records,” which means we could not get or maintain identifying information about some stakeholders. Solutions: 1.Applied to deaf-blind former HKNC consumers. 2.Worked with HKNC to contact interview subjects using Westat guidance on objective and effective recruitment techniques. 3.Assigned random numbers as a permanent reference to the members of the sample frame so we would have a shared reference point. 4.Held frequent updates with HKNC during recruitment to provide guidance on next steps.

8. Challenges and Strategies 8 Data Collection Challenge #4: They live where? How to identify a sample frame without knowing exactly where subjects live. Solution: 1.Proximity to site visit locations was a selection criteria. 2.Used zip codes to sort distance to interview locations. 3.Selected locations with largest number of former students (Boston, NY, LA). 4.Selected a large enough sample to ensure that response rates were met. 5.When is 45 miles not 45 miles? In Los Angeles! Used different interview locations to make participation easier for stakeholders.

9. Challenges and Strategies 9 Data Collection Challenge #5: ”Yes, I got the letter. What is it about?” Challenges in recruiting deaf-blind participants. Solutions: 1.Contact materials were simplified as much as possible and reviewed by researchers who worked with deaf-blind. 2.Used multiple media to contact individuals, including email, video phone, letters, and Braille versions. 3.HKNC central office actively tracked and pursued study subjects after regional representatives confirmed and updated contact information. 4.HKNC regional representatives and staff explained the study and gained cooperation with careful instructions to ensure that sample was not biased.

10. Challenges and Strategies 10 Data Collection Challenge #6: Interview subjects cannot see, cannot hear, and have at least ten different ways to communicate…how do we talk to them? Solution: 1.Hire good interpreters. 2.Make sure that one of your researchers: has been an interpreter, in an academic setting no less; has lots of connections; has worked in vocational rehabilitation; has good research skills and is great with people; is fluent in ASL (albeit rusty), interpret in a pinch, and monitor interpreter and interviewee communication.

11. Conclusion 11 Sometimes you have to adjust to driving conditions.

12. Contact Information 12 For more information or a copy of this presentation please contact: David J. Bernstein, Ph.D. Senior Study Director Westat 1600 Research Boulevard, RA1292 Rockville, MD 20850 (301) 738-3520 davidbernstein@westat.com