1. Patricia I. Documét 1, MD, DrPH, Yll Hyseni 2, MPH, Edmund M. Ricci 3, PhD, MLitt, Caroline Feller 4, MS, OTR, CCM Evaluation of the Brain Injury Association of America’s National Brain Injury Information Center (NBIIC) 1 Asst. Professor in the Dept. of Behavioral and Community Health Sciences at the Graduate School of Public Health, University of Pittsburgh 2 Early Warning Program Coordinator of the United Nations Development Program in Pristina, Kosovo 3 Director, Institute for Evaluation Science in Community Health, University of Pittsburgh 4 4 Brain Injury Association of America The Evaluation Institute Graduate School of Public Health

2. Background: The University of Pittsburgh’s Institute for Evaluation Science in Community Health conducted a utilization-focused evaluation of the Brain Injury Association of America‘s (BIAA) National Brain Injury Information Center (NBIIC), a project funded by the Centers for Disease Control and Prevention. The project aimed at helping the information and resource (I&R) programs of the BIAA’s state affiliates improve the quality of life of individuals with brain injury and their families by providing accurate, appropriate and timely information. Methods: Guided by a logic model, the evaluation methodology included document review, a survey with NBIIC callers and interviews with I&R staff in pilot states, their supervisors and NBIIC staff. Abstract

3. Results: Between September of 2005 and January of 2006, the NBIIC initiated services in three pilot states: Minnesota, Michigan, and Mississippi. Project staff developed and produced a caller information packet, I&R staff training and procedures manuals, and outreach materials; developed and implemented an outreach plan; established a telephone system connecting callers to their state program; developed project management tools; and trained I&R staff. Our findings include: 1) NBIIC callers have simplified access to resources through this system; 2) the NBIIC provides pilot states a standard structure for their work through the training, manuals, management materials and information packet; 3) pilot states have developed partnerships and are sharing information and procedures. Recommendations: Extend the NBIIC to include more states, so they can benefit from the standardized structure it provides. Fine-tuning the materials and processes. Abstract

4. The Evaluation This was a utilization-focused evaluation 1 with process and outcomes components. 2 It was based on the steps outlined by the Centers for Disease Control and Prevention’s (CDC) Evaluation Working Group in September of 2005. 3 1.Engage stakeholders 2.Describe the program 3.Focus the evaluation design 4.Gather credible evidence 5.Justify conclusions 6.Ensure use and share lessons learned

5. The Stakeholders The project was funded by the Centers for Disease Control and Prevention (CDC), via a cooperative agreement. There are several individuals and groups that have a stake in this project. They include the BIAA (the commissioning organization), the CDC’s Center for Injury Prevention and Control that funds the project, the state affiliates, as well as individuals with traumatic brain injury and their families. Additionally, the NBIIC relied on an External Expert Committee (EEC) to guide its work.

6. The Program: Background According to the Brain Injury Association of America 4 CDC has recommended that access should be improved to the information for families of and individuals with traumatic brain injuries. The information is to be user friendly and tailored to the specific conditions of the area of the country where each individual and his/her family resides. The BIAA also states that there is lack of research evidence that documents the utility and effectiveness of providing Information and Resources (I&R) to those who are experiencing brain injury and their caregivers. The present evaluation has the potential to provide some of that evidence.

7. The Program: Expected Effects The NBIIC pilot project’s ultimate goal was to improve the quality of life of individuals with brain injury and their families by providing accurate, appropriate and timely information. The pilot project was set up to test a mechanism to offer that information, an outreach plan to advertise the mechanism, materials and a training program. At the same time, information gathered from callers was expected to be used to improve that NBIIC system.

8. The Program: Activities Since the beginning of the project in 2003, the NBIIC has conducted a series of activities to achieve its objectives: Produce an information packet to be distributed to callers Develop training and manuals for I&R staff and administrators Develop outreach materials for brain injured individuals and their families/caregivers Develop and implement an outreach plan Establish a 1-800 telephone system that connects callers to their state program Develop project management tools Train I&R staff Initiate services in pilot states.

9. Evaluation Questions: Process 1.Was the project planning process effective? 2.Were materials (basic packet, outreach, training and procedures) developed as planned? 3.Was I&R staff training conducted as planned? 4.Were outreach procedures developed and implemented? 5.Is the phone system operating (receiving and responding to calls from callers)? 6.Was a caller resource database developed? 7.Were management materials developed and implemented?

10. Evaluation Questions: Outcomes 1.Do the users have simplified access to information and resources? 2.What is the effect of the NBIIC on pilot states? 3.How did callers learn of the helpline?

11. The logical connection of the NBIIC activities to the

12. desired outputs and outcomes is explained by the Logic Model

13. Methods Data Collection –Document review –Telephone survey with NBIIC callers (n=44, response rate 63.8%) –Telephone interviews with I&R staff in pilot states their supervisors (n=5) –Telephone interviews with NBIIC staff (n=1) Data Analysis –Descriptive analysis of quantitative data –Chi-square test of independence –Content analysis of qualitative data –Triangulation of results This study was approved by the University of Pittsburgh IRB # 0605065 The NBIIC was developed and this evaluation was supported by cooperative agreement # U17/CCU323329-02 between BIAA and the Centers for Disease Control and Prevention. For more information, contact the Brain Injury Association of America at 1-800-444-6443.

14. The NBIIC Pilot States

15. For outreach, the NBIIC distributed posters, flyers, model letters and press releases. Each state adapted the outreach procedures as appropriate. The phone system is currently operating. The volume of calls has been consistently low (4-15). A caller resource database was developed and is currently operational. The Data Element, to gather individual data, and the Monthly Report, to summarize data, were the management materials developed and implemented. They are being used by I&R staff, although there are still some clarity and electronic transfer problems. Results: Process Evaluation (cont.)

16. Results: Outcomes NBIIC callers have simplified access to resources through this system. A sizable majority felt the NBIIC met their needs in a timely fashion and provided them with easy to read material that met their expectations. The pilot states have benefited from the NBIIC in several ways. First and most importantly, the NBIIC has provided a high-quality standard structure for many aspects of their work with brain injury individuals, their families and caregivers. The standard training, manuals and guides for staff, management materials and information for callers as well as the leadership from BIAA provide this structure. Additionally, states have benefited from developing partnerships among themselves and sharing information and work procedures. Callers learned about the helpline mostly from the internet, through word of mouth and through advertisements in hospitals and rehabilitation centers.

17. Results: Table: 1 Frequencies of Positive Responses: NBIIC responses by State and Minnesota Control responses NBIIC Callers Minnesota Control Nr (%) I am at ease when I talk to the Staff37 (84.1)10 (100.0) The Staff gave me information when I needed it29 (65.9)9 (90.0) The center staff answered all my questions30 (68.2)9 (90.0) The NBIIC met my needs27 (61.4)10 (100.0) The items sent to me were easy to read37 (84.1)10 (100.0) The information in the packet met my expectations28 (63.6)10 (100.0) The items the center sent to me were easy to use32 (72.7)10 (100.0)

18. Discussion Strengths: NBIIC provides a standardized structure to the state brain injury affiliates and its workers NBIIC prepared a standard training that was valued by state I&R staff The quality of the training and the materials reflected on the callers’ responses was significantly larger for those callers that stated that they felt at ease talking with the I&R staff Using NBIIC data, the possible development of a large database of TBI callers, useful for planning and program development purposes. Weaknesses: NBIIC line competition with well established state help lines Slow NBIIC promotion, yielding low call volume Improvements to NBIIC guides and manuals, to include information on managing calls from people with severe TBI Case management materials can be more user-friendly to take less time from I&R staff

19. Users have simplified access to information: A sizable majority felt the NBIIC met their needs in a timely fashion and provided them with easy to read material that met their expectations. Pilot states have benefited from the standardized structure provided by the NBIIC and from increased communications amongst their respective staff. Callers obtained information through the internet, word of mouth and rehabilitation centers. Conclusions

20. The NBIIC should consider Extending the NBIIC to include progressively more states, so they can benefit from the standardized structure it provides. Deciding if the NBIIC phone system is to eventually replace the local state lines or if the NBIIC materials and training will be offered to the states regardless of their participation in the phone line system. Inviting state brain injury associations to use the standardized system and to produce a caller database with BIAA assistance. Improving its quality by continuing to offer states opportunities to partner and refining their materials and fine-tuning the phone system. Developing a national database of monthly data by state and state caller databases. Conducting a targeted evaluation in one year after the implementation of the recommendations. Recommendations

21. Glossary NBIIC: National Brain Injury Information Center BIIA: Brain Injury Association of America I&R: Information and Resources CDC: Centers for Disease Control and Prevention EEC: External Expert Committee TBI: Traumatic Brain Injury

22. 1.Patton, M.Q. (1997), Utilization Focused Evaluation, Thousand Oaks, CA: Sage Publications. 2.Wholey, J.S., Hatry, H.P. & Newcomer, K.E. (2004), Handbook of Practical Program Evaluation, 2nd ed. San Francisco: Jossey-Bass. 3.CDC Evaluation Working Group. Accessed on January 3, 2006. Online: http://www.cdc.gov/eval/ 4.Brain Injury Association of America. (2005). NBIIC Administrator’s guide for information and resource program. McLean, VA: BIAA. 5.W.K. Kellogg Foundation. (2001), Introduction to Logic Models, Logic Model Development Guide, Chapter 1, Battle Creek, MI. http://www.exinfm.com/training/pdfiles/logicModel.pdf, pp 1-14. http://www.exinfm.com/training/pdfiles/logicModel.pdf References