Presentation on theme: "Measuring Interorganizational Collaboration in Communities: Baltimores Strategy to Improve Birth Outcomes as a Case Study Donna Strobino and Meredith Matone."— Presentation transcript:
Measuring Interorganizational Collaboration in Communities: Baltimores Strategy to Improve Birth Outcomes as a Case Study Donna Strobino and Meredith Matone Department of Population, Family and Reproductive Health Johns Hopkins Bloomberg School of Public Health
Gaps in Current Efforts to Improve Birth Outcomes Individual initiatives are funded inadequately and separately Individual initiatives are funded inadequately and separately Poor coordination of services Poor coordination of services Lack of adequate primary health care Lack of adequate primary health care Lack of health insurance Lack of health insurance Lack of minimum standard of care Lack of minimum standard of care Limited community mobilization to promote healthy behaviors prior to or between pregnancies Limited community mobilization to promote healthy behaviors prior to or between pregnancies
General Overview The Strategy to Improve Birth Outcomes (SIBO) is a citywide and community-led strategy Reframes the way people, providers, services, and community agencies interrelate around the wellbeing of women throughout their reproductive life cycle (preconception, pregnancy and postpartum) and their infants Takes a population-based approach to solving intractable disparities in birth outcomes
High Impact Service Areas (1) Primary health care in a medical home (7) Smoking cessation (2) Obstetric care(8) Family planning (3) Home visiting(9) Nutrition support (4) Drug and alcohol treatment(10) Breastfeeding promotion (5) Domestic violence Interventions (11) Safe sleep education (6) Mental health care
Family League of Baltimore City RFP SIBO challenges communities to mobilize around improving birth outcomes in a community and increasing demand for and utilization of the eleven high-impact services. The Family League is funding: a citywide communications campaign an evaluation and performance monitoring strategy resident and community mobilization around infant mortality and collaboratives capable of connecting all women in a community with appropriate care and services.
Five Critical Components: 1 Policy mandate emphasizing Improved birth outcomes as a priority of the Baltimore public health landscape. Funding streams to facilitate systems and services to address gaps in coverage, improve effectiveness of practices and policies, and replace inefficient implementation of existing practices with evidenced-based alternatives.
Five Critical Components: 2 Reorganize or retool health care delivery systems and practices for mothers and infants (including fathers, partners, families and communities who care for and support them). Assess and refine service provider practices and policies, especially related to risk assessment and create new operational models of service provision, where needed.
Five Critical Components: 3 City-wide social marketing intervention to influence the behavior of community residents, with a focus on expecting mothers and their partners and families, caregivers, service providers, and policy makers. Target all citizens
Five Critical Components: 4 Community-led focused initiative designed to coordinate already existing, culturally relevant, local health and social services within target areas Customized approach requiring strong community input and collaboration, leading to transferable models for other neighborhoods. NO new services
Five Critical Components: 5 A collaborative strategy driven by a set of partnership principles defining the quality and character of stakeholder interactions which guide the tenor of policies, agreements and protocols. Important in defining the community led component Applies to CIT and Steering Committee
FLBC RFP for Community Collaboratives: Must Demonstrate the Capacity to: Outreach to all pregnant women in the community to connect them with appropriate care and services Participate fully in the new citywide triage system for pregnant women into home visiting programs; Link pregnant women with appropriate supports and services (home and center based) 12 targeted communities with 3 awards of $400,000 - 600,000 for the 21 month period April 2010 – December 2011
Expected Impact in RFP SIBOs long term success: improvements in rates for pre-term birth, LBW, and deaths from unsafe sleep in targeted communities and in Baltimore City Tracking of several indicators related to the activities of collaborative teams that contribute to the achievement of the key outcomes. Success also measured by whether communities develop partnerships collaborations to assure access to the 11 high-impact services and link the various components of the health care system and human resources in the community to assure this access.
Evaluation Objectives Assess implementation of SIBO core components related to policy, services, communities, and families and individuals; Monitor implementation throughout the duration of SIBO to allow for modifications and trouble-shooting in real time (including the collaboration strategy); and Produce data to understand the impact of the first four core components on infant mortality in Baltimore City.
SIBO Evaluation Design Quasi-experimental design to evaluate changes in indicators in communities funded for the community-led initiative with the other communities with high infant mortality rates not initially funded: funded communities compared to remaining targeted communities
Data Source for Collaboratives Data from community partners about collaborations among agencies and community leaders involved in implementation of the community-led focused intervention in targeted communities
Overview Significance Measurement problem Collaboration definition, models, research Evaluation strategies for collaboration Case Study: The Strategy to Improve Birth Outcomes in Baltimore City
Significance Collaboration between health and social service agencies within a community that creates a population-based system of care are ideal because complex issues are inseparable from social and physical determinants of health, which cluster by community.
Significance Proposed benefits of collaboration: Promote cost-effectiveness Reduce duplication of services Improve communication Comprehensive service provision Increase community capacity What is the evidence to support these claims?
A Measurement Problem: What is collaboration? Conceptually: logical and feel-good results indicator Reality: plagued with measurement issues
Collaboration Definition Collaboration occurs when a group of autonomous stakeholders of a problem domain engage in an interactive process, using shared rules, norms, and structures, to act or decide on issues related to that domain. Wood and Gray 1991 Collaboration in public health… A relationship between two organizations that involves exchanging information and sharing resources in order to coordinate services for mutual benefit and to achieve common goals, such as facilitating care of treatment of clients. Friedman et al. 2007
Collaboration Models Level of Collaboration models Attempt to classify the stage, or intensity, of relationships between organizations Indicators generally include purpose (goals), structures (central body, leadership) and behaviors (communication, budget)
State of the Research Most community-level research to date has focused on costs and benefits of collaboration to organizations and/or accounts of the collaboration formation process Stressors of collaboration in public sector Characterization of nonprofit organizations likely to participate in formal collaboration Effective arrangements of collaborative efforts
State of the Research Collaboration research in community health promotion, specifically grant-funded initiatives and public and nonprofit organizations, is underdeveloped Done little to advance measurement tools
Evaluation Strategies Social Network Analysis Survey Thomson, Perry, and Miller Method Qualitative Analysis
Social Network Analysis Social network analysis is a method of studying the linkages among individuals, groups, organizations, and other units (termed actors) Patterns of relationships are assessed and used to understand the structure, function, and performance of a network
SNA in Public Health Broad applications: Transmission networks: disease and information Social networks: social support and social capital Organizational networks: health systems
SNA Methodology: Respondents Create sampling frame of organizations of relevance in evaluation Known upfront, or Formative research: snowball sampling, etc. Designate organization representatives: individual or group Validity **This methodology is applicable to all evaluation methods
SNA Content Rating scales of relationship type or intensity Questionnaires often draw heavily from the collaboration models for indicators of relationships
Survey Content Measures more variable in type and complexity Like SNA, can draw heavily from the collaboration models for indicators of linkage levels or, Frequency of action measures e.g. number of communications in the past year
Survey Data Longitudinal Numeric Conventional statistical methods Graphical Microsoft PowerPoint: models using circles to represent partners and lines to represent linkages (Frey et al. 2003)
Thomson, Perry, and Miller 2007 Goal: to make collaboration quantifiable with an empirically tested measurement tool Built on the Wood and Gray (1991) definition Categorize collaboration into five dimensions: Governance, Administration, Mutuality, Norms, Organizational Autonomy
TPM Content 56 item closed-ended Likert scale questions on organizational attitudes and behaviors surrounding each of the five dimensions of collaboration From the 56-indicator questionnaire, 17 indicators were empirically validated in statistical structural equation modeling 17 indicators cross-validated in a 2004 study of 133 collaborators in a Family Preservation Program
TPM Data The Thomson, Perry, and Miller strategy measures solely the presence of collaboration using a fixed model of attitudinal and behavioral indicators developed as proxies for the dimensions of an operational definition of collaboration Collaboration is standardized More readily compared across initiatives
Qualitative Methodology Facilitated focus groups or in-depth interviews Standard data analysis methods
Qualitative Data Formative Who are the players? Process/Outcome Lists of collaborators, collaborative process info Contextual information: costs/benefits, barriers/facilitators to collaboration Mixed-method approaches In addition to SNA, survey, or TPM Validate numeric data Provide explanatory data
Degree of Complexity: instrument development, data analysis techniques and software, and required expertise Cost: material and non-material resources required by the data collection strategy, and the associated burden on the research team and respondents Evaluation Strategies Matrix
The Strategy to Improve Birth Outcomes in Baltimore City
Choosing an Evaluation Method Data Ideal…and, Realistic Timeframe Degree of complexity Cost
Data Matches ideal data Timeframe Case studies of similar scale Complexity SNA Evaluator on evaluation team Cost Software for large analyses Quick rating-scale questionnaire administered at time of other evaluation activities (e.g. community surveys) SIBO Community Collaboration Component Proposed Evaluation Design
Cognitive Mapping Formative Who are the players? What should the system look like? What should the exchanges between agencies look like? 3 Perspectives Core Implementation Team Service Providers Steering Committee
Cognitive Mapping 1.Inform sampling frame 2.Inform data instrument construction 3.Inform Technical Assistance Toolkit for Collaboration