Enacting Multiple Strategies and Limiting Potential Successes: Reflections on Advocacy Evaluation, Competing Objectives, and Pathways to Policy Change Presentation to American Evaluation Association Conference November 2011 Jacqueline Berman, Mathematica Policy Research, Inc. Hannah Betesh, Social Policy Research Associates

Presentation Overview Paper Question and Hypothesis Overview of California Forward The Challenge Evaluation Design and Methodologies Research Questions Logic Models and Project Phases Design and Methodologies Performance Indicators Lessons Learned About Evaluating Advocacy Efforts Implications for Advocacy Evaluation Design

Paper Question and Hypothesis

Paper Question and Hypothesis Question: What evaluation designs and methodologies can be used to conduct a comprehensive evaluation of advocacy efforts in real time? How to decide which elements to prioritize in formative evaluations given time constraints and time lags in advocacy evaluation? Hypothesis: Tracking change in advocacy evaluation, especially in real time, is improved when multi-level participatory logic modeling, defined performance measures, and measures and methods to track social capital are all included and used to provide both summative and formative findings.

Overview of California Forward

Overview of California Forward, 2008-2010 Created out of California’s political stalemate and economic challenges (2008) Formed from four existing civic organizations and funded by 5 major CA Foundations Embraced a bipartisan, issue-specific, project-based, data- driven approach to systemic reform of California’s governance systems Sought to utilize lessons from past reform efforts & existing governance and fiscal reform expertise

Overview of California Forward, 2008-2010 Informed by key principles in order to reform state governance systems Transparent Citizen-informed Evidence-based Effective Pursued several existing and planned reform projects Redistricting Reform Budget and Fiscal Reform Plans for Local Governance Reform Plans for Statewide Dialogue

The Challenge

The Challenge Faced by California Forward: Faced by the evaluation: Build a broad coalition of support for change while also Enacting bold enough reforms to engender real change Faced by the evaluation: Identify impacts without an experimental design Track and analyze multiple types of data collected by the advocacy organization and the evaluation Identify findings able to create better advocacy strategies Feed relevant information back to CF and funders on a time frame short enough to affect on-going advocacy efforts 9 9

Evaluation Design and Methodologies 10

Evaluation Research Questions How has California Forward sought to Frame the reform agenda in California? Enact key governance reforms: supporting their articulation, acceptance, and passage? Support the implementation of key governance reforms? Build networks to support sustained change? What progress and outcomes has California Forward been able to realize toward Framing the reform agenda? Enacting reform? Implementing reform? Building networks to support change?

Organizational & Evaluation Logic Models Worked with CF and funders to develop logic models that mapped organizational and project concepts onto anticipated outcomes and desired change Developed both an organizational and individual project logic models Created an evaluation logic model laid over the CF organizational model in order to track all project phases/activities, planned outputs, and any emergent outcomes

California Forward Project Phases CF develops and pursues each reform project in overlapping phases: Phase 1: Feasibility and Design Phase 2: Public Engagement Phase 3: Advocacy Campaign Phase 4: Support for Reform Implementation Phase 5: Assessment of Next Opportunities

Evaluation Design and Methodology Challenge of capturing real time institution building, real time advocacy efforts, and real time outcomes and influence Challenge of including a very large and varied set of stakeholders and potential inputs in a short period Identification of diverse methods able to address these challenges and capture this multiplicity Interviews and snowball sampling  Document review  Observations of CF events  Independent tracking of media and blogs  SNA  Web analytics  Administrative data analysis  Survey of stakeholders Importance of qualitative data analysis software to Deal with such diverse and voluminous data Render analysis systematic, transparent, and ‘rigorous’ Create intermediate analysis steps, especially post-coding memos 14

Sample Progress by Indicator 15

Sample Progress by Indicator 16

Lessons Learned About Evaluating Advocacy Efforts 17

Lessons Learned About Evaluating Advocacy Efforts Accommodate a rapidly evolving policy environment Adjust to strategic shifts in advocacy campaigns that seek to take advantage of evolving policy environment Employ clear organizational logic models that help anticipate points in time when shifts might emerge Recognize limitations in methods and adjust evaluation plans to them: SNA, survey, policy scoring Evolve a set of strategic feedback points that allow advocacy orgs. to put evaluation findings to work Retrospect certain evaluation findings while maintaining a prospective focus

Implications for Advocacy Evaluation Design 19

Implications for Advocacy Evaluation Design Include key data at a strategic distance from the client: snow ball sampling, blogs, Google Alerts Identify threats to evaluability: available data systems Develop strategies and tools to operationalize recommends in real time: indicators and policy scoring Consider pathways for policy change Role of evaluation in strengthening advocacy and reform efforts to realize change Importance of opportunities and mechanisms to feedback and integrate evaluation findings and implement recommendations 20

For Further Information Please contact Dr. Jacqueline Berman, jberman@mathematica-mpr.com Hannah Betesh, hannah_betesh@spra.com Manuscript (forthcoming) Enacting Dual Strategies and Limiting Potential Successes in Real Time: Reflections on Advocacy Evaluation, Competing Objectives, and Pathways to Policy Change, 2011.