1. Enacting Multiple Strategies and Limiting Potential Successes: Reflections on Advocacy Evaluation, Competing Objectives, and Pathways to Policy Change
Presentation to American Evaluation Association Conference
November 2011
Jacqueline Berman, Mathematica Policy Research, Inc.
Hannah Betesh, Social Policy Research Associates

2. Presentation Overview
Paper Question and Hypothesis
Overview of California Forward
The Challenge
Evaluation Design and Methodologies
Research Questions
Logic Models and Project Phases
Design and Methodologies
Performance Indicators
Lessons Learned About Evaluating Advocacy Efforts
Implications for Advocacy Evaluation Design

3. Paper Question and Hypothesis

4. Paper Question and Hypothesis
Question: What evaluation designs and methodologies can be used to conduct a comprehensive evaluation of advocacy efforts in real time? How to decide which elements to prioritize in formative evaluations given time constraints and time lags in advocacy evaluation?
Hypothesis: Tracking change in advocacy evaluation, especially in real time, is improved when multi-level participatory logic modeling, defined performance measures, and measures and methods to track social capital are all included and used to provide both summative and formative findings.

5. Overview of California Forward

6. Overview of California Forward, 2008-2010
Created out of California’s political stalemate and economic challenges (2008)
Formed from four existing civic organizations and funded by 5 major CA Foundations
Embraced a bipartisan, issue-specific, project-based, data- driven approach to systemic reform of California’s governance systems
Sought to utilize lessons from past reform efforts & existing governance and fiscal reform expertise

7. Overview of California Forward, 2008-2010
Informed by key principles in order to reform state governance systems
Transparent
Citizen-informed
Evidence-based
Effective
Pursued several existing and planned reform projects
Redistricting Reform
Budget and Fiscal Reform
Plans for Local Governance Reform
Plans for Statewide Dialogue

8. The Challenge

9. The Challenge Faced by California Forward: Faced by the evaluation:
Build a broad coalition of support for change while also
Enacting bold enough reforms to engender real change
Faced by the evaluation:
Identify impacts without an experimental design
Track and analyze multiple types of data collected by the advocacy organization and the evaluation
Identify findings able to create better advocacy strategies
Feed relevant information back to CF and funders on a time frame short enough to affect on-going advocacy efforts 9 9

10. Evaluation Design and Methodologies10

11. Evaluation Research Questions
How has California Forward sought to
Frame the reform agenda in California?
Enact key governance reforms: supporting their articulation, acceptance, and passage?
Support the implementation of key governance reforms?
Build networks to support sustained change?
What progress and outcomes has California Forward been able to realize toward
Framing the reform agenda?
Enacting reform?
Implementing reform?
Building networks to support change?

12. Organizational & Evaluation Logic Models
Worked with CF and funders to develop logic models that mapped organizational and project concepts onto anticipated outcomes and desired change
Developed both an organizational and individual project logic models
Created an evaluation logic model laid over the CF organizational model in order to track all project phases/activities, planned outputs, and any emergent outcomes

13. California Forward Project Phases
CF develops and pursues each reform project in overlapping phases:
Phase 1: Feasibility and Design
Phase 2: Public Engagement
Phase 3: Advocacy Campaign
Phase 4: Support for Reform Implementation
Phase 5: Assessment of Next Opportunities

14. Evaluation Design and Methodology
Challenge of capturing real time institution building, real time advocacy efforts, and real time outcomes and influence
Challenge of including a very large and varied set of stakeholders and potential inputs in a short period
Identification of diverse methods able to address these challenges and capture this multiplicity
Interviews and snowball sampling  Document review  Observations of CF events  Independent tracking of media and blogs  SNA  Web analytics  Administrative data analysis  Survey of stakeholders
Importance of qualitative data analysis software to
Deal with such diverse and voluminous data
Render analysis systematic, transparent, and ‘rigorous’
Create intermediate analysis steps, especially post-coding memos 14

15. Sample Progress by Indicator15

16. Sample Progress by Indicator16

17. Lessons Learned About Evaluating Advocacy Efforts17

18. Lessons Learned About Evaluating Advocacy Efforts
Accommodate a rapidly evolving policy environment
Adjust to strategic shifts in advocacy campaigns that seek to take advantage of evolving policy environment
Employ clear organizational logic models that help anticipate points in time when shifts might emerge
Recognize limitations in methods and adjust evaluation plans to them: SNA, survey, policy scoring
Evolve a set of strategic feedback points that allow advocacy orgs. to put evaluation findings to work
Retrospect certain evaluation findings while maintaining a prospective focus

19. Implications for Advocacy Evaluation Design19

20. Implications for Advocacy Evaluation Design
Include key data at a strategic distance from the client: snow ball sampling, blogs, Google Alerts
Identify threats to evaluability: available data systems
Develop strategies and tools to operationalize recommends in real time: indicators and policy scoring
Consider pathways for policy change
Role of evaluation in strengthening advocacy and reform efforts to realize change
Importance of opportunities and mechanisms to feedback and integrate evaluation findings and implement recommendations 20

21. For Further Information
Please contact
Dr. Jacqueline Berman,
Hannah Betesh,
Manuscript (forthcoming)
Enacting Dual Strategies and Limiting Potential Successes in Real Time: Reflections on Advocacy Evaluation, Competing Objectives, and Pathways to Policy Change, 2011.