2About FSG and Shared Measurement Nonprofit consulting and research firm founded in 2000Staff of 120 in Boston, San Francisco, Seattle, Washington DC, GenevaSuccess in strategy and evaluation with over 400 foundations, corporations, and nonprofitsThought leader: articles published in HBR, SSIR, American Journal of Evaluation including Collective Impact (SSIR 2010)Breakthroughs in Shared Measurement and Social Impact (funded by Hewlett Foundation) published in 2009: examined 20 approaches to performance, outcome, impact measurementFSG is driven by the same passion that drives our clients: a passion for greater social impact using data as a key lever
3The Premise: There Are Several Types of Problems SimpleComplicatedComplexRight “recipe” essentialGives same results every timeExample: Baking a cake“Formulas” neededExperience built over time and can be repeated with successExample: Sending a rocket to the moonNo “right” recipes or protocols Outside factors influence Experience helps, but doesn’t guarantees successExample: Raising a childTraditional approach in social sector has been to treat problems as simple or complicatedSource: Adapted from “Getting to Maybe”
4Current approaches not conducive to complex problems: Traditional Ways of Approaching Social Change Are Not Working to Address Our Toughest ChallengesCurrent approaches not conducive to complex problems:Funders select individual grantees that offer the most promising solutionsGrantees work separately, compete to produce the greatest independent impactEvaluation attempts to isolate a particular grantee’s impactLarge scale change assumed to depend on scaling a single organizationCorporate, government, philanthropy and nonprofit sectors often disconnected from efforts of foundations and nonprofitsIsolated Impact
5Imagine a Different Approach Isolated ImpactShift mindset from a “technical” to an “adaptive” approach:Understand that social problems – and their solutions – arise from interaction of many organizations within larger systemLarge scale impact depends on increasing cross-sector alignment and learning among many organizationsNonprofits, government, philanthropy and corporates actively coordinate their action and share lessons learnedProgress depends on working toward the same goal and measuring the same things – SHARED MEASUREMENTCollective Impact
6Benefits of Using Shared Measurement Shared Measurement Is a Critical Piece of Pursuing a Collective Impact ApproachDefinitionIdentifying meaningful common metrics for tracking progress toward a common agenda across organizations, and providing scalable platforms to share data, discuss learnings, and improve strategy and actionBenefits of Using Shared MeasurementTracking Progress Toward a Shared GoalEnabling Coordination and CollaborationLearning and Course CorrectionCatalyzing Action in the FieldAudience engagement activity: Which of these benefits is most important for your organization? (hold up a finger) Are there other benefits?
7Developmental Phases in Creating a Shared Measurement System 123DesignDevelopDeployShared vision for the system and its relation to broader goals, theory of change or roadmapView of current state of knowledge and dataGovernance and organization for structured participationIdentification of metrics, data collection approach, including confidentiality/ transparencyDevelopment of data collection tools and technology platformRefinement and testing of platform and toolsStaffing for data management and synthesisLearning forums and continuous improvementOngoing infrastructure supportImprove system based on a pilot, review, refinement, and ongoing evaluation of usability and impact
9Shared Measures for 100Kin10 Partner Organizations AdvanceDevelop & ImproveInductPlace/HirePrepareRecruitSTEM Teacher ExperienceDRAFTExcellence Outcomes (quality of teachers)Candidates demonstrate:Academic proficiencyLeadership potentialAppropriate mindset to succeed in STEM classrooms and school environments*Candidates demonstrate basic:Instructional practiceSTEM content knowledgePedagogical content knowledgePreparedness for the context in which they will teachQualified STEM teachers are:Placed/hiredPrepared for the context in which they are placed/hiredNew STEM teachers demonstrate improved STEM-related:Instructional practicePedagogical content knowledgeSTEM Content knowledgeStudent engagementSTEM teachers demonstrate improved STEM-related:Instructional practicePedagogical content knowledgeSTEM Content knowledgeStudent engagementStudent achievementSTEM teachers:Take on instructional leadership rolesMove into administrative rolesEngage more deeply with community stakeholdersInfluence peers and colleagues* Mindset persists throughout STEM teaching career
10Shared Measures for Portland Metro STEM Partnership
11Ways Network Partners Use Shared Measurement System To Learn with and from Peer Programs and OrganizationsTo Inform Internal Decision-making About PracticesTo Identify Practices of High-performing OrganizationsTo Identify Other Relevant PartnersTo Uncover Areas Where Further Data or Research Is Needed
12Challenges in Developing and Implementing Shared Measurement Systems Difficulty in coming to agreement on common outcomes and indicatorsConcerns about relative performance / comparative measurement across providers working in the same spaceLimited capacity (time and skill) for measurement and data analysis within participating organizationsAligning funders to ask for the common measures as part of their reporting requirementsTime and cost of developing and maintaining a system, both for human capital and technologyAudience engagement: Talk with a partner – which of these challenges resonate with you the most?
13Critical Factors in the Development of Shared Measurement Systems Help Overcome These Challenges Effective Funding RelationshipsStrong leadership and substantial funding throughout multi-year development periodIndependence from funders in devising indicators and managing the systemBroad and Open EngagementBroad engagement in design process by many organizations in field, with clear expectations about confidentiality/ transparencyVoluntary participation open to all relevant organizationsInfrastructure for DeploymentEffective use of technologyOngoing staffing to provide training, facilitation, and to review accuracy of dataPathways for Learning and ImprovementTesting and continually improving system through user feedbackFacilitated process for participants to gather periodically to share results, learn from each other, and coordinate effortsSource: Breakthroughs in Shared Measurement and Social Impact, FSG, 2009
14FSG Lessons Learned in Implementing Shared Measurement Systems Don’t Underestimate the Value of Partner EngagementProceed IterativelyFocus on Learning and UseAddress Both Technical and Political ChallengesEnsure Long-term Funding and SustainabilityA well designed and structured Shared Measurement effort will pay dividends for years to come