Presentation on theme: "The challenges of managing and evaluating partnered programs, from program theory to implementation PADM 522—Fall 2011 Dr. Mario A. Rivera Lecture 2."— Presentation transcript:
The challenges of managing and evaluating partnered programs, from program theory to implementation PADM 522—Fall 2011 Dr. Mario A. Rivera Lecture 2
The changing nature of public program management and service delivery We have moved from hierarchy (command and control within organizations) to markets and market-modelled management (the “New Public Management”) to collaborative network management organized around public-private and cross-sector partnerships Move toward recognizing a plurality of stakeholders for virtually any program Move from governments to governance “Public management is getting things done through other organizations” (Metcalfe & Richards, 1990, p220)
Networked or partnered programs It has become a critical managerial ability and an essential part of the public manager role to be able to create and leverage participation in network-based programs. Pertinent are such organizational processes as increased internal- external linkages and connectivity; partnerships, alliances, and coalitions; cross-boundary teams; interdependencies; strategic alliances (with agencies which are at times competitors, at times collaborators), involving people from disparate venues and agencies; building consensus; ensuring that contractors are accountable; encouraging the involvement of stakeholders; participating in public dialogue; creating a joint knowledge base.
Collaborative Relationships Some are informal, essentially inter-personal, one-to-one relationships across agencies More formalized, contractual collaborative relationships, as well Partnerships between and among organizations mediated by individual brokers and ‘boundary-spanners’ Strategic alignment along interorganizational lines is crucial Collaborative networks require holistic analysis, consistent with systems theory (Chen, Chapter 1). While it is important to fashion strategic consensus, one needs to be careful with the insular tendencies of networks (Rivera-Rogers) One also needs to balance participatory management and evaluation with direction sufficient to a particular program.
Systems complexity and systems-level interventions System dynamics are inherently complex Constantly changing; Governed by feedback; Non-linear, history-dependent; Adaptive and evolving; Characterized by trade-offs; Characterized by complex causality—coordination complexity, sequencing complexity, causal complexity due to multiple actors and influences, etc. There’s too much focus in evaluation on a single intervention as the unit of analysis; Understanding connectivity between programs is important; Many complex interventions require programming (and therefore also evaluation) at multiple levels, e.g., at the community, neighborhood, school and individual level; Multilevel alignment is required across interventions.
Network Theory Derived from ‘group theory’ in sociology and anthropology, network theory describes intricate sets of formal & informal relationships which create both functional and dysfunctional forms of complexity; Actor autonomy is pulled by forces of inter-dependency, and of power- and resource-dependency Networks involve the exchange of information and resources Involves complementarity of roles, functions, capabilities Reflects and grows out of the ‘hollowing out’ of the state Entails shared knowledge & interorganizational learning From relatively stable internal networks to dynamic and changeable trans-organizational networks; networks may institutionalize over time (e.g., the United Way).
Network questions Why enter network relationships (partnerships, alliances, collaborative networks? What resources become available when you enter a network? What position is held in the network by one’s own organization? How easy is access? How do members view each other? What is to be gained by joining a network? Is there true partnership or mere agglomeration? How do you derive value from network-building activities? Increased visibility? Fundraising with more connections? Empowered policy advocacy? How much do you give up building the network rather than your organization? What are the tradeoffs?
Managing Interorganizational Relationships Managing across jurisdictions Managing different stakeholders Managing implementation Managing exchanges Managing incrementally Managing negotiations Managing multiple network functions: Diffusion: Rapid spread of ideas, policy innovations, via influential opinion leaders and early technology adopters (Rogers) Alignment: Aligning ideas, efforts toward a common set of goals Mobilization: Reaching stakeholders and motivating them to act Exchange: Sharing of information, knowledge, other resources Advocacy: development of a collective voice for change Delivery: Bringing resources and assistance to increase capacity
Program partnerships as types of network Leverage expertise and strength of each program to deliver improved outcomes Often requested in grants Like individual members of teams, agency network members bring to the given enterprise their own network resources Examples of program partnerships? When you have worked in one, have you felt as though collaboration was real? Is it possible to develop partnership-wide plans (strategic, functional—fund-raising, collaborative research)? What are the challenges of evaluating partnerships and other networks?
Defining characteristics of effective inter- organizational and inter-sectoral networks They are integrated (some with substantial centralization, even if the centrality of different actors changes and the identities and roles of central actor(s) change; They are most likely to be found in situations when fiscal and human resources are relatively plentiful and where information asymmetries are minimal; They are most likely under conditions of stability in intergovernmental and interorganizational fiscal relations; They require social capital (i.e., trust-based relationships) as much as they do fiscal and human capital; They have or come to have shared values & commitments; They are capable of generating aligned, conjoint strategies
Performance measurement and program evaluation in networks Tracking and analyzing performance indicator data enables evaluation and accountability of collaborative work, and is therefore a critical contributor to the capacity to manage for results across agencies. It also increases the legitimacy of collaboration, introducing a new level of accountability. There needs to be reasonable agreement on both goals and the way to best measure and evaluate the extent of goal attainment. There also needs to be agreement as to whom a program or policy initiative is accountable. Most existing performance measures assess the accomplishments of individual agencies, yet most meaningful policy results require interagency collaboration— participatory, collaborative planning, implementation, and evaluation. We need new tools to assess networked programs when there are multiple agencies involved in program design and implementation.
Program theory: what is it? A plausible and sensible model of how a program is supposed to work. L. Bickman, 1987 A systematic delineation of cause and effect relationships. M. Scheirer, 1987 Identification of resources, activities, and outcomes of a program and the causal assumptions that connects these. J. Wholey, 1987 Scientific explanations of how social change is possible. C. McClintock, 1987 A specification of what must be done to achieve the desirable goals, what other important impacts may also be anticipated, and how these goals and impacts would be generated. H. Chen, 1990, cited in our text, p. 16.
How should theory be constructed? There are essentially two camps, and Chen’s “both/and” approach: Program theory should be constructed by key program stakeholders, in a fully participatory fashion (Wholey) Program theory should stress social and behavioral science theory and knowledge (Rossi) Program theory should be based on stakeholder knowledge (stakeholder theory) as well as science or social science theory (Chen)
Prescriptive theories How to go about solving the problem or bringing about change; how should activities be constructed? How should program interventions be implemented? What is the causal connection to behavioral or social or other change?
Chen’s model and partnerships Chen divides program theory into action into two component parts: An action model and a change model. The action model incorporates both program ecological context and inter-agency collaboration. Some programs require collaborative partnership. Using the example of a spouse abuse program (p. 26), Chen points out that it would fail if it lacked a working relationship with the courts, police, and community partners including advocacy groups. It is therefore essential to craft strategies for collaboration with other agencies. Partnered programs may have different program theories and models of change at work, or they may operate on different concepts of a single set of theories and models.
Evaluating Partnerships What do we most want to know? 1.That the partnership has been effective in achieving its aims 2.That the partners have all benefitted from their involvement 3.That the partnership approach was or is the best way to do it What are the key underlying questions? We need to understand the underlying elements that create partnership but that are not obvious and that we cannot readily appreciate In particular, we need to understand the relationship between component elements of partnership function the synergies of partnership-level functioning
AN EMERGING EVALUATION APPROACH FOR PARTNERSHIPS: 1 IMPACT ASSESSMENT 2. PARTNERHIP RELATIONSHIP and DEVELOPMENT REVIEW 3 EVALUATING PARTNERING AS A MECHANISM
We evaluate partnerships so as to understand: Transaction costs reduction Value added Sustainability of outcomes Strategic influences Systemic impact In sum, whether a partnership approach has been or is to be ultimately better than the next best alternative Evaluating partnering as a mechanism
One formula for assessing the ‘added value’ of a partnership AV = (OP + SC) – (RC + NA + EC + OC + FC) Key: AV = A dded V alue of a Partnership OP = O utcomes of P artnership SC = S ocial C apital RC = R esources C ontributed NA = N et Benefit of the Next Most Likely A lternative EC = E nvironmental C ontributions OC = O pportunity C osts (e.g., time spent) FC = F acilitation C osts
To reveal: Value created by the partnership for partner organizations (and other stakeholders) –expected –unexpected –potential Extent of strategic alignment and goal congruence among partners Degree of effectiveness/efficiency/impact Relative level of influence (sectoral/strategic) Relative impact of different partners on program outcomes—can this be determined from program data? Partner relationship evaluation
Complex effects chain in partnered programs Partners 1, 2, 3, etc. Shared Common Outcomes Attribution difficulties; transparency & accountability challenges
This form of evaluation links to: A review of partnering principles in practice (i.e. is the partnership equitable, transparent and mutually beneficial?) Whether the partnership is achieving the individual goals and satisfying the underlying interests of partner organizations or not—along with partnership-level goals and corporative interests Exploring whether the partners have made maximum use of the range of resources available Whether the partnership could work better–if so, how? Whether the partnership could do something quite different–if so, what?
Other elements of partnership evaluation Any truly valid and effective evaluation of a partnered program always needs to: Involve all partners and key stakeholders in design and data collection Include a genuine feedback loop so that the process effectively informs the development of the partnership Find a good balance between external projection of the partnership as a corporative entity, and the internal experience of the partnership—strategic dynamics of partnership identity
General principles of evaluation Needs → Inputs → Outputs → Impacts: Logic models in often static sequence, focused on single agency actors remains the usual focus of most evaluation questions Identifying causal links and quantifiable indicators is stressed Evaluations should specify minimum requirements concerning the representativity of the data to be collected or analyzed, and state how the analysis of causality will be approached
Partnered, participatory evaluation is complex, requiring a correspondingly complex evaluative approach that can adequately deal with multiple layers of causality and dynamic effects. Case study is an approach to evaluation (as such) that can capture such complexity. Kushner: “ This capacity for case study to both provide insight into the unique, but also to generalise helped the evaluation community develop a methodology and a practice freed from the normal constraints of academic and professional disciplines. For example, the study of curriculum and pedagogical change in one school, say, provides a platform for generalising to other schools, curriculum projects and pedagogical strategies. But, beyond that, that same school case study provided insights into, say, the link between professional and organisational development; the integration of knowledge and action; the tension between policy, institutional management and professional practice; or the grounding of professional action in biographical experience. These insights were transportable to other contexts...”
Best practice guidelines Benefits of the comparative case study framework Definition and selection Pluralistic methods to identify causal structure and process Interpretation by iterative evaluation Validation The trouble with case studies is or has to do with Generalizability Difficulty Comprehensibility Validity Cost
Sources 1.Huxham (eds.) 1996. Creating Collaborative Advantage. London: Sage 2.Kickert, W., Klijn, E-H. & Koppenjan, J.F.M. [eds.] 1997. Managing Complex Networks: Strategies for the Public Sector. London: Sage 3.Marsh, D. & Rhodes, R. (eds.) 1992a. Policy Networks in British Government. Oxford: Clarendon. 4.Richardson, J. & Jordon, G. 1979. Governing Under Pressure. Oxford: Martin Robertson. 5.Rose, A. & Lawton, A. 1999. Pubic Services Management. Harlow: Prentice Hall. 6.Rhodes, R. 1997. Understanding Governance. Buckingham: Open University Press 7.Rhodes, R. 1996. ‘The new governance: governing without government.’ Political Studies, Vol.44, pp.652-67.
Program theory sources L. Bickman (ed.) Using program theory in evalulation. New Directions for Program Evaluation, no. 33. San Francisco: Jossey- Bass, Spring, 1987. (multiple authors) Chen, H-T. Issues in constructing program theory. New Directions for Program Evaluation, no. 47, Fall 1990. Chen, H-T and Rossi, P. H. The theory driven approach to validity. Evaluation and program Planning, 1987, 10, 95-103. Patton, M. Q. A context and boundaries for theory-driven approach to validity. Evaluation and Program Planning, 1989, 12, 375-377. Wilder Research Center. Program theories and logic models. St. Paul, MN: Wilder Research Center. http://chicano.umn.edu/pdf/resourcesProgram%20Theories%20Exa mple%20Logic%20Model.pdf http://chicano.umn.edu/pdf/resourcesProgram%20Theories%20Exa mple%20Logic%20Model.pdf