Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 EVALUATION OF US DISPUTE RESOLUTION PROGRAMS FOR ENVIRONMENTAL AND COMPLEX PUBLIC POLICY DISPUTES Andy Rowe GHK International

Similar presentations


Presentation on theme: "1 EVALUATION OF US DISPUTE RESOLUTION PROGRAMS FOR ENVIRONMENTAL AND COMPLEX PUBLIC POLICY DISPUTES Andy Rowe GHK International"— Presentation transcript:

1 1 EVALUATION OF US DISPUTE RESOLUTION PROGRAMS FOR ENVIRONMENTAL AND COMPLEX PUBLIC POLICY DISPUTES Andy Rowe GHK International ghkint.us@ghkint.comhkint.us@ghkint.com www.ghkint.com

2 2 DISCLAIMER This presentation has been prepared by the author and represents only my views. The agencies participating in or contracting the work on which this presentation is based do not necessarily subscribe to any or all of what is contained in the presentation. Andy Rowe

3 3 CHEAPER FASTER BETTER? é There appears to be more belief than evidence underlying this claim é This is really a statement about program effectiveness: Collaborative processes are more effective than a reasonable or likely alternative process éThe evaluation system being used by important U.S. federal and state agencies is starting to provide evidence for this claim of effectiveness

4 4 BACKGROUND éThe evaluation system specifically addresses environmental and complex public policy disputes éFeatures of the system: Takes a results-based accountability / performance management approach Takes an improvement orientation Addresses expected internal and external performance reporting requirements (state and federal) Self administered Has a high level of face validity for both ADR and evaluation professions Is proving to have a high level of validity and reliability éFocus on utilization. Results will: Be easy to interpret Be useful for improving performance Be easily comprehended by ADR professionals

5 5 APPLICATIONS éU.S. Federal agencies USEPA Conflict Prevention and Resolution Center US Institute for Environmental Conflict Resolution Other federal environmental agencies through contact to USIECR éU.S. State agencies Oregon Dispute Resolution Commission Florida Conflict Resolution Consortium éOther applications Base for Maryland Alternative Dispute Resolution Applied in pilot project of 9 th Circuit

6 6 A PERFORMANCE MANAGEMENT APPROACH éPrimary goal is to use evaluation as a mechanism to focus organizations on results and achieve success While satisfying evaluation goals and standards Address possible Government Performance Results Act (GPRA) requirements éJohn A. Koskinens three GPRA questions are what the evaluation attempts to address (John McLaughlin version): What is your program / agency supposed to achieve? How will you know when you have achieved this? How are you doing now?

7 7 KEY ELEMENTS IN THE APPROACH éAdopted a collaborative outcome and utilization approach Accountability for results (GPRA, Oregon Legislature) based on 3 GPRA questions Developed outcome charts for the programs to capture their program theory Confirmed use of program theory, reasonable level of standardization of practice, and concordance of practice with program theory Gained buy-in from key stakeholders to both the program theory, evaluation use and design Supported wider use of the evaluation by the organization éMonitoring & evaluation approach Standardized questionnaires almost entirely focused on the nested lower level outcomes M&E undertaken by the programs and generates reports for internal and external stakeholders

8 8 IMPLEMENTATION éQuestionnaires to parties, practitioners and project manager at conclusion of the process and a follow-up to parties at a later date Mail, web and email administration options éResponse rates are high (will exceed 70% in Oregon pilot) and 100% of practitioners (included their contracts) éInformation was systematically tested and has proved valid and reliable éMeets performance standards for the design Managers are can answer to fluent for first two questions Resource requirements = 10 hours per case for admin, analysis & reporting Culture of the organizations shifts towards results focus Internal stakeholders very satisfied, external not yet known

9 9 OUTCOME CHARTS ARE THE KEY TO THE DESIGN éSimilar to logic models / logframes but much more focused on results éOutcome charts are best developed collaboratively with the program managers / staff éIllustrations that follow are from the Public Policy Program of the Oregon Dispute Resolution Commission This is an ADR collaborative process – similar to applications in other agencies

10 10 Increased efficiency of agencies through use of ADR Decreased cost of resolving disputes Increase agency and public satisfaction with process and results of ADR OREGON PUBLIC POLICY DISPUTE RESOLUTION PROGRAM OUTCOMES ADR is successful (time and resources spent in disagreement and conflict are now redirected to more constructive purposes) ADR in State Government Oregon state agencies have practices and systems that ensure ADR is available Practice of ADR supports agency mission, and receives positive feedback Good Process Good agreements Case Management Dispute Systems Designs Mediator Roster Dispute resolution systems are improved Roster provides expedited access to qualified and appropriate providers

11 11 EXTRACTED OUTCOMES FOR OREGON CASE MANAGEMENT ADR successful (resources spent in disagreement are now used more constructively) Good agreements Good outcomes from process ADR more effective (better benefits for resources expended) than other options for this dispute All parties involved in an ADR process are satisfied that the process was fair and open. Durable and implementable agreement reached using ADR Use of ADR narrows disagreements Agreements reached with ADR are complete - there are no hard issues left or deferred. Right parties continue to be engaged, new parties added as required. Use of ADR helps adversarial parties collaborate Party interactions constructive Appropriate mediator leads ADR ADR used appropriately Design of the ADR process is appropriate for the dispute and needs of parties Right parties engaged in process ADR is used where it is the best approach for this case Stakeholder capacity to use ADR is improved through experience with this case Government decision making is improved through use of the process Non-ADR processes used where the best approach

12 12 OREGON PILOT RESULTS ADR successful (resources spent in disagreement are now used more constructively) Good agreements Good outcomes from process (7.1) ADR effective (better benefits for resources expended) than other options for this dispute (7.9) All parties involved in an ADR process are satisfied that the process was fair and open. (6.9) Durable and implementable agreement reached using ADR (7.7) Use of ADR narrows disagreements (6.4) Agreements reached with ADR are complete - there are no hard issues left or deferred. (6.0) Right parties continue engaged, required new parties added. (6.5) Use of ADR helps adversarial parties collaborate Party interactions constructive (7.8) Appropriate mediator leads ADR ADR used appropriately (7.5) Design of the ADR process is appropriate for the dispute and needs of parties (6.7) Right parties engaged in process (7.9) ADR is used where it is the best approach for this case (6.1) Stakeholder capacity to use ADR is improved through experience with this case Government decision making is improved through use of the process Non-ADR processes used where the best approach

13 13 Simplifying Complex Practices éOutcome charts seek to display the responses to the first of the GPRA questions: What is my agency/unit trying to achieve? éThe complexity of any one element of ADR practice is difficult to articulate in a single outcome but this complexity is required to respond to the second GPRA question: How will we recognize this? The mechanism I am currently using to simplify this is nested outcomes – mini outcome charts under individual outcomes éEven the nested outcomes can be at too high a level of generalization to be able to respond to the third GPRA questions How are we doing now? This is either addressed through further nesting or directly by the questions we ask

14 14 Appropriate Neutral Guides the Process Appropriate neutral – parties satisfied with neutral Appropriate neutral – neutral understands issues Appropriate neutral – manages process well Appropriate neutral – available for process Appropriate neutral – parties understand process At not time did one of the parties dominate to the detriment of the process or others The neutral helped us manage our time well. The neutral made sure that the concerns of all parties were addressed. The neutral made sure that the concerns of all parties were heard. The parties followed the direction of the neutral. The neutral ensured that all parties were fully engaged in the process. When things got tense the neutral was always able to find ways to move ahead constructively Nested Outcomes Expressed as questions Nested OutcomesCharted Outcome ILLUSTRATION OF CHARTED AND NESTED OUTCOMES

15 15

16 16

17 17

18 18 OTHER ELEMENTS IN THE EVALUATION SYSTEM éCollaborative Processes Program Intake ECR/ADR/Collaborative Processes (longer and shorter forms Conflict Assessment / Collaborative Assessment / Situation Assessment Meetings: information gathering / public input Dispute System Design éOther ADR Services Training Roster program Contract for neutral services

19 19 THE EVALUATION DESIGN IS SUCCESSFUL éHave established a program theory for ADR in environmental and complex public policy disputes éApproach is being implemented by a growing number of agencies éPositive peer external reviews (practice and evaluation) éInformation is helping program improve éInformation is addressing internal and external accountability requirements

20 20 ASSESSMENT OF THE EVALUATION APPROACH éGains Program outcomes now explicit (can evaluate) Providing information for program improvement Utilization is high Bought time to address shortcomings by providing systematic information about the achievements of ADR éShortcomings Assessment of the incremental benefits of the approach will not be clear nor indisputable Assessment of effectiveness requires additional research before evaluation can adequately cover that terrain

21 21 MAIN REMAINING CHALLENGES éIncremental Benefits Inherently difficult because of nature of many ADR cases In some settings this can be addressed without difficulty éEffectiveness Inherently difficult because of nature of most or perhaps all ADR cases Lack of useful applied research in this area éBoth incremental benefits and effectiveness are highly charged concepts in the field Many strong believers have vigorously argued the merits of ADR based on intuitive appeal, inappropriate straw dogs, limited conceptual rigor and poor evidence These are also linked to an even more charged debate between proponents of process versus proponents of agreements éThere is a compelling logical and some empirical evidence that ADR will prove to deliver incremental benefit and be an effective approach when used appropriately

22 22 CHALLENGE 1: INCREMENTAL BENEFITS éDisputes tend to be heterogeneous Arises from some or all of: characteristics of the dispute, parties, practitioner, general setting in time and place Very challenging to find a counterfactual Counterfactuals will always need to be unique to the case Disputes often engage a variety of resolution options over their life éLitigation is not the only alternative When litigation is the likely alternative we can get some reasonable information For many disputes the likely alternatives are: do nothing, legislation or administrative rule making éAddressing the challenge The strength of the program theory, fact that claims made by the program theory are not exceptional likely to should suffice as an evaluation judgment of incremental benefits

23 23 CHALLENGE 2: EFFECTIVENESS éCosts and benefits occur in three phases: Prior to the dispute resolution process. Many costs have already occurred such as fact finding, establishing the technical, legal and economic claims, identifying rights During the dispute resolution process. Main costs are time of parties, attorneys, practitioner, travel. Ability to pay can vary dramatically (e.g. dispute involving a major utility, environmental groups and individual small farmers) After the dispute resolution process. The hypothesis is that a good agreement will significantly reduce costs éShelf life of this information is relatively short éThe classes and characteristics of costs and benefits have not been enumerated, nor weights or likely ranges

24 24 ADDRESSING THE CHALLENGES éIncremental Benefits Use litigation as an alternative where feasible Undertake GOOD case studies to create illustrative comparators that can be referred to Use likely to approach in the interim éEffectiveness Case studies to establish the categories, ranges and coefficients Include limited number of cost questions in the meantime Manage the dance of doing no harm by not collecting bad cost data in the interim

25 25 MY CLAIM 1.Getting the organization to focus on results generates immediate and significant benefits a)Outcome chart program intent is clear = improved evaluability b)3 GPRA questions requirement for monitoring outcomes = performance information 2.Worst case Improved evaluation a)Utilization enhanced b)Can report on outcomes c)Credible information for program improvement 3.Better case Organizations adopt an evaluative and performance culture

26 26 CONTACT INFORMATION Andy Rowe GHK International 2707 Stratford Rd. Columbia SC 29204 (803) 782-2182 voice (803) 782-3618 fax ghkint.us@ghkint.com www.ghkint.com


Download ppt "1 EVALUATION OF US DISPUTE RESOLUTION PROGRAMS FOR ENVIRONMENTAL AND COMPLEX PUBLIC POLICY DISPUTES Andy Rowe GHK International"

Similar presentations


Ads by Google