Strategic Planning Session David Rudder, Ph.D. Rudder Consultants, LLC. May 17, 2006

EVALUATING THE EFFECTIVENESS OF NONPROFIT ORGANIZATIONS Chapter 14 Vic Murray

What is Evaluation and why is it important?  Evaluation is the process of gathering information on the results of past activities for the purpose of making decision about them.  Organizational effectiveness evaluation (OEE) occurs when this process is applied to assessing the state of the organization as a whole. Typically, this refers to how well it is achieving its stated mission and involves looking at goal attainment as well as how efficiently it has performed.

What is Evaluation and why is it important?  The focus of most evaluations is on the organization, but much still boils down to looking at the results of the actions of many units and individuals.  Increasing “accountability,” refers to the belief that nonprofits, and the people who run them, should be more accountable to those they are created to serve and those who provide the money to operate them. This is the primary cause of the growing interest in organizational evaluation.

Politics  Politics is inevitable in evaluation because there is so much room for subjectivity that differences can easily arise between the parties involves. Key parties are:  Evaluators  Evaluates  Other interested stakeholders

Stages of the Evaluation Process  The Design Stage – What is the purpose of organizational evaluation? What exactly will be measured – inputs, activities or processes, outputs or outcomes?  Implementation Stage – How will the information be gathered?  Interpretation Stage – What will be considered a “success” or a “failure”? If an evaluation measure reveals problems, there is a question of drawing conclusions about why these occurred in order to make decisions about the future.  Application Stage – How will the information be used in subsequent decision making?

Problems with Evaluations  Works best when the measurements can be compared to clearly stated goals, objectives, or standards that a given organization is trying to achieve.  Often when goals are vague and ambiguous, the development of valid measures of these kinds of things is technically challenging, costly, and subjective.  A design that is measuring one level of an organization but generalizing to another.

Problems with Evaluations  Even when everyone focuses on outcomes and agrees on what should be evaluated, there are inevitable difficulties over the extent to which outcome measures really capture the goals they are intended to measure.  Most evaluation systems are unable to provide conclusive analyses of why the results came out as they did. Most outcomes have multiple causes, and opinions can easily differ over which are the most important.

The United Way Approach  The outcome information is intended to be used by UW to help member agencies improve program performance, to identify and achieve UW priorities (funding allocation criteria), and to broaden the base of financial and volunteer support.  Implementation of the outcome measurement system is divided into six stages: 1) Building agency commitment and clarifying expectations 2) Building agency capacity to measure outcomes 3) Identifying outcomes, indicators, and data collection methods 4) Collecting and analyzing outcome data 5) Improving the outcome measurement system 6) Using and communicating outcome information

United Way Continued  Agencies are not expected to establish targets until they have at least one year of baseline data.  The system discourages the use of benchmark-based relative standards or those that involve comparison with similar programs that are considered exemplary until accurate outcome data are available.  It is generally understood that in the first few years of an outcome measurement system, the data often say more about what is wrong with the evaluation system than about what is taking place in the programs.

The Balanced Scorecard  A multiattribute system for conceptualizing and measuring performance. It assumes that the primary goal of a business is long run profit maximization. It argues that this will be achieved through a “balanced scorecard of performance attributes” grouped around four perspectives”: 1) The funders perspective, measuring various financial performance indicators or primary interest to shareholders 2) The client perspective, comprising measures of client satisfaction 3) The internal business perspective, which means internal efficiency and quality 4) The innovation and learning perspective, which attempts to measure the organization’s ability to adapt to changes required by a changing environment In the case of nonprofit organizations, their mission statement becomes the endpoint to be reached through these perspectives. The process starts with defining what that is and identifying outcome indicators that will reveal the extent to which it is being achieved.

CCAF/FCVI Framework  Puts forward 12 “attributes of effectiveness,” suggesting that organizations can be audited in terms of how well they manifest these attributes. They are: 1) Management direction 2) Relevance 3) Appropriateness 4) Achievement of intended results 5) Acceptance 6) Secondary Impacts 7) Costs and Productivity 8) Responsiveness 9) Financial Results 10) Working environment 11) Protection of Assets 12) Monitoring and Reporting

Conclusion  There is no tried and tested evaluation system that can be applied by most nonprofit organizations to provide a valid picture of how well the organization is performing.  We should focus on trying to improve the dialogue around the evaluation process.  If a prior relationship of trust does not exist before evaluation begins, it must consciously be worked on as the process is developed, particularly those who are to be evaluated.

Conclusion Continued All parties involved (evaluator, evaluates, and stakeholders) should have a voice in deciding the following six questions: 1) What is the purpose of the evaluation? 2) What should be measured? 3) What evaluation methods should be used? 4) What standards or criteria should be applied to the analysis of the information obtained? 5) How should the data be interpreted? 6) How will the evaluation be used?