1. Performance Measurement for Justice Integration Projects The Illinois and Cook County Experience Mark Myrent Illinois Criminal Justice Information Authority

2. June 20, 2005 2IL Criminal Justice Information Authority What is Performance Measurement? Performance measurement can be defined as the process of routinely measuring the outputs and outcomes produced by a project, thereby allowing one to assess the effectiveness of project investments and activities

3. June 20, 2005 3IL Criminal Justice Information Authority Why Measure Performance? Build consensus & commitment within the justice community Build consensus & commitment within the justice community Obtain & allocate resources Obtain & allocate resources Plan and manage project execution Plan and manage project execution Demonstrate success & improve accountability Demonstrate success & improve accountability

4. June 20, 2005 4IL Criminal Justice Information Authority A Method for Measuring Success Statement of business problem Statement of business problem Goals Goals Objectives Objectives  Project objectives  Functional objectives  Business objectives

5. June 20, 2005 5IL Criminal Justice Information Authority Developing Business Objectives Identify a BASIC MEASURE Identify a BASIC MEASURE Indicate DIRECTION of the measure Indicate DIRECTION of the measure Identify the OBJECT of the measure Identify the OBJECT of the measure Identify the expected VALUE of the measure Identify the expected VALUE of the measure Identify WHERE the measurement will occur Identify WHERE the measurement will occur Identify WHEN the measure will be obtained Identify WHEN the measure will be obtained

6. June 20, 2005 6IL Criminal Justice Information Authority The LOGIC MODEL APPROACH ACTIVITIES OUTCOMES INPUTS OUTPUTS INPUTS are your resources, while OUTPUTS are the product of an activity OUTCOMES are not what the program or initiative itself does, but the consequences of the program/initiative efforts -- changes in conditions, attitudes or behavior of individuals or outcomes for agencies and communities. The STRATEGIC PLAN and THEORY OF CHANGE links these together

7. June 20, 2005 7IL Criminal Justice Information Authority GOALS In order to solve these problems Activities We will do the following activities and events Targets For these people and for this amount of time Initial/Short & Intermediate Term Outcomes We will know these changes have occurred If: ___________ THEORY Of CHANGE This activity will lead to changes in these factors __________, which in turn will lead to solving these problems LONG TERM OUTCOMES We will know we are reaching our goals If: ___________ Thinking Through a Logic Model and Articulating a Theory of Change

8. June 20, 2005 8IL Criminal Justice Information Authority Developing Business Objectives Identify a BASIC MEASURE Identify a BASIC MEASURE Indicate DIRECTION of the measure Indicate DIRECTION of the measure Identify the OBJECT of the measure Identify the OBJECT of the measure Identify the expected VALUE of the measure Identify the expected VALUE of the measure Identify WHERE the measurement will occur Identify WHERE the measurement will occur Identify WHEN the measure will be obtained Identify WHEN the measure will be obtained

9. June 20, 2005 9IL Criminal Justice Information Authority The Illinois Experience: Statement of Business Problem The absence of standardized arrest booking procedures creates inconsistencies in the criminal history reporting process, which contributes to breakdowns in the quality of Illinois rap sheets.

10. June 20, 2005 10IL Criminal Justice Information Authority The Illinois Experience: Strategic Goal for Business Problem Recommend, and implement mechanisms and processes to inventory, develop, adopt, publish, disseminate, and maintain standards that apply to justice information sharing.

11. June 20, 2005 11IL Criminal Justice Information Authority Ilinois Misstep: Performance Measures from Strategic Plan Objective: By July 2003, Standards Committee to the Governing Board will identify procedures for evaluating, developing, approving, disseminating, and maintaining standards. Performance measures:  Percentage of stakeholders represented & providing input on advisory committee  Number of standards adopted by IIJIS Gov. Bd.  Percentage of IL agencies employing justice information standards

12. June 20, 2005 12IL Criminal Justice Information Authority Project Initiation Strategic Planning Tactical Planning Project Strategic Planning is only the Start of Integrated Justice Planning

13. June 20, 2005 13IL Criminal Justice Information Authority Performance measures derived from strategic planning are different from those derived from tactical planning Strategic plans are often written at a a very high level in order to convey the conceptual framework (particularly in state-level plans) Strategic plans are often written at a a very high level in order to convey the conceptual framework (particularly in state-level plans) Strategic performance measures often address pre-implementation “process” activities, thus producing a preponderance of “outputs” rather than “outcomes” Strategic performance measures often address pre-implementation “process” activities, thus producing a preponderance of “outputs” rather than “outcomes”

14. June 20, 2005 14IL Criminal Justice Information Authority New Illinois Direction: Business Objectives as Performance Measures (from Tactical Plan) Proposed Solution for Business Problem: Expand the pilot project in Kane County in which: 1. 1.Individual police departments submit data file containing arrestee charges, demographics, and tracking number to state CHRI repository 2. 2.After file is posted to CHRI database, it is instantaneously sent electronically to court clerk’s office, along with fingerprint ID number, and is downloaded to clerk’s database, without the need for additional data entry.

15. June 20, 2005 15IL Criminal Justice Information Authority Developing Business Objectives as Performance Measures 1. Identify a basic measure: Actual Measure: Percentage of offender records in CHRI system having final dispositions

16. June 20, 2005 16IL Criminal Justice Information Authority Developing Business Objectives as Performance Measures 2. Indicate direction of the measure: Actual Measure: Increase the percentage of offender records in CHRI system having final dispositions

17. June 20, 2005 17IL Criminal Justice Information Authority Developing Business Objectives as Performance Measures 3. Identify the object of the measure: Actual Measure: Increase the percentage of felony arrests in CHRI system having final dispositions

18. June 20, 2005 18IL Criminal Justice Information Authority Developing Business Objectives as Performance Measures 4. Identify the expected value of the measure of the measure: Actual Measure: Increase the percentage of felony arrests in CHRI system having final dispositions to 95%

19. June 20, 2005 19IL Criminal Justice Information Authority Developing Business Objectives as Performance Measures 5. Identify where the measurement will occur: Actual Measure: Increase the percentage of felony arrests in CHRI system having final dispositions to 95% in the target county

20. June 20, 2005 20IL Criminal Justice Information Authority Developing Business Objectives as Performance Measures 6. Identify when the measurement will be obtained: Actual Measure: Increase the percentage of felony arrests in CHRI system having final dispositions to 95% in the target county within the 1st year after project implementation

21. June 20, 2005 21IL Criminal Justice Information Authority Lessons from the Field: Performance measures cannot be objectively selected; stakeholders must reach consensus Stakeholders, depending on their role within the justice enterprise, are likely to have strongly divergent views Stakeholders, depending on their role within the justice enterprise, are likely to have strongly divergent views Higher-level goals and outcomes will usually be easier to agree upon than specific performance measures Higher-level goals and outcomes will usually be easier to agree upon than specific performance measures Performance measure selection, however, is not entirely subjective, and some measures are likely to be agreed upon by all Performance measure selection, however, is not entirely subjective, and some measures are likely to be agreed upon by all

22. June 20, 2005 22IL Criminal Justice Information Authority Lessons from the Field: It is nearly impossible to definitively establish causal linkages to public safety outcomes Outcomes become more diffuse the further out you get from actual processes Outcomes become more diffuse the further out you get from actual processes Improved information sharing is only one of a number of factors affecting public safety Improved information sharing is only one of a number of factors affecting public safety A single performance measure may be inadequate to indicate that a system is working (or not) A single performance measure may be inadequate to indicate that a system is working (or not) A carefully selected family of measures is preferable, and should be thoroughly tested before being deployed. A carefully selected family of measures is preferable, and should be thoroughly tested before being deployed.

23. June 20, 2005 23IL Criminal Justice Information Authority Lessons from the Field: Public sector outcome measures are driven by different factors than for the private sector Greater integration success by private industry due to clearer lines of accountability Greater integration success by private industry due to clearer lines of accountability While the private sector is driven by the bottom line, government is driven by the competing needs of many agencies with differing missions and purposes; lines of accountability are therefore more diffuse. While the private sector is driven by the bottom line, government is driven by the competing needs of many agencies with differing missions and purposes; lines of accountability are therefore more diffuse. Government is accountable not only to the governor, legislature, and public officials, but to the public as well Government is accountable not only to the governor, legislature, and public officials, but to the public as well Concern that the public believes that the justice system works as it is portrayed on television creates another type of accountability Concern that the public believes that the justice system works as it is portrayed on television creates another type of accountability

24. June 20, 2005 24IL Criminal Justice Information Authority Performance Measurement Caveats Most people (including your employees and consultants) can learn to make the measures come out the way they think you want them to, while exerting a minimum of effort in actually improving a process Most people (including your employees and consultants) can learn to make the measures come out the way they think you want them to, while exerting a minimum of effort in actually improving a process Always question the measures you’ve defined, keeping in mind that the people applying them could find ways of boosting the measures without really improving anything Always question the measures you’ve defined, keeping in mind that the people applying them could find ways of boosting the measures without really improving anything Test each measure to determine if it operates as expected. Does it always go one way when things get better and the other when things get worse? Test each measure to determine if it operates as expected. Does it always go one way when things get better and the other when things get worse?

25. June 20, 2005 25IL Criminal Justice Information Authority Performance Measurement Musts Early and often, measure and evaluate progress toward the goals and objectives that have been defined by the governing body Early and often, measure and evaluate progress toward the goals and objectives that have been defined by the governing body Continually verify that defined measures actually correspond with the achievement of goals and objectives Continually verify that defined measures actually correspond with the achievement of goals and objectives Resist overreaching : measures of long-term, global outcomes tend to be unreliable. Resist overreaching : measures of long-term, global outcomes tend to be unreliable. Performance measurement is complex and requires significant expertise Performance measurement is complex and requires significant expertise

26. June 20, 2005 26IL Criminal Justice Information Authority Resources for Assistance with Performance Measures (& Overall Justice Integration Planning) SEARCH SEARCH www.search.org Center for Society, Law, and Justice (University of New Orleans) Center for Society, Law, and Justice (University of New Orleans) www.cslj.net

27. June 20, 2005 27IL Criminal Justice Information Authority Developing a Theory of Change Process for Surfacing & Articulating a Theory of Change Use a Focus Group Format Use a Focus Group Format Start with Long-term Outcomes Start with Long-term Outcomes Work Backwards Toward Initial Activities Work Backwards Toward Initial Activities Map Required Existing Resources Map Required Existing Resources Reconcile Multiple Theories of Change Reconcile Multiple Theories of Change Adapted from J.P. Connell et al., 1995, “New Approaches to Evaluating Community Initiatives.” Aspen Institute.

28. June 20, 2005 28IL Criminal Justice Information Authority The Logic Model/Theory of Change Method for Developing Performance Measures Program Logic Model and Chain of Events Category Program Feature and Activity Initial Outcomes Intermediate Outcomes Intermediate Outcomes II Final Outcomes/Goals Accomplished Measures: 1._______ 2._______ 3._______ 4._______ 1._______ 2._______ 3._______ 4._______ 1._______ 2._______ 3._______ 4._______ 1._______ 2._______ 3._______ 4._______ 1._______ 2._______ 3._______ 4._______