1. The Standardized Program Evaluation Protocol ~SPEP~
Division of Adult Correction and Juvenile Justice
The Standardized Program Evaluation Protocol ~SPEP~
Understanding the Tool
and
Program Improvement Implications

2. First State-wide community-based alternatives (CBA) ~ late 1970s
Historical Context of North Carolina’s Juvenile Justice Evidence–Based Practices
First State-wide community-based alternatives (CBA) ~ late 1970s
First “wraparound” model in the US via Willie M. Program~ 1979 class action lawsuit
1998 NC Juvenile Justice Reform Act ~ Second state to mandate evidence-based services
I review these “firsts” because I think it is important to stress that NC has been the first with the aide of JCPCs to move much of the juvenile justice reform for NC in the positive direction that it has taken. NC adopted evidenced-based practices very early on and mandated it by legislation (point out that this was an unfunded mandate). This leads to further discussion of how the SPEP was first implemented in NC.

3. Historical Context of North Carolina’s Juvenile Justice Evidence–Based Practices
First state to fully adopt the OJJDP Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders
First legislated statewide, county-level Juvenile Crime Prevention Councils (also first statewide use of a risk factor assessment beginning in the late 1990’s)
Most effective statewide use of a juvenile offender risk and needs assessment instrument along with a disposition matrix to reduce confinement (Nov, 2001)

4. Historical Context of North Carolina’s Juvenile Justice Evidence–Based Practices
First state to begin implementation of a state-wide gang prevention and intervention initiative (Community-Based Youth Gang Violence Prevention Project) 2009 (Use of Recovery Act funds) (THE first U. S. experiment to fund statewide Comprehensive Gang Model programming!)

5. Historical Context of SPEP in NC
1998 NC Juvenile Justice Reform Act required a statewide evaluation of programs funded through the JCPC
The Standardized Program Evaluation Protocol ~SPEP~ was developed specifically for this purpose in NC. 2006
First statewide Level II services based on SPEP primary service types (2010)
Here is the discussion about the context of SPEP and historically how it fits with what we are currently doing.

6. Most (57%) JJ programs reduce recidivism: Outcomes of 556 studies (Dr
Most (57%) JJ programs reduce recidivism: Outcomes of 556 studies (Dr. Mark Lipsey, 2002)
SPEP 1.0 lessons learned included control studies whereby meta-analysis indicated that juvenile justice programs were effective. We learned that there were several domains that we focused upon in this literature…….primary/supplemental services, frequency and duration, and targeted risk level youth. The purpose of this training is to building upon the research and learning about the changes in the tool in order to impact programming at the local level to effect recidivism of youth served.

7. The SPEP Process
Key Steps to SPEP Score

8. EVALUATE/PROGRAM IMPROVEMENT
IDENTIFY
SERVICES
SPEP SCORING PROCESS
MATCH AGAINST RESEARCH-BASED CATEGORIES
DATA (DEMOGRAPHIC,RISK,QUALITY, QUANTITY FOR EACH SERVICE
SPEP
SCORE
This process for programs will be completed step by step with the ultimate goal of program improvement.
EVALUATE/PROGRAM IMPROVEMENT

9. IDENTIFY/ MATCH SERVICES
DATA (DEMOGRAPHIC,RISK,QUALITY, QUANTITY FOR EACH SERVICE)
Each of the factors found in the meta-analysis to be importantly related to program effectiveness is represented in the SPEP and associated with a certain maximum number of points to provide a score. The first step is identification of the primary service and a possible qualifying supplemental service. A crosswalk into the appropriate therapeutic approach and service type is important to understand. Sixty-five percent of the scoring comes from data generated by the program through the use of various tools (quality of service), duration/frequency(hrs) NCALLIES, and Level of Risk (risk assessment tool). NCALLIES will be an integral part of the scoring.
Point out that the Primary Service Type groupings are directly proportionate to the contributing factors associated with that type of program group that effects recidivism. There are some program types that are more effective than others.
The risk levels of juveniles served must be based on the results of a valid risk assessment instrument (North Carolina’s Assessment of Juvenile Risk of Future Offending ). Ratings of the amount of service must be based on management information system data, in this case, NCALLIES, that report the service received by each juvenile.
SPEP is data driven.

10. Classification of Program Intervention
Getting Started…….
Classification of Program Intervention

11. SPEP 101 Definitions
Structure or Format: Setting within which the program services are delivered
Example: Residential, YDC, Day Treatment
Program: Services that are delivered within the context of a structure or format.
Service: The treatment ingredient of the program. Services target key criminogenic risk factors and treatment needs and are direct interventions with the juvenile or with others (peers or family) in order to alter juvenile behavior that leads to reduced recidivism.
For continuation programs and JCPC members that have been with us for the long-haul , this was the basic teaching in the first phase of SPEP. Review.

12. Identification of “Philosophies” toward Altering Juvenile Behavior
Program Services Fall into 3 Broad Categories:
External Control Techniques
Behavioral change through instilling discipline, fear, or detection of bad behavior in the absence of treatment
Therapeutic Techniques
Behavioral change through improved skills, relationships, insight.
These are the only services for which SPEP ratings are completed.
Other :Services for which there is insufficient research to estimate the effects on recidivism
Meta-analysis is limited to recidivism effects in the study. The focus is upon recidivism with the SPEP. Meta-analysis allows for sorting of the program types that have the greatest effect on recidivism. The sorting within the meta-analysis generated broad categories of interventions/ techniques that fell into 2 main categories: External Control Techniques: discipline, fear (bootcamp) programs; Therapeutic Techniques: those that improved skills, relationships, insight.
Focus was upon the therapeutic approaches given that they had positive impact on recidivism reduction. JCPC need to understand that they need to fund therapeutic approaches. There will be structures, however, that are important to the continuum of services that play a vital role in providing services to youth.

13. Program Types Sorted by General Approach: Average Recidivism Effect
Multiple services
Counseling*
Skill building**
Restorative
Surveillance
Deterrence
Discipline
-10 -5 5 10 15
% Recidivism Reduction from .50 Baseline
Therapeutic
approaches
Control
approaches
Control
approaches
Broad categories were generated from the meta-analysis. Programs sort into two large / broad categories.
Control approaches , not very effective, Discipline: like “Scared Straight” and boot camp/para military, “pushups before breakfast” style programs actually have a negative impact on recidivism.
Deterrence programs such jail visitations , scared straight programs.
Surveillance: intensive surveillance supervision, positive effects
Therapeutic programs and approaches reduce recidivism and some approaches are clearly more effective than others.
All on average have positive effects on recidivism.
It is important to note here that there will be a great variance in the recidivism reduction. Point out that programs along a bar graph, though they may have positive effects, their positive effects can vary greatly from one program to another in the meta-analysis studies.

14. Further Sorting by Intervention Type within, e. g
Further Sorting by Intervention Type within, e.g., Counseling Approaches
Point out that within program groupings, there is further distinction between the program type and impact on recidivism; though all have some impact on recidivism reduction. Make a point of stressing that there is great variance in the effects of programming on recidivism, some with greater effects (i.e. mentoring…some as great as about 22% reduction) but yet there were some programs in the meta-analysis data that had very little, yet positive effects.
All on average show positive effects on recidivism, but there is also variation. Some approaches seem to have greater effects than others.
Mentoring is grouped with counseling because the counseling approach is considered to be “a personal relationship between the offender
and a responsible adult who attempts to exercise influence on the juvenile’s feelings, cognitions, and behavior.” Mark Lipsey

15. Further Sorting by Intervention Type within, e. g
Further Sorting by Intervention Type within, e.g., Skill-building Approaches
Ditto
Last category does not show strong positive effects in this literature, but keep in mind the average age in about 14 in the literature. Point out variation of effects.

16. SPEP 101 Definitions Primary Service: Main focus of the program
Most time/effort spent on this
Recognized as the dominant theme
Other program elements support its effectiveness

17. SPEP 101 Definitions Supplemental Service: Other distinct service
Less time/effort spent on this
Serves to reinforce or complement the primary service
Note: Supplemental service can be deemed as a “qualifying” one in order to receive additional points on the SPEP tool OR points may be automatic based on lack of evidence in the research base to couple a supplemental service with the identified primary service.

18. WHAT IS THE PROGRAM’S PRIMARY SERVICE?
IN WHICH OF THE 5 SERVICE GROUPS DOES THE SERVICE FALL?
IS THERE A QUALIFYING SUPPLEMENTAL SERVICE ?

19. Things to Consider………
JCPCs will need to fund structures or formats. (Teen Court, Structured Day Programs)
It is important that an analysis of key therapeutic element (s) within the structures be explored.
JCPCs need to fund Therapeutic Approaches/Interventions
JCPCs need to understand the SPEP process in order to understand SPEP scores and their implications for program improvement.

20. DATA
Examining the Service Quality, Amount of Service (Duration and Contact Hours), and Risk Level

21. Data Used for SPEP Scores
Quality of Service:
SPEP Quality of Services Checklist~ Completed with the Area Consultant
Examines the following:
Written program protocol ~ program manual; written protocol that describes how the intended services is delivered
Staff training /staff retention
Monitoring of Program effectiveness, protocol and corrective action to correct deviations

22. SPEP Quality of Services Checklist Score
Point out that the Quality Services Checklist is a tool used by the consultant to review with the provider those areas noted in the previous slide. The score from the checklist would generate up to 20 points on the SPEP.

23. Data Used for SPEP Scores
Amount of Service: (NCALLIES)
Duration of Service
Target number of weeks specified for each service type
% of youth who received targeted weeks of service
Maximum number of points ~10
Contact Hours
Target number of hours specified for each service type
% of youth who received targeted hours of service
SPEP scores are data driven and 20 points would come from information derived from the % of terminated clients during the examination period that meet the targeted number of weeks/targeted number of hours specified for the service type.

24. Data generated from NC ALLIES
Again, the SPEP is data driven based in the data entered by the program. I think it is important here to stress that this data is based on the % of terminated youth that meet the targeted number of weeks/targeted number of hours specified for the service type during the examination period.

25. Data Used for SPEP Scores
Risk level of Youth Served:
Program’s use of a validated risk assessment instrument/tool to provide juvenile risk level data on each juvenile receiving services
North Carolina’s Assessment of Juvenile Risk of Future Offending ~ Validated tool currently being used by Juvenile Court Services. This risk assessment tool has been selected for use by program providers. The department will train providers on its use.
NC ALLIES will be modified with the addition of a Risk Level field. Risk Level Scores from referrals from Juvenile Court would be entered as well as risk level scores generated by the program.
Future modifications will include incorporating the risk assessment tool into NCALLIES.
Consider that there are generally 2/3rds of youth that are low risk for reoffending in NC. The research supports this and NC data from NCJOIN also suggests this. It is important to realize that the SPEP research centers around recidivism reduction of youth that have involvement with the juvenile justice system. Many JCPC programs serve a variance of youth risk levels. Court services utilizes the Juvenile Risk of Future Offending tool which has been validated in 6 studies. There is a strong leaning toward use of this tool by providers to determine the level of risk for reoffending for youth being served. Risk levels/scores would come with youth referred from juvenile court services and programs would generate risk levels based on its administering of a risk tool for each youth served. (The department has now decided to use the North Carolina Assessment of Juvenile Risk of Future Offending Tool. Providers will be trained on the use of the tool.)

26. Data Generated via a Validated Risk Assessment Tool and entered into NC ALLIES

27. The SPEP SCORE
Examines how a specific program is performing compared to the effective practice for that service type in the research
Provides opportunities for program improvement for programs that fall short of the evidence-based effective practice profiles.

28. Systematic Correction to Optimize Recidivism reduction Effectively
SPEP allows for a systematic means by which we (the JCPC and the Department) can help programs correct their programming/agency practices to optimize recidivism reduction in an effective and meaningful manner.

29. Scoring = Improvement Opportunity
Initial Score
Understanding the Score
Implications for Improvement
Program Improvement Plan
Subsequent Scoring
Repeat
Scoring is just an improvement or enhancement opportunity.
SPEP 2.0 will provide a “baseline” score; with time and planning for improvement; score again; cyclical.

30. Program Improvement
Program Improvement Plans address the areas needing improvement as identified through the program’s SPEP score.
Timeframes for program improvement will be established
Monitoring of the status of program improvement will be incorporated into the JCPC monitoring/planning processes and Consultant Program Monitoring
JCPCs would continue to evaluate the effectiveness of their funded programs through general oversight and supporting program improvement plans.
There is a role for the JCPCs in the program improvement of their locally funded JCPC programs. JCPCs can be supportive to the program improvement planning efforts through their local JCPC monitoring/planning processes.

31. Central Area Consultant
Division of Adult Correction and Juvenile Justice
Questions?
Ronald Tillman
Central Area Consultant