1. Centre for Research on Children and Families

2. National Outcomes Matrix:
Evidence of Intervention Effectiveness
Nico Trocmé
with Tony Esposito, Martin Chabot & Lorry Coughlin
Theory & Wisdom: Promoting Evidence Based Decision Making Edmonton, AB , May 28th
Centre for Research on Children and Families

3. Management Practice An Incremental Multi-Level Outcomes Approach
Financial Based Management
Client Tracking System Using Proxy Measures
Management
Integrated Outcomes Tracking System
An Incremental Multi-Level Outcomes Approach
Practice
Outcome Based Case-Planning +
Measures for Clinical Practice
Reactive Practice
Centre for Research on Children and Families

4. Selecting indicators:
Can indicators be clearly linked to seminal child welfare service objectives?
Are they easily understood?
Can they be measured using existing information systems?
Centre for Research on Children and Families

5. Competing objectives of child welfare
Child Protection
Child Well-Being
Family and Community Support
Centre for Research on Children and Families

6. NOM & Pilot Test Indicators
Family & Community Support
Permanence
Well-Being
Safety
1.Recurrence & 2.Injury
3.School delay & 4.YCJA
8. Court 9. Housing stability & 10. Community-based placements
5.Placement 6. Moves in care & 7.Time in care
Centre for Research on Children and Families

7. NOM Dashboard: Pilot Test 02-08
02-03
03-04
04-05
05-06
06-07
07-08
Recurrence Cases served re-substantiated within 12 months of closing
15%
16%
13% 9%
School Delay
Agency children who are at least 1 year behind age appropriate grade compared to norm
24% 5%
32% 6%
30%
27% 4%
25%
Youth Criminal Justice Rate Investigated children (12 to 15 years old) who experienced youth criminal justice services within 36 months of initial investigation
22%
17%
12% NA
Placement Rate Investigated children who experience at least one placement (>3 days) within 36 months of initial investigation
21%
23%
Moves in care Placement changes (>3 days) or reunification breakdowns within 36 months of initial placement
2.5
2.1
2.4
1.90
Time in care
(% current status / median days per current status)
Return home
Adoption
Other
Still in care
260
56% / 107
2% / 850
19% / 183
22% /1025
255
52% / 70
1% / 502
24% / 224
23% /1011
385
55% / 201
3% / 805
16% / 313
26% /1095
Court Involvement Children who experience at least one court event within 36 months of initial investigation
40%
42%
33% 7

8. Recurrence cohort: All closed Cases served, tracked for 12 months
Mean Age
2002
580
8.1
2003
495
8.0
2004
575
7.8
2005
530
8.3
2006
522
8.4
2007
395
8.2
6 year average
SDC AM
AM / IT IT
SDNC
Nico
Recurrence cohort: All closed Cases served, tracked for 12 months 8

9. NOM Dashboard: Pilot Test 02-08
02-03
03-04
04-05
05-06
06-07
07-08
Recurrence Cases served re-substantiated within 12 months of closing
15%
16%
13% 9%
School Delay
Agency children who are at least 1 year behind age appropriate grade compared to norm
24% 5%
32% 6%
30%
27% 4%
25%
Youth Criminal Justice Rate Investigated children (12 to 15 years old) who experienced youth criminal justice services within 36 months of initial investigation
22%
17%
12% NA
Placement Rate Investigated children who experience at least one placement (>3 days) within 36 months of initial investigation
21%
23%
Moves in care Placement changes (>3 days) or reunification breakdowns within 36 months of initial placement
2.5
2.1
2.4
1.90
Time in care
(% current status / median days per current status)
Return home
Adoption
Other
Still in care
260
56% / 107
2% / 850
19% / 183
22% /1025
255
52% / 70
1% / 502
24% / 224
23% /1011
385
55% / 201
3% / 805
16% / 313
26% /1095
Court Involvement Children who experience at least one court event within 36 months of initial investigation
40%
42%
33% 9

10. Recurrence by reason for report
Total
Closed cases
373 44 12 92 8
529
Non recurring
322 40 7 72
449
Recurring 51 4 5 20 80
Recurrence rate
13% 9% --
21%
15%

11. Recurrence by age at service start
Methodological notes:
Maintenant, meme enfant peut etre dans plusieurs cohortes, 17 ans et moins a fin de AM
Recurrence of maltreatment tracks children who had at least one substantiated recurrent event of maltreatment within 12 months of their file closing.
Retained recurrence tracks maltreatment for children while receiving services. (Cohort ). For example: approximately 24% of children receiving AM services had a recurring event of maltreatment while receiving AM services.
RRAM Numerator () = number of AM closed with at least one retained recurrent event while the child is receiving AM services.
RRAM Denominator () = number of AM closed.
RRAM Total = 23,8 %.
The NOM preliminary measures suggest that 22% of children who had ongoing child welfare involvement that ended in the referring year had a recurring event of maltreatment within that year.
Total
Closed cases 82
102
134
122 89
529
Non recurring 75
118 98 69
449
Recurring 7 13 16 24 20 80
Recurrence rate 8%
12%
11%
19%
22%
15% 11

12. NOM Dashboard: Pilot Test 02-08
02-03
03-04
04-05
05-06
06-07
07-08
Recurrence Cases served re-substantiated within 12 months of closing
15%
16%
13% 9%
School Delay
Agency children who are at least 1 year behind age appropriate grade compared to norm
24% 5%
32% 6%
30%
27% 4%
25%
Youth Criminal Justice Rate Investigated children (12 to 15 years old) who experienced youth criminal justice services within 36 months of initial investigation
22%
17%
12% NA
Placement Rate Investigated children who experience at least one placement (>3 days) within 36 months of initial investigation
21%
23%
Moves in care Placement changes (>3 days) or reunification breakdowns within 36 months of initial placement
2.5
2.1
2.4
1.90
Time in care
(% current status / median days per current status)
Return home
Adoption
Other
Still in care
260
56% / 107
2% / 850
19% / 183
22% /1025
255
52% / 70
1% / 502
24% / 224
23% /1011
385
55% / 201
3% / 805
16% / 313
26% /1095
Court Involvement Children who experience at least one court event within 36 months of initial investigation
40%
42%
33% 12

13. School Delay Rate Proportion of agency children who are at least 1 year behind age appropriate grade by placement type (all years combined)
Methodological notes:
Maintenant, meme enfant peut etre dans plusieurs cohortes, 17 ans et moins a fin de AM
Recurrence of maltreatment tracks children who had at least one substantiated recurrent event of maltreatment within 12 months of their file closing.
Retained recurrence tracks maltreatment for children while receiving services. (Cohort ). For example: approximately 24% of children receiving AM services had a recurring event of maltreatment while receiving AM services.
RRAM Numerator () = number of AM closed with at least one retained recurrent event while the child is receiving AM services.
RRAM Denominator () = number of AM closed.
RRAM Total = 23,8 %.
The NOM preliminary measures suggest that 22% of children who had ongoing child welfare involvement that ended in the referring year had a recurring event of maltreatment within that year.
7 to 17 years of age
TOTAL (N)
63%
26%
24%
5000 13

14. School delay by age & sex: All years
P=0.000

15. NOM Dashboard: Pilot Test 02-08
02-03
03-04
04-05
05-06
06-07
07-08
Recurrence Cases served re-substantiated within 12 months of closing
15%
16%
13% 9%
School Delay
Agency children who are at least 1 year behind age appropriate grade compared to norm
24% 5%
32% 6%
30%
27% 4%
25%
Youth Criminal Justice Rate Investigated children (12 to 15 years old) who experienced youth criminal justice services within 36 months of initial investigation
22%
17%
12% NA
Placement Rate Investigated children who experience at least one placement (>3 days) within 36 months of initial investigation
21%
23%
Moves in care Placement changes (>3 days) or reunification breakdowns within 36 months of initial placement
2.5
2.1
2.4
1.90
Time in care
(% current status / median days per current status)
Return home
Adoption
Other
Still in care
260
56% / 107
2% / 850
19% / 183
22% /1025
255
52% / 70
1% / 502
24% / 224
23% /1011
385
55% / 201
3% / 805
16% / 313
26% /1095
Court Involvement Children who experience at least one court event within 36 months of initial investigation
40%
42%
33% 15

16. Service cohort: all retained cases, up to 36 months after report
Mean Age
2002
950
7.8
2003
870
2004
750
7.6
2005
890
7.7
4 year average
SDC AM IT
Nico
Service cohort: all retained cases, up to 36 months after report
SDNC
Centre for Research on Children and Families 16

17. NOM Dashboard: Pilot Test 02-08
02-03
03-04
04-05
05-06
06-07
07-08
Recurrence Cases served re-substantiated within 12 months of closing
15%
16%
13% 9%
School Delay
Agency children who are at least 1 year behind age appropriate grade compared to norm
24% 5%
32% 6%
30%
27% 4%
25%
Youth Criminal Justice Rate Investigated children (12 to 15 years old) who experienced youth criminal justice services within 36 months of initial investigation
22%
17%
12% NA
Placement Rate Investigated children who experience at least one placement (>3 days) within 36 months of initial investigation
21%
23%
Moves in care Placement changes (>3 days) or reunification breakdowns within 36 months of initial placement
2.5
2.1
2.4
1.90
Time in care
(% current status / median days per current status)
Return home
Adoption
Other
Still in care
260
56% / 107
2% / 850
19% / 183
22% /1025
255
52% / 70
1% / 502
24% / 224
23% /1011
385
55% / 201
3% / 805
16% / 313
26% /1095
Court Involvement Children who experience at least one court event within 36 months of initial investigation
40%
42%
33% 17

18. Youth Criminal Justice involvement by reason for the report
Total
Children in cohort 73 34 7 67 3
184
No YCJA 62 31 49
152
YCJA 11 18 32
YCJA Rate
15% 8%
*--%
26%
17%
* Too few cases to calculate a meaningful rate 18

19. Youth Criminal Justice involvement by age
Total
Methodological notes:
Maintenant, meme enfant peut etre dans plusieurs cohortes, 17 ans et moins a fin de AM
Recurrence of maltreatment tracks children who had at least one substantiated recurrent event of maltreatment within 12 months of their file closing.
Retained recurrence tracks maltreatment for children while receiving services. (Cohort ). For example: approximately 24% of children receiving AM services had a recurring event of maltreatment while receiving AM services.
RRAM Numerator () = number of AM closed with at least one retained recurrent event while the child is receiving AM services.
RRAM Denominator () = number of AM closed.
RRAM Total = 23,8 %.
The NOM preliminary measures suggest that 22% of children who had ongoing child welfare involvement that ended in the referring year had a recurring event of maltreatment within that year.
Children in cohort 57 48 79
184
No YCJA 50 37 65
152
YCJA 7 11 14 32
YCJA Rate
12%
22%
17%
Centre for Research on Children and Families 19

20. NOM Dashboard: Pilot Test 02-08
02-03
03-04
04-05
05-06
06-07
07-08
Recurrence Cases served re-substantiated within 12 months of closing
15%
16%
13% 9%
School Delay
Agency children who are at least 1 year behind age appropriate grade compared to norm
24% 5%
32% 6%
30%
27% 4%
25%
Youth Criminal Justice Rate Investigated children (12 to 15 years old) who experienced youth criminal justice services within 36 months of initial investigation
22%
17%
12% NA
Placement Rate Investigated children who experience at least one placement (>3 days) within 36 months of initial investigation
21%
23%
Moves in care Placement changes (>3 days) or reunification breakdowns within 36 months of initial placement
2.5
2.1
2.4
1.90
Time in care
(% current status / median days per current status)
Return home
Adoption
Other
Still in care
260
56% / 107
2% / 850
19% / 183
22% /1025
255
52% / 70
1% / 502
24% / 224
23% /1011
385
55% / 201
3% / 805
16% / 313
26% /1095
Court Involvement Children who experience at least one court event within 36 months of initial investigation
40%
42%
33% 20

21. Placement rate by reason for the report
Total
Children in cohort
524
110 29 92 6
761
No placement
407 95 26 49 3
580
Placement
117 15 43
181
Placement Rate
22%
13%
--%
46%
23%

22. Placement rate by age Total 117 170 190 205 79 761 77 150 162 149 42
Methodological notes:
Maintenant, meme enfant peut etre dans plusieurs cohortes, 17 ans et moins a fin de AM
Recurrence of maltreatment tracks children who had at least one substantiated recurrent event of maltreatment within 12 months of their file closing.
Retained recurrence tracks maltreatment for children while receiving services. (Cohort ). For example: approximately 24% of children receiving AM services had a recurring event of maltreatment while receiving AM services.
RRAM Numerator () = number of AM closed with at least one retained recurrent event while the child is receiving AM services.
RRAM Denominator () = number of AM closed.
RRAM Total = 23,8 %.
The NOM preliminary measures suggest that 22% of children who had ongoing child welfare involvement that ended in the referring year had a recurring event of maltreatment within that year.
Children
117
170
190
205 79
761
No Placement 77
150
162
149 42
580
Placement 40 20 28 56 37
181
Placement Rate
34%
11%
14%
27%
46%
23%
Centre for Research on Children and Families 22

23. NOM Dashboard: Pilot Test 02-08
02-03
03-04
04-05
05-06
06-07
07-08
Recurrence Cases served re-substantiated within 12 months of closing
15%
16%
13% 9%
School Delay
Agency children who are at least 1 year behind age appropriate grade compared to norm
24% 5%
32% 6%
30%
27% 4%
25%
Youth Criminal Justice Rate Investigated children (12 to 15 years old) who experienced youth criminal justice services within 36 months of initial investigation
22%
17%
12% NA
Placement Rate Investigated children who experience at least one placement (>3 days) within 36 months of initial investigation
21%
23%
Moves in care Placement changes (>3 days) or reunification breakdowns within 36 months of initial placement
2.5
2.1
2.4
1.90
Time in care
(% current status / median days per current status)
Return home
Adoption
Other
Still in care
260
56% / 107
2% / 850
19% / 183
22% /1025
255
52% / 70
1% / 502
24% / 224
23% /1011
385
55% / 201
3% / 805
16% / 313
26% /1095
Court Involvement Children who experience at least one court event within 36 months of initial investigation
40%
42%
33% 23

24. Placement Cohort AM SDC IT SDNC
Mean Age
2002
330
10.4
2003
325
11.6
2004
294
10.5
2005
320
11.5
4 year average
11.1
Placement cohort: all placed children, up to 36 months after initial placement
Nico AM
SDC IT
SDNC
Centre for Research on Children and Families 24

25. NOM Dashboard: Pilot Test 02-08
02-03
03-04
04-05
05-06
06-07
07-08
Recurrence Cases served re-substantiated within 12 months of closing
15%
16%
13% 9%
School Delay
Agency children who are at least 1 year behind age appropriate grade compared to norm
24% 5%
32% 6%
30%
27% 4%
25%
Youth Criminal Justice Rate Investigated children (12 to 15 years old) who experienced youth criminal justice services within 36 months of initial investigation
22%
17%
12% NA
Placement Rate Investigated children who experience at least one placement (>3 days) within 36 months of initial investigation
21%
23%
Moves in care Placement changes (>3 days) or reunification breakdowns within 36 months of initial placement
2.5
2.1
2.4
1.90
Time in care
(% current status / median days per current status)
Return home
Adoption
Other
Still in care
260
56% / 107
2% / 850
19% / 183
22% /1025
255
52% / 70
1% / 502
24% / 224
23% /1011
385
55% / 201
3% / 805
16% / 313
26% /1095
Court Involvement Children who experience at least one court event within 36 months of initial investigation
40%
42%
33% 25

26. Moves in Care Number of moves (# of children) 1 2-3 4+ Total children
28%
37%
19% 1
21%
20%
25%
2-3
23%
30% 4+
26%
Total children
332
100%
322
289
Total Moves
767
641
723
Average moves
2.5
2.1
2.4 26

27. Moves in Care: by reason for report
Total
Number of moves
22%
13%
25%
14% 54
19% 1
27%
29%
23%
44% 76
26%
2-3
30%
20%
31% 85 4+ 21
38%
32%
11% 74
Total children
155
100% 24 4 97 9
289
Total Moves
338 68 8
293 16
Average moves
2.2
2.8
2.1
3.1
1.7 27

28. Moves in Care by age at entry
Total
Number of moves
32%
19%
17%
11%
20% 54 1
16%
47%
39%
21% 76
26%
2-3
44%
25%
27%
31% 85
30% 4+ 8% 9%
37%
33% 74
Total children 38
100% 32 41 67
111
289
Total Moves 58 47 72
227
319
Average moves
1.5
1.4
1.6
3.5
2.8 28

29. NOM Dashboard: Pilot Test 02-08
02-03
03-04
04-05
05-06
06-07
07-08
Recurrence Cases served re-substantiated within 12 months of closing
15%
16%
13% 9%
School Delay
Agency children who are at least 1 year behind age appropriate grade compared to norm
24% 5%
32% 6%
30%
27% 4%
25%
Youth Criminal Justice Rate Investigated children (12 to 15 years old) who experienced youth criminal justice services within 36 months of initial investigation
22%
17%
12% NA
Placement Rate Investigated children who experience at least one placement (>3 days) within 36 months of initial investigation
21%
23%
Moves in care Placement changes (>3 days) or reunification breakdowns within 36 months of initial placement
2.5
2.1
2.4
1.90
Time in care
(% current status / median days per current status)
Return home
Adoption
Other
Still in care
260
56% / 107
2% / 850
19% / 183
22% /1025
255
52% / 70
1% / 502
24% / 224
23% /1011
385
55% / 201
3% / 805
16% / 313
26% /1095
Court Involvement Children who experience at least one court event within 36 months of initial investigation
40%
42%
33% 29

30. Time in Care by age at entry
Outcome after 36 months
percentage by outcome (median in days)
Total Children %(N)
Return Home
41% (225)
34% (27)
47%(58)
41% (66)
65%(106)
52%(167)
Adoption
10% (502)
1%(3)
Other
28% (437)
26% (558)
14%(29)
22% (107)
27%%(250)
24%(77)
Still in Care
21% (1005)
40% (1008)
39%(999)
37% (1105)
8%(1017)
23%(1022)
Total Children 29 35 42 71
145
322 30 30

31. NOM Dashboard: Pilot Test 02-08
02-03
03-04
04-05
05-06
06-07
07-08
Recurrence Cases served re-substantiated within 12 months of closing
15%
16%
13% 9%
School Delay
Agency children who are at least 1 year behind age appropriate grade compared to norm
24% 5%
32% 6%
30%
27% 4%
25%
Youth Criminal Justice Rate Investigated children (12 to 15 years old) who experienced youth criminal justice services within 36 months of initial investigation
22%
17%
12% NA
Placement Rate Investigated children who experience at least one placement (>3 days) within 36 months of initial investigation
21%
23%
Moves in care Placement changes (>3 days) or reunification breakdowns within 36 months of initial placement
2.5
2.1
2.4
1.90
Time in care
(% current status / median days per current status)
Return home
Adoption
Other
Still in care
260
56% / 107
2% / 850
19% / 183
22% /1025
255
52% / 70
1% / 502
24% / 224
23% /1011
385
55% / 201
3% / 805
16% / 313
26% /1095
Court Involvement Children who experience at least one court event within 36 months of initial investigation
40%
42%
33% 31

32. Court Involvement by reason for report
Total
Children in cohort
524
110 29 92 6
761
No Court Involvement
295 86 22 47 2
452
Court Involvement
229 24 7 45 4
309
Court Rate
43%
21%
--*
48%
40%
* Too few cases to calculate a meaningful rate

33. NOM Dashboard: Pilot Test 02-08
02-03
03-04
04-05
05-06
06-07
07-08
Recurrence Cases served re-substantiated within 12 months of closing
15%
16%
13% 9%
School Delay
Agency children who are at least 1 year behind age appropriate grade compared to norm
24% 5%
32% 6%
30%
27% 4%
25%
Youth Criminal Justice Rate Investigated children (12 to 15 years old) who experienced youth criminal justice services within 36 months of initial investigation
22%
17%
12% NA
Placement Rate Investigated children who experience at least one placement (>3 days) within 36 months of initial investigation
21%
23%
Moves in care Placement changes (>3 days) or reunification breakdowns within 36 months of initial placement
2.5
2.1
2.4
1.90
Time in care
(% current status / median days per current status)
Return home
Adoption
Other
Still in care
260
56% / 107
2% / 850
19% / 183
22% /1025
255
52% / 70
1% / 502
24% / 224
23% /1011
385
55% / 201
3% / 805
16% / 313
26% /1095
Court Involvement Children who experience at least one court event within 36 months of initial investigation
40%
42%
33% 33

34. For more information: www.cecw-cepb.ca nico.trocme@mcgill.ca
Centre for Research on Children and Families

35.
Centre for Research on Children and Families