1 Using a Statewide Evaluation Tool for Child Outcomes & Program Improvement Terry Harrison, Part C Coordinator Susan Evans, Autism Project Specialist New Jersey Early Intervention System NJ Department of Health and Senior Services

2 A look at New Jersey Part C  NJ has 21 counties  Each county has at least one dedicated Targeted Evaluation Team (TET). All eligibility evaluations are done by the TETs.  Evaluators administer a standardized tool for all children at entry and a percentage of children at exit to answer OSEP Outcome questions 3.A, 3.B, and 3.C

3 Battelle Developmental Inventory 2 nd edition  Chosen based on following criteria:  Commercially available  Domains answer Child Outcome questions  Reliable and valid  Can be administered by NJEIS evaluators  Norm referenced  Can be used to help determine eligibility  Can be used for Part C and 619

4 Exit Plan  5 -6 counties each year over 4 years conduct exit evaluations when children leave the system.  To be assessed on exit a child has to:  Have an intake BDI-2  Be in the system for at least 6 months  Reside in a county doing exit evaluations  NJ reported exit data in APR 2008 for 63 children

5 OSEP APR Reporting

6 Reporting Decisions For APR indicators 3.B and 3.C NJEIS makes decisions based on two BDI2 domains OSEPBDI-2 Domain A.Positive social-emotional skills (including social relationships) Personal/Social B. Acquisition and use of knowledge and skills (including early language/communication) Communication Cognition C. Use of appropriate behaviors to meet their needs Adaptive Motor

7 Standard Score  NJEIS uses BDI-2 derived Standard Scores by domain for the basis of reporting  The Standard Score represents the child’s development in relation to children in the same age group  Mean = 100, Sd = 15

8 Standard Score  Scores of 90 to 100 are considered as “average”,  Scores between 80 and 89 considered as “low average”.  Scores below 80 indicate “mild to more severe developmental delay”

9 Same age peers  NJEIS considers children as functioning with same age peers when their standard score in each domain is 80 or greater.  Children have to be in the “low average” group or higher.

10 Initial and Exit Scores NJEIS is using four BDI-2 data elements from each domain to “calculate” a cross walk to OSEP a, b, c, d, e  Initial Raw – is the raw score at entry  Initial Standard – is the standard score at entry  Exit Raw – is the raw score at exit  Exit Standard – is the raw score at exit

11 Reporting Categories  Assignment to a, b, c is evaluated independent from d, e  For 3.B & 3.C the assignment to a, b, and c will be based on the maximum little score assigned to a domain in each indicator. (i.e. a is less then b)  In the case of 3.A the score for the one domain will be reported

12 Business Rules a, b, c Report in “c” Percentage of children who improved functioning to a level nearer to same aged peers but did no reach it. Exiting Raw > Initial Raw AND Exiting Standard > Initial Standard

13 Business Rules a, b, c a. Percentage of children who did not improve functioning Exiting Raw =< Initial Raw AND Exiting Standard < 80 b. Percentage of children who improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers. Exiting Raw > Initial Raw AND Exiting Standard <= Initial Standard AND Exiting Standard < 80

14 Example Outcome 3.B category c  Percentage of children who improved functioning to a level nearer to same aged peers but did no reach it. Cognitive Domain Raw = 49 Standard = 61 Cognitive Domain Raw = 25 Standard = 55 Communication Domain Raw = 57 Standard = 71 Communication Domain Raw = 33 Standard = 64 < < EntryExit Raw and Standard score increase; however exiting standard below 80. Therefore, little c.

15 Business Rules d, e d. Percentage of children who improved functioning to reach a level comparable to same aged peers. Initial Standard = 80 e. Percentage of children who maintained functioning at a level comparable to same- aged peers. Initial Standard >= 80 AND Exiting Standard >= 80

16 Business Rules d, e  Only be assigned to d, or e if both domains indicate that the child is comparable to same aged peer  If only one of two domains is comparable to same aged peer report in c  If one domain is in d and another falls in e then the child will be assigned to d

17 Example Outcome 3.C category d Adaptive Domain Raw = 44 Standard = 87 Adaptive Domain Raw = 33 Standard = 76 Motor Domain Raw = 118 Standard = 102 Motor Domain Raw = 96 Standard = 86  Percentage of children who improved functioning to reach a level comparable to same aged peers. EntryExit < < Initial Standard score below 80. Therefore, little d. Initial Standard score below 80. Therefore, little e. Child is reported in little d because the lower little scores is used.

18 Exit 2008 – Outcome #3.B Knowledge and skills a) Did not make progress 12% b) Improved but not nearer to peers 23% c) Improved nearer to peers 1625% d) Reached peers 1727% e) Maintained functioning with peers 2743% TotalsN= 63100%

19 Exit 2008 – Outcome #3.B Behaviors to meet needs a) Did not make progress 00% b) Improved but not nearer to peers 12% c) Improved nearer to peers 1219% d) Reached peers 1016% e) Maintained functioning with peers 4063% TotalsN= 63100%

20 Exit 2008 – Outcome #3.A Social Skills a) Did not make progress 35% b) Improved but not nearer to peers 46% c) Improved nearer to peers 35% d) Reached peers 23% e) Maintained functioning with peers 5181% TotalsN= %

21 Applying Technology

22 Part C & BDI-2  Each evaluator uses a palm pilot which contains the full BDI-2  Results:  Scoring errors are minimized  Evaluators synch the palm to the web  Agencies have access to reports at local level

23 Web-based Data System  Lead agency has access to individual and agency data via the web-based data system  Lead agency uses the web-based data system to export data for federal reporting  Data is also used by lead agency for:  Procedural Safeguards Contacts  Program compliance with child outcomes project  Quality control of evaluators via desk audits

24 General Supervision

25 Data  NJEIS has started to use BDI-2 data as part of its general supervision and monitoring system  Monitoring:  Appropriateness of IFSP services based on initial evaluation  Eligibility decisions  Evaluator qualifications and quality assurance

26 General Supervision: Appropriate Services  NJEIS charted children whose eligibility evaluation showed more that 1.5 Sd below the mean.  Compared this data to authorized service hours based on IFSPs.  This data raises questions related to appropriate type and intensity of service decisions made by IFSP teams.

27

28 Next Steps Appropriate Services  Compare the areas of need ( by domains & sub-domains identified by the BDI-2 more than 1.5 Sd below) with type, frequency and intensity of services identified on the IFSP  Monitor appropriate justification of IFSP Team service decisions.  Provide Training & Technical Assistance

29 General Supervision: Eligibility Decisions  NJEIS teams use BDI-2 as part of the eligibility decision process  First time state-wide use of same instrument as part of the eligibility process  Other tools are completed as needed

30 Next Steps: Eligibility  Pending Part C final regulations, NJ is considering implementing the screener portion of the BDI-2

31 General Supervision: Evaluators  Use of statewide tool & subsequent training activities identified the need to establish minimum standards for qualified NJEIS evaluators.  The lead agency surveyed TET agencies regarding personnel criteria for their evaluators.

32 Survey Results  16 TET agencies responded  6 agencies had specific “evaluator” job descriptions  The remaining agencies reported having job descriptions related to each discipline that also included evaluation as a job duty

33 Survey Results  Agency Requirement of EI Experience  6 - require 2+ years  4 - require 1 year  1 - requires 400+ hours in EI  1 - required 1 year for a licensed professional and 2+ years for other disciplines  4 - had no requirements

34 Survey Results  Most of TET agencies do not require coursework or training in evaluation.  Mentoring Plan  4 have no mentoring plan  7 have procedures for mentoring or pairing with experienced evaluators  6 did not have any plans specific to being an evaluator

35 Next Steps: Evaluators  Review standard personnel criteria for evaluators established in other states  Develop NJ standards  Challenges:  Quantifying competencies for hiring and monitoring  Recruitment  Should the state consider “grandfathering” of current evaluators?

36 Child Outcome Costs

37 Implementation Costs  DHSS supplied all training and materials to agencies, including technology component. Cost over three years:  First year $107,165  Second year $151, 975  Third year $ 48,210  Totals $ 307,350

38 Training/Evaluations  To date 236 evaluators & program staff have been trained.  Average time of eligibility evaluation has increased by 15 minutes.  Factors for increase include:  Learning curve for new evaluators  Use of technology  Use of additional tools in areas where more information is needed

39 Weighing the costs  Each evaluator one time start-up cost has been approximately $1,300 (materials & training)  Additional evaluation time (15 min * 2 evaluators) cost increase averaged $50.00 per eval.  To implement COSF or a similar procedure the projected cost is:  $100 per staff, per hour, to review & note progress on each form for each child included in Child Outcome Reporting

40 Thank you