Using Individual Project and Program Evaluations to Improve the Part D Programs Dr. Herbert M. Baum

Session Theme # 3 Part D programs have their own priorities for evidence, which project directors need to support

Objectives By the end of this presentation you will: Know the difference between program measurement and program management Understand how to use logic models for developing program measures See examples of how current program information is being used to enhance the Part D Programs.

The trickle up approach (1) Each OSEP/RTP Program has a logic model Each OSEP/RTP Program has an approved series of performance measures. Each OSEP/RTP Program funds projects

The trickle up approach (2) Number and Type of Measures by OSEP/RTP Program ProgramAnnualLong-termEfficiency Personnel Development 521 PTIC321 TA&D321 Tech and Media 321

The trickle up approach (3) How does each OSEP/RTP Program know that they are meeting their targets? o Your projects need to provide that How does each OSEP/RTP Program use the data to manage? o Keeping project officers informed of where the Program is relative to the target. o Using the measures to direct new grant applications to address how they will help OSEP/RTP meet their targets. o By meeting targets OSPE/RTP is in a stronger position to ask for additional funding.

Performance Measurement

Performance Management

Evaluation in Performance Management

Performance Management Steps 1.Assess – review program purpose and design 2.Plan – set strategic performance targets 3.Measure – measure program performance 4.Analyze – evaluate program results 5.Improve – implement program enhancements 6.Sustain – manage program effectiveness

Evaluation in the Context of PART The Program Assessment Rating Tool (PART) was developed to assess the effectiveness of federal programs, and help inform management actions, budget requests, and legislative proposals directed at achieving results. The PART Assess if and how program evaluation is used to inform program planning and to corroborate program results. 4 sections of a PART review: o Program purpose and design (20%) o Strategic planning (10%) o Program management (20%) o Program results (50%)

Evaluation and PART (2) Question 2.6 “…[a]re independent evaluations of sufficient scope and quality conducted on a regular basis, or as needed to support program improvements, and evaluate effectiveness and relevance to the problem, interest, or need.” Question 4.5 asks if “independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results.”

What is a logic model? The underlying rationale for the evaluand's design, usually an explanation of whythe various components of the program (for example) have been created and what-and how-each of them is supposed to contribute towards achieving the desired outcomes. Some logic models include environmental factors, some do not. Note that we are talking about the alleged ‘theory of operation’ and the evaluation may discover considerable discrepancies between this-the view of the designers and possibly also the managers of the program-and the views of the service deliverers who are the hands-on staff engaged in dealing with the recipients of service or product.”

Simplified Logic Model Inputs What the program needs to accomplish it outcomes. Activities What programs do to accomplish their outcomes. Outputs What programs produce to accomplish their outcomes. Outcomes What changes the program expects based on their inputs, activities and outputs. [Short- term, Intermediate, and Long- term (impact)]

*Fully Qualified = Highly Qualified for special education teacher; Qualified for paraprofessional/aide; Fully Certified for administrator/coordinator, for related or supportive services in a school setting, or for teacher, related services, or supportive services in early intervention, early childhood. OSEP Personnel Development Program - Logic Model A blueprint to enhance understanding of the Program Goal: To improve results for children with disabilities and their families INPUTS OUTPUTSOUTCOMES Program Investments ActivitiesParticipationShort TermLong Term Project Officers Funding Evidence-Based & Best Practices Research Program & Grants Policy Technology Process Measures Outcome Measures Intermediate Increased supply of fully qualified personnel* with awareness and knowledge of EBP & best practices Increased collaboration - SEAs, IHEs, LEAs & lead agencies Increased training opportunities Increased placement of fully qualified* personnel Improved personnel development infrastructures Increased retention of fully qualified* personnel in workforce – schools & programs, educational & lead agencies, & IHEs. Grantees Faculty Students in IHEs SEAs & LEAs Lead Agencies Practitioners Administrators Children Families Train personnel Redesign & build models & networks for collaboration Develop and disseminate resources CONTEXT Federal Law & Regs Time Develop priorities & manage competitions Monitor grants Build models & networks for collaboration

Measure 1.1 (Annual) The percentage of Special Education Personnel Preparation projects that incorporate evidence- based practices in the curriculum. Home Measure 1.2 (Long-Term) The percentage of scholars completing Special Education Personnel Preparation funded training programs who are knowledgeable and skilled in evidence-based practices for infants, toddlers, children, and youth with disabilities.

Measure 2.1 (Annual) The percentage of Special Education Personnel Preparation funded scholars who exit training programs prior to completion due to poor academic performance. Home

The percentage of low incidence positions that are filled by personnel who are fully qualified under IDEA. Measure 2.2 (Long-Term) Home The percentage of Special Education Personnel Preparation funded degree/certification recipients who are working in the area(s) for which they were trained upon program completion and who are fully qualified under IDEA. Measure 2.4 (Annual)

The percentage of degree/certification recipients who maintain employment for 3 or more years in the area(s) for which they were trained and who are fully qualified under IDEA. Measure 2.5 (Long-Term) Home

Example from the Personnel Development Program Measure 1.1 of 2 (Annual) The percentage of Special Education Personnel Preparation projects that incorporate evidence-based practices in the curriculum.

Example from the TA& D Program Measure 1.1 of 1 (Long-term) The percentage of school districts and service agencies receiving Special Education Technical Assistance and Dissemination services regarding scientifically- or evidence-based practices for infants, toddlers, children and youth with disabilities that implement those practices.

Questions Contact information Herbert. M. Baum, Ph.D. ICF Macro