325K: Combined Priority for Personnel Preparation Webinar on the Annual Performance Report for Continuation Funding Office of Special Education Programs U.S. Department of Education Washington, DC February 22, 2012

O S E P Annual Performance Report (APR)   An annual report of your performance in meeting the approved objectives of the project and responsible use of federal funds   Required for all active grants, including those in no cost extension (NCE)   OSEP reviews the report to determine if substantial progress has been made in order to grant continued funding or a NCE 2

O S E P Overview   Recognize strong project objectives that can be associated with high-quality performance measures   Develop relevant, measurable, outcome- oriented performance measures related to your objectives that maximize the potential for meaningful data and positive outcomes   Complete the ED Grant Performance Report (ED 524B) 3

O S E P Why Is This Important? High-quality objectives and measures …   Make it easier for you to measure your progress for the purpose of grant management   Allow you to report progress easily and quantitatively   Establish targets (both short-term/annual and long- term)   Allow OSEP staff to gather evidence of program effectiveness 4 Taken from the Center for Evaluation and Education Policy (CEEP) presentation at

O S E P Goal – Objectives - Measures 5 Taken from the Center for Evaluation and Education Policy (CEEP) presentation at

O S E P Project Objectives What are you trying to accomplish? Objectives should answer this question. Preferred format for objectives: Begin the objective with a verb and define a desired outcome or condition 6

O S E P Attributes of High-Quality Project Objectives   Relevance How relevant is the project objective to the overall goal of the program and/or the goal of your project?   Applicability How applicable is the project objective to the specific activities that are being conducted through your particular project? 7 Taken from the Center for Evaluation and Education Policy (CEEP) presentation at

O S E P Attributes of High-Quality Project Objectives   Focus How focused is the project objective?   Measurability Are there concepts in the project objective that lend themselves to measurement? If so, is measurement feasible? 8 Taken from the Center for Evaluation and Education Policy (CEEP) presentation at

O S E P Project Objectives -- Examples   Recruit and retain high-quality scholars   Implement a high-quality personnel development program that is aligned with State and national professional organization standards   Prepare high-quality personnel to serve children with disabilities and their families   Disseminate information and outcomes of the program within State and nationally 9

O S E P Performance Measures How are you measuring your progress in meeting your objectives? Performance measures should answer this question. 10

O S E P Performance Measures are …   Measurable indicators used to determine how well objectives are being met.   Important to establish -   How progress will be assessed   How much progress will constitute success   How will it be known if an objective or part of an objective has been achieved 11 Taken from the Center for Evaluation and Education Policy (CEEP) presentation at

O S E P Performance Measures 12 Taken from the Center for Evaluation & Educational Policy (CEEP) at Indiana University

O S E P 2 Types of Performance Measures   Program All grantees funded under the Personnel Development Program (PDP) must report on the PDP program performance measures established by OSEP.   Project Each grantee reports on the approved project performance measures established to meet their project objectives. 13

O S E P Program Performance Measures   Program Measures were established by OSEP for the Personnel Development Program. Measures apply to all grants funded under the Personnel Development Program. Results on these measures are reported to Congress under the Government Performance and Results Act of

Personnel Development PROGRAM Performance Measures #1: The percentage of Special Education Personnel Development projects that incorporate evidence-based practices into their curricula. #2: The percentage of scholars completing Special Education Personnel Development funded programs who are knowledgeable and skilled in evidence-based practices for infants, toddlers, children, and youth with disabilities. #3: The percentage of Special Education Personnel Development funded scholars who exit preparation programs prior to completion due to poor academic performance. #4: The percentage of Special Education Personnel Development funded degree/certification program recipients who are working in the area(s) in which they were prepared upon program completion. #5: The percentage of Special Education Personnel Development funded degree/certification recipients who are working in the area(s) for which they were prepared upon program completion, and who are fully qualified under IDEA. #6: The percentage of Special Education Personnel Development degree/certification recipients who maintain employment in the area(s) for which they were prepared for 3 or more years, and who are fully qualified under IDEA. #7: The Federal cost per fully qualified degree/certification recipient. 15

O S E P Program Performance Measures   OSEP collects data on the program performance measures each year.   Grantees should propose a project performance measure that addresses each program performance measure reported for the 325K program.   The focus of the project measure must be the same as and clearly aligned with the program measure. 16

O S E P Program Performance Measure #1. The percentage of Special Education Personnel Development projects that incorporate evidence-based practices into their curricula. Example of related project performance measure: By the beginning of Year 3, 100% of syllabi from core courses will be rated as “evidence-based” by the Project Advisory Board using a rubric designed by the Project. 17

O S E P Program Performance Measure #2. The percentage of scholars completing Special Education Personnel Development funded programs who are knowledgeable and skilled in evidence-based practices for infants, toddlers, children, and youth with disabilities. Example of related project performance measure: At the end of their program, 100% of scholars will achieve a minimum of 3 on a 4-point rubric for a portfolio that provides evidence of their competencies in providing interventions to children with disabilities. 18

O S E P Program Performance Measure #3. The percentage of Special Education Personnel Development funded scholars who exit preparation programs prior to completion due to poor academic performance. Example of related project performance measure: As measured annually, no scholars (0%) will have exited the program due to poor academic or field-based performances. 19

O S E P Program Performance Measure #4. The percentage of Special Education Personnel Development funded scholars who are working in the area(s) in which they were prepared upon program completion. Example of related project performance measure: By the end of Year 5 of the project, 90% of scholars will be working in the area for which they were trained following program completion. 20

O S E P Program Performance Measure #5. The percentage of Special Education Personnel Development funded degree/certification recipients who are working in the area(s) for which they were prepared upon program completion, and who are fully qualified under IDEA. Example of related project performance measure: By the end of Year 5 of the project, 90% of program graduates will be working in the area for which they were trained and fully qualified under IDEA. 21

O S E P Program Performance Measure #6. The percentage of Special Education Personnel Development degree/certification recipients who maintain employment in the area(s) for which they were prepared for 3 or more years, and who are fully qualified under IDEA. Example of related project performance measure: By the end of Year 5 of the project, 90% of eligible program graduates will have maintained employment in the area for which they were trained for 3 or more years, and be fully qualified under IDEA. 22

O S E P Program Performance Measure #7. The Federal cost per fully qualified degree/certification recipient. (Efficiency Measure) Example of related project performance measure: The Federal cost is measured in Year 5 by dividing the total grant award (e.g., $250,000 x 5) by the number of scholars counted for Performance Measure #2. 23

O S E P Project Performance Measures   Project Measures that the grantee establishes to meet their project objectives   P roject performance measures can address both the process of working towards an objective and the outcome related to meeting the objective   Ensure a mix of both process and outcome measures, but most will be outcome 24 Taken from the Center for Evaluation and Education Policy (CEEP) presentation at

O S E P High-Quality Performance Measures High-quality performance measures show:   What will change.   How much change you expect.   Who will achieve the change.   When the change will take place. 25 Taken from the Center for Evaluation and Education Policy (CEEP) presentation at

O S E P Project Performance Measure Process measure example - Project staff (who) will implement 4 (how much) targeted recruitment activities (what ) during the first year of the grant (when). 26

O S E P Project Performance Measure Outcome measure example – Inquiries from potential candidates (who) about the program (what) will increase 20% from previous recruitment efforts (how much) during the first year of the grant (when). 27

O S E P Project Performance Measure Outcome measure example – At the end of each semester (when), 100% of scholars (who) will rate their mentoring support (what) as 5 or higher on a 7-point scale (how much) using an on-line survey developed for the project. 28

O S E P Project Performance Measure Outcome measure example – At the end of their program (when), 90% of scholars (who) will demonstrate advanced critical reflection skills (what) by achieving a proficiency score of at least 3 on a 4-point rubric (how much) developed by the project. 29

O S E P Project Performance Measure Process measure example – By the end of the project period (when), the project staff (who) will present on activities and outcomes (what) of the project at a minimum of 3 state and national meetings (how much). 30

O S E P Common Problems   Activities are NOT performance measures   If the best response is “Yes, we did that,” it is likely an activity (not a performance measure)   Examples of activities:   Scholars complete required coursework   Hold an Advisory Board meeting quarterly   Establish a mentoring program   Evaluate the project 31 Taken from the Center for Evaluation and Education Policy (CEEP) presentation at

O S E P Common Problems   Performance measures need to be measurable   Examples with measurement problems (activities rather than outcomes):   Maintain meaningful collaborative partnerships   Provide academic support in areas of concern to scholars   Establish field placements in low income schools 32 Taken from the Center for Evaluation and Education Policy (CEEP) presentation at

O S E P Need additional information on writing performance measures?   All grantees are strongly encouraged to seek training on writing performance measures.   For further information on developing performance measures and logic models, see

O S E P Summary   Projects should have a few clear objectives that explain what the project is doing to support their overall goal(s)   Each objective should have a few, specific performance measures to demonstrate how progress toward meeting the objective is being measured   Both program and project performance measures are reported 34 Taken from the Center for Evaluation and Education Policy (CEEP) presentation at

O S E P Completing the ED 524B The ED 524B is a required annual and final performance reporting form with specific instructions.   The form is used by all ED grants and has been approved by the Office of Management and Budget (OMB). Project Directors must follow the directions listed in the Dear Colleague letter and ED 524B Instructions provided by OSEP   Word or PDF versions of the forms are available at

O S E P x 36 X

Reporting Period: For first year grants, the date is the beginning of the project year to February 29th. For grants in years 2-5, it is the date from the end of the previous reporting period to February 29th. ANNUAL PERFORMANCE REPORTS __

ANNUAL PERFORMANCE REPORTS Budget Expenditures: Report the expenditures during the “Reporting Period.” Must be data or information from the business or grants office. 38

Signatory must have authority to sign on behalf of the institution since the grant is from the Department to the institution and not to an individual. The Authorized Representative signs; not the Project Director. Performance Measure Status: This will be checked “No” since OSEP is asking for data for the award reporting period, not for the budget period. The date entered here will be the due date for your Final Performance Report; which is 90 days after the end of the grant. ANNUAL PERFORMANCE REPORTS 39

EXECUTIVE SUMMARY SHEET 40 OMB No Exp. 02/28/2011 OMB No Exp. 02/28/2011 *** Provide highlights of the project's activities and the extent to which the expected outcomes and performance measures were achieved during the reporting period. Do NOT include the project abstract. H325 K

PROJECT STATUS CHART 41 H325 K

Enter one of the project’s objectives; on subsequent pages, you will enter additional project objectives as submitted in your grant application. PROJECT STATUS CHART 42 H325 K

Enter the PROGRAM performance measure(s) that align with the objective immediately followed by any PROJECT measure(s) developed to address that program measure. Enter any additional PROJECT performance measures that show you are measuring progress toward meeting the objective (e.g., 1c., 1d.,). PROJECT STATUS CHART 43

Here you identify if the performance measure is a PROGRAM measure, “PRGM,” or a PROJECT measure, “PROJ.” Note: Program measure refers to one of OSEP’s 7 performance measures for the Professional Development Program. Project measures are unique to your grant. PROJECT STATUS CHART 44 PRGM PROJ

PROJECT STATUS CHART QUANTITATIVE DATA 45 Depending on your measure, enter either a raw number, or a ratio and percentage. Enter the target number identified in the performance measure and then the actual data for this year. If complete data are not available for the measure, enter “999” (if no baseline) or “NA” in the “Raw Number” or “%” column, as appropriate. Provide an explanation at the bottom of the page under “Explanation of Progress.” 86

O S E P Information to Include in the Explanation of Progress Section  Describe the data provided (e.g., what data collection methods were used, when were the data collected, how was a sample drawn, are there missing/incomplete data, what was the response rate, was a reliability measure taken). Your Project Officer should be able to understand and interpret the numbers in the chart from your description in this section.  What changes in the data occurred since last APR (i.e., trend)?  What activities were undertaken to achieve the targets?  If targets were not met, what are possible reasons?  How will activities that failed to meet targets be improved? 46

QUALITATIVE DATA If measure requires the collection of qualitative data, report the performance measure and type (PROG or PROJ) and then, enter “N/A” under the Raw Number and Percentage columns. N/A In the “Explanation of Progress” section of the page, referencing the performance measure by number, report applicable qualitative data along with other information about how these data were collected, targets and activities –refer to previous slide for additional content requirements. PROJECT STATUS CHART 47

O S E P By Year 3,

O S E P Explanation of Progress (Include Qualitative Data and Data Collection Information) (1b) The program faculty are committed to reviewing the entire curriculum of our personnel development program and ensuring evidence-based practices are integrated into core courses, including field experiences. As part of that plan, 100% of the syllabi would be thoroughly reviewed by the Project Advisory Board using an evidence-based practices (EBP) rubric developed for this Project to decide whether revision was necessary to meet our new state professional practice standards for licensure which include knowledge and skills on the use of EPB. By the end of Year 2, a full review of all 20 syllabi (100%) was completed and it was determined that 18 met the target (90%); 2 require a revision to include EBP. We attached the rubric to Section C of this report. (1c) Since this grant is designed with a focus on universal designs for instruction, a required seminar on this topic was identified for more in-depth evaluation. The seminar was offered during the Fall 2010 semester. All students in the course completed the rating. Given that the target was not met, students will be interviewed to see how the seminar might be improved for future scholars, with the goal of meeting the “usefulness” score target of 4.5 the next time the class is offered and evaluated. 49 Implement a high-quality personnel development program that is aligned to State and national professional organization standards H325 K

Final Page of the Report Section B: Refer to the instructions for Section B with the ED 524B Section C: Refer to the instructions for Section C with the ED 524B—put attachments here (e.g., recruitment material, evaluation instruments, journal articles) 50 H325 K

O S E P Section B – Budget Information This section is never blank! A table can be helpful!   Provide actual expenditures for this reporting period (through 2/29/2012)   Estimate anticipated expenditures for the rest of this budget period and balance remaining, if any.   Explain why you did not expend funds at the expected rate.   Indicate how you plan to use the unexpended funds (carryover) in the next budget period. 51

O S E P Section B – Budget Information   Describe any significant changes to your budget resulting from modifications of project activities.   Describe any changes to your budget that affect your ability to achieve your approved project activities and/or project objectives.   Describe any anticipated changes in your budget for the next budget period that require prior approval from the Department. Any questions … Talk to your Project Officer 52

O S E P Section C – Additional Information   Provide a list of current partners on your grant and indicate if:   Any partners changed during the reporting period. If there were changes, please describe both the changes and any impact that resulted in your ability to achieve approved project objectives and/or project activities.   Any partners are anticipated to change during the next budget period. If so, please describe both the changes and any impact the change might have on your ability to achieve approved project objectives and/or project activities.   Describe any changes that you wish to make in the grant’s activities for the next budget period that are consistent with the scope and objectives of your approved application. 53

O S E P Section C – Additional Information   If requesting changes to the approved Project Director and/or other key personnel, please include the person’s name, title, and contact information. Indicate his/her proposed start date, and percentage of time working on the grant, and attach a resume or curriculum vitae to the annual performance report being submitted.   Do not report on any key personnel changes that were already made during the current or previous budget period(s).   Note: Departmental approval must be requested and received prior to making key personnel changes.   Provide any other information about your project including unanticipated outcomes or benefits 54

O S E P Submitting the ED 524B   Submit the ED 524B at   Instructions for using G5 are in the continuation packet.   Signed ED 524B Cover Sheet must be scanned and ed in PDF format to your Project Officer.   Special cases require regular submission of the 524B and signed cover sheet in PDF format to your Project Officer rather than submission through G5 –   Final Performance Reports or APRs for grants in their last performance period, or no-cost extension   Grants that have been front-loaded (forward-funded) last year 55

Due Date: April 27, 2012 (Friday) Contact your OSEP Project Officer with any questions! 56