What is Indicator 14 and why is it important? Dawn A. Rowe, NPSO Coordinator Rhode Island Advanced Transition Training Providence, RI January 26, 2011 National Post-School Outcomes Center

Session Objectives To learn what I-14 is and it’s utility To learn about federal data collection and reporting efforts for post-school outcomes To learn about Rhode Island’s data collection and reporting efforts for post-school outcomes To introduce strategies that have evidence to support positive outcomes for youth with disabilities

The National Post-school Outcomes Center [NPSO] A national technical assistance & dissemination center funded by the U.S. Department of Education Office of Special Education Programs. We help State Education Agencies develop practical, yet rigorous data collection systems to describe the further education and competitive employment experiences of youth with disabilities as they transition from high school to adult life. Akzidenz-Grotesk BQ Reg

Purpose of IDEA To ensure that all children with disabilities have available to them a free and appropriate public education that emphasizes special education and related services designed to meet their unique needs and prepare them for further education, employment, and independent living. IDEA Regulations §300.1(a)

Federal Reporting Requirements Under IDEA, States are required to submit: State Performance Plan (SPP) Annual Performance Report (APR) http://www.ride.ri.gov/OSCAS/SPPAPR/sppapr.aspx

Federal Requirements State’s performance plan and annual report are based on 20 Part B indicators 4 specific to secondary transition: 1 Percent (%) of youth who graduate from high school 2 Percent (%) of youth who drop out of high school 13 Percent (%) of youth with IEPs age 16 or above with an IEP that includes specified transition components (e.g., postsecondary goals) 14 Percent (%) of youth who achieve post-school outcomes Akzidenz-Grotesk BQ Reg

Critical Interrelationships for Achieving PSO Staying in school (Indicator 2) Quality IEPs (Indicator 13) Positive post-school outcomes (Indicator 14) Graduating (Indicator 1) Kohler (NSTTAC), 2007

Part B SPP/APR Requirements for Indicator 14 Percent of youth who are no longer in secondary school, had IEPs in effect at the time they left school, and were: Enrolled in “higher education” In “competitive employment” Enrolled in “some other postsecondary education or training” In “some other employment”

Competitive Employment Other Postsecondary Education or Training PSO Definitions full- or part-time community college (2-year program) college/university (4- or more year program) one complete term Higher Education pay at or above the minimum wage setting with others who are nondisabled 20 hours a week for at least 90 days (includes military) Competitive Employment at least 1 complete term education or training program (e.g., Job Corps, adult education, workforce development program, vocational technical school which is less than a 2-year program) Other Postsecondary Education or Training pay or been self-employed at least 90 days. This includes working in a family business (e.g., farm, store, fishing, ranching, catering services, etc.) Other Employment

Indicator 14 for Federal Reporting Click to edit Master text styles States must report a percentage for each A, B, and C in the SPP/APR and to the public. States must also provide the actual numbers for each 1, 2, 3 and 4.   States are not required to report those former students who were “doing something other than” or “not engaged” in 1, 2, 3, or 4. However, those leavers will be part of the “total respondents”.

Findings from state data collection efforts are used to: Nationally Rhode Island Report at the national, state, and local levels Guide and improve transition programs for transition age youth with disabilities Continue improvement in data collection focusing on improving representativeness Develop capacity to improve outcomes for specific populations of youth who are not engaged at the same rate as others after high school (e.g., youth with ED, AA, Hispanic) . (FFY 2010 SPP)

Collecting Post-school Outcomes Data Who are the data collected on? What data are collected? How are the data collected? When are the data collected? Who collects the data?

Who are data collected on? Nationally Rhode Island Those with IEPs who leave high school: With a diploma – regular or modified With a certificate By aging out By leaving early /dropping out Youth with IEPs who leave high school by: Graduating with a regular diploma Age out Left school early (i.e., dropped out) Akzidenz-Grotesk BQ Reg Challenge: Finding leavers one year out of school

What data are collected? Nationally Rhode Island In school: Demographic data Disability Gender Race/ethnicity Age Method of exit One year out: Higher education Competitive employment Other postsecondary education or training Other employment In school: Demographic data Disability Gender Race/ethnicity Age Method of exit Student Contacts One year out: Higher education Competitive employment Other postsecondary education or training Other employment Akzidenz-Grotesk BQ Reg

How are data collected nationally? Census v. Sample Method of Collecting Every district over the life of the SPP ADM > 50K Representative sample Disability Gender Race/ethnicity Age Survey Phone Mailed Face-to-face Web- or Internet-based Extant database Akzidenz-Grotesk BQ Reg Challenge: Finding leavers one year out of school

How are data collected in Rhode Island? Census Method of Collecting Attempt to contact every leaver in every district over the life of the SPP Use unique identifiers Survey Combination of Phone and Online survey (district personnel complete) Akzidenz-Grotesk BQ Reg Challenge: Finding leavers one year out of school

When are data collected? Nationally Rhode Island April through September When youth have been out of school for at least one year Same Akzidenz-Grotesk BQ Reg Challenge: Finding leavers one year out of school

Who collects these data? Nationally Rhode Island SEA or LEA staff Teachers and support staff Administrators Outside contractor University Survey Center Student’s last known case manager (certified special education teacher) LEA Census Clerks Special Education Administrators (Regional training provided via WebEx) Akzidenz-Grotesk BQ Reg Challenge: Finding leavers one year out of school

Results of State’s Efforts: Outcomes for Students with Disabilities as measured by Indicator 14 United States Department of Education, Office of Special Education Programs (2011). Part B State Performance Plan/ Annual Performance Report 2011 Indicator Analyses.

Rhode Island’s Post-school Outcomes Data

Rhode Island’s PSO Data These data represent youth who left school during the 2009-2010 school year. Of the 1579 leavers 1101 (70%) responded to the PSO survey. The responders were found to represent youth across the state based on disability category, gender, and minority status. Dropouts were slightly underrepresented. Of those who responded, 62% reported being enrolled in higher education, competitively employed, enrolled in some other postsecondary education or training program, or other employment since leaving high school.

How well do those who responded represent all leavers in 2009-2010? Overall Response Rate: 70% Is our sample size large enough to draw meaningful conclusions? States are asked to describe in the SPP how well their data obtained from those who respond to the survey represent the total leavers in the state. For RI in 2009-2010, of the 1579 total leavers (census), 1101 responded to the survey, for a 70% response rate. Remember, the question we want to answer is how well do those who responded represent all leavers in 2009-10? To answer this question we look beyond the response rate and compare the proportion of respondents to leavers on key characteristics: disability, gender, minority status, and method of exit. A difference of +-3% in the proportions indicates an importance difference between those who responded and the total leavers. One category exceeds that 3% difference in proportions: youth of who dropout. Differences greater than ± 3% are important

Challenges Finding leavers 1-year out of school Contacting leavers 1-year out of school

Rhode Island’s Method of Exit Of the 1100 youth who responded across the state… Facilitator's Notes: (Purpose #2) This chart shows what percentage of state youth who responded to the survey exited school by graduating, dropping out, aging out or other means of exit. If numbers do not total 100%, you should use a chart with columns (i.e., bars), not a pie chart. In most cases, the number of youth who graduate and dropout will not total 100% of the youth who left school in a year for several reasons: Differences in the definitions Differences in the data used to calculate graduation and dropout rates Some youth die, or move out of State, and may not be categorized as either.   To reach 100% on the pie chart, you may need to include categories of other methods of exit; for example, youth who reach maximum age or age out, or Other, as defined by the State or district. Numbers in the chart are place holders. Change the numbers in the graph by doing the following: -Double click on a slice of the pie graph to open the Excel file -When the Excel spreadsheet opens: -Click on the tab labeled “Sheet 1” -Change the numbers you want to change -Click on the tab labeled Pie Chart -Close the spreadsheet by clicking outside of the Excel box Data Source: RI PSO SY 2009-10

Rhode Island’s Engagement Rate: Of the 1100 respondents Convert percents to numbers Change data RI PSO SY 2009-10

Let’s look at examples of how data can be disaggregated. So What?? You are probably asking: What do these data tell me? What use are these data? How can these data be used to improve programs for youth with disabilities? Engagement rate alone is insufficient to inform program decisions. The engagement rate should be disaggregated by subgroups of students based on key characteristics, such as demographic data, geographic location, or other relevant information. Let’s look at examples of how data can be disaggregated.

Are Males & Females engaged at the same or similar rate? Brainstorm questions to answer about the engagement rates of males and females. Do we do better with certain disability groups or genders? Facilitator's Notes: (Purpose #3) Ask the stakeholder group to brainstorm the questions they would like to answer about the engagement rates of males and females. You may ask people to write questions on sticky notes and read them aloud or display them on chart paper.   Place the group’s questions on chart paper. Using the data, answer the questions. Examples of questions: Are males and females competitively employed at approximately the same rate? If not, which group is employed at a higher rate? What do you think contributes to the higher employment rate for some groups? What could the district do to promote competitive employment for groups with a lower employment rate?

Percent of Males & Females Engaged Engagement Rate Facilitator's Notes: (Purpose #3) Take a couple of minutes to make observations about what’s displayed. Record observations on chart paper. Of the questions the group wants to answer, do the data answer any of the group’s questions? Numbers in the chart are place holders. Change the numbers in the graph by doing the following: Right click on a bar in the graph Select “edit data” When the Excel spreadsheet opens: Change the numbers you want to change Push the return key on the keyboard Close the spreadsheet 28 Data Source: RI PSO SY 2009-10 28

Are youth with various disabilities engaged at the same or similar rate? Brainstorm questions to answer about the engagement rates of youth with different disabilities. Do we do better with certain disability groups or genders? Facilitator's Notes: (Purpose #3) Ask the stakeholder group to brainstorm the questions they would like to answer about the post-school outcomes for youth of different disability categories. You may ask people to write questions on sticky notes and read them aloud or display them on chart paper.   Place the group’s questions on chart paper. Using the data, answer the questions.

Engagement Rate by Disability Categories AO = All Other Disabilities Engagement Rate Facilitator's Notes: (Purpose #3) Take a couple of minutes to make observations about what’s displayed. Record observations on chart paper. Of the questions the group wants to answer, do the data answer any of the group’s questions? Numbers in the chart are place holders. Change the numbers in the graph by doing the following: Right click on a bar in the graph Select “edit data” When the Excel spreadsheet opens: Change the numbers you want to change Push the return key on the keyboard Close the spreadsheet 30 Data Source: RI PSO SY 2009-10 30

Are youth from various race or ethnic groups engaged at the same or similar rate? Brainstorm questions to answer about the engagement rates of youth from different backgrounds. Do we do better with certain groups of youth? Facilitator's Notes: (Purpose #3) Ask the stakeholder group to brainstorm the questions they would like to answer about the post-school outcomes for youth of different disability categories. You may ask people to write questions on sticky notes and read them aloud or display them on chart paper.   Place the group’s questions on chart paper. Using the data, answer the questions.

Engagement Rate by Ethnicity Categories AO = All Other Disabilities Engagement Rate Facilitator's Notes: (Purpose #3) Take a couple of minutes to make observations about what’s displayed. Record observations on chart paper. Of the questions the group wants to answer, do the data answer any of the group’s questions? Numbers in the chart are place holders. Change the numbers in the graph by doing the following: Right click on a bar in the graph Select “edit data” When the Excel spreadsheet opens: Change the numbers you want to change Push the return key on the keyboard Close the spreadsheet 32 Data Source: RI PSO SY 2009-10 32

How can disaggregated PSO data be used? What are other states doing?

How other states are using PSO data to inform transition programs NC revised their PSO survey to collect more programmatic data to examine whether in-school programs and services correlate with better post- school outcomes. MD is sharing PSO data with the state Interagency Council to inform and improve the cross-agency collaborative services provided while youth are in-school.

How other states are using PSO data to inform transition programs SD compared outcomes of students who participated in certain transition programs to those who did not to determine program effectiveness.

Let’s Look Closer: South Dakota Question: Are youth who participate in 4 state specific in- school programs more likely to be engaged one-year out of school than youth who do not participate in these programs? Programs: Catch The Wave, Youth Leadership Forum, Self-Advocacy, & Project Skills Used an odds ratio to measure effect size, describing the strength of association between two dichotomous values: youth engaged (i.e., working or enrolled) versus youth not engaged (i.e. not working or enrolled).

OVERALL for 08-09: Odds Ratio of engagement for students who participated in Catch The Wave, Youth Leadership Forum, Self-Advocacy, or Project Skills # engaged # not engaged Participate in CTW, YLF, Self-Advocacy, or Project Skills 177 19 Did not participate in CTW, YLF, Self-Advocacy, or Project Skills 120 26 177/19 120/26 = 9.316/4.615 OR of 2.03 n= 342 Those who participated in these programs were 2.03 times more likely to be engaged in work or school than those who did not participate in these programs. 177/19 120/26 = 9.316/4.615 OR of 2.03

Empirical Evidence South Dakota has empirical evidence that youth who participated in Catch The Wave, Youth Leadership Forum, Self-Advocacy, or Project Skills were consistently more likely to be positively engaged 1-year out of high school than youth who did not participate in these programs. With sufficient data, you could do something similar to examine whether the programs offered in RI or your district contribute to positive post-school outcomes.

The Logic of Using PSO Data for Program Improvement Data Collection Positive Post-school Outcomes Data Collection Data Use Positive Post-school outcomes

Thinking about Transition Related Program Improvement In what areas are youth with disabilities doing well? What areas need improvement? What is the state/district doing well? What does the state/district need to improve? Facilitator's Notes: (Purpose #4) Place each question on individual sheets of chart paper. Ask the group to answer each question. Depending on group size, you may want them answer on sticky notes, to allow individual think time, or go one by one around the group. The goal with this slide is to start to look critically at the district or school program and identify how the program contributes to positive post-school outcomes and how it does not. If you have several areas that need improvement, you may need to prioritize the needs. There are several criteria that can be used to prioritize needs. Examples of criteria include: Resource investment (people, money, or equipment/materials): no additional resources, few resources, or significant resources time investment: changes can be addressed immediately, in the short-term (defined as ….), or in the long-term (defined as …) Degree of change anticipated Number of youth/families affected

How do you make the connection between post-school outcomes of youth with disabilities and practices implemented in the district, school, or classroom?

Evidence-Based (Special)Education… …is a decision-making process for deciding what to teach based on: frameworks or principles best available research evidence professional judgment student needs and values A third response has been to look at EBP as a more generic, decision-making process…two examples… 1. See TEC, Jan/Feb, 2010, pages 42-53, “Knowledge-to-Action Guides” (Buysse, Wesley, Snyder, & Winton, 2006; Cook, 2010 Detrich, Spencer, & Slocum, 2009;Turnbull et al., 2010; Cook, 2010)

Create Individualized Interventions Adapt Interventions Frameworks Provide a Systematic Basis for using Professional Judgment to: Select Interventions Create Individualized Interventions Adapt Interventions (Spencer, 2009)

Student-Focused Planning Student Development Family Involvement A Framework for Secondary Transition: The Taxonomy for Transition Programming Student-Focused Planning Student Development Family Involvement Interagency Collaboration Program Structure Provides a framework to guide secondary transition instructional planning and practices (Kohler, 1996)

Need to Consider Inputs Outputs Outcomes OAVSNP, 2010 Keynote Deanne Unruh, Ph.D., dkunruh@uoregon.edu

Inputs Outputs Outcomes OAVSNP, 2010 Keynote What resources do we have in place or need? Inputs What strategies/ evidence-based practices will we use? Outputs What student outcomes do we need to improve? Outcomes Deanne Unruh, Ph.D., dkunruh@uoregon.edu

Focus on What you Can Control Post-School Outcomes Predictors Practices Lesson Plans

PREDICTORS Post-School Outcomes Independent Living Employment Education Independent Living

In-School Predictors by Post-School Outcome Area Predictors/Outcomes Education Employment Independent Living Academic/General Education x Career Awareness Community Experiences Exit Exam Requirements/High School Diploma Status Interagency Collaboration* Occupational Courses Paid Work Experience* Parental Involvement School Integration*

In-School Predictors by Post-School Outcome Area Predictors/Outcomes Education Employment Independent Living Self-Advocacy/Self-Determination* x Self-Care/Independent Living Social Skills Student Support* Transition Program* Vocational Education* Work Study*

Linking Post-School Outcome Data to Evidence-based Practices and Predictors How do I do this?

How do Predictors Apply to Transition Planning and Instruction? OAVSNP, 2010 Keynote How do Predictors Apply to Transition Planning and Instruction? Provides practitioners information about secondary transition program characteristics that have been empirically linked to improved post- school success for students with disabilities Can be used to: develop and expand programs evaluate existing programs Can help IEP teams design annual IEP goals and transition services that are more likely to help students achieve their stated post-school goals Deanne Unruh, Ph.D., dkunruh@uoregon.edu

Predictors of Post-School Success for Students with Disabilities Youth Transition Program (YTP) Transition planning focused on post-school goals Instruction in academic, vocational, independent living, and personal social areas Paid job training while in the program Assistance to secure employment or enter postsecondary education upon leaving the program Follow up support for up to 2 years after leaving the program Transition to independence Process System (TIP) Relies on a transition facilitator/specialist Youth-centered and focused on youth engagement Transition plans developed that access various service agencies based on unique needs of individual

Sample State’s Engagement Rates Of the 400 youth who responded to the interview/survey across the state… Facilitator's Notes: (Purpose #2) This chart shows what percentage of state youth who responded to the survey were in each engagement category (i.e., higher education, competitive employment, postsecondary education or training, and other employment). If numbers do not total 100%, you should use a chart with columns (i.e., bars), not a pie chart. In most cases, the categories of higher education, competitive employment, other employment, and postsecondary education or training will not total 100% of youth who left school. To reach 100%, you may need to include an additional category representing the remaining percentage of youth surveyed. In this example, we label the category not engaged to represent those youth who are not counted in one of the other categories. The state may prefer to use labels such as Other or Neither. If these categories are used in the State, define who is represented in these categories.   For example, Other may include youth who are incarcerated. Neither may include youth who are not engaged in any positive post-school outcome. Numbers in the chart are place holders. Change the numbers in the graph by doing the following: -Double click on a slice of the pie graph to open the Excel file -When the Excel spreadsheet opens: -Click on the tab labeled “Sheet 1” -Change the numbers you want to change -Click on the tab labeled Pie Chart -Close the spreadsheet by clicking outside of the Excel box Data Source: Sample PSO

TIP Youth Transition Program Interagency Collaboration Increased Competitive Employment TIP Youth Transition Program Interagency Collaboration

Rhode Island’s Engagement Rate: Of the 1100 respondents Convert percents to numbers Change data RI PSO SY 2009-10

What is Rhode Island doing that could lead to higher engagement after high school? Community-based Training (e.g., The Sports Exchange) Post-School Engagement Rhode Island Transition Academies Career Development Project Rhode Island Guides for Students in Transition Rhode Island “My Transition” website Check out the new Rhode Island web site for students www.ritap.org/MyTransition Being informed is the best way to advocate for the services and programs you may need to effectively transition from high school to employment, post-secondary education & training and community living. This page is divided into sections that may help you navigate the transition process and meet with success in the transition from school to adult life Understanding the Transition Process The Rhode Island Department of Education works closely with school districts, public adult services agencies, parent organizations, universities and colleges to provide current and useful information. One tool that many young adults have found useful is and on-line tool called Youthhood. At Youthhood you will find tools to assist you in planning for your life after high school. It is a great place to start. The RI Transition Academies are a unique educational program that offers students with disabilities, 18 to 21 years old,  the opportunity to complete their high school education on a college campus and/or in various employment and community settings. It is geared toward the student who has completed or nearly completed the academic courses necessary for graduation and would benefit from an additional one or two years to acquire the functional life skills, vocational skills and social skills necessary to be better prepared for adult life. Potential student candidates must be in special education with an Individual Education Plan (IEP), be currently enrolled in school and near completion of their academic program. Candidates must also be eligible for services through the Office of Rehabilitation Services and must have a current application with the Office of Rehabilitation Services. College Resources & planing: This directory helps students who receive supports through Individual Education Programs (IEPs), or 504 Plans, to learn about post-secondary education opportunities. This guide will help to: - Plan and prepare for post-secondary education - Identify available supports to aid access to post-secondary education - Find disability-related information about colleges in Rhode Island Employment Resources and Planning: Designed for students in transition, this guide walks them through the steps of planning a career and conducting an employment search. The workbook format encourages students to enter their own information for future use. This guide complements School to Career training programs and curricula. Living & particpating in your Community: this guide will help students plan and access the supports they need to live and participate in their communities. Rhode Island Transition Council Regional Transition Centers

Predictors Post-School Outcomes Evidence-based Practices Lesson Plans “While the evidence-based practices were designed to teach students specific transition-related skills, to date, the experimental literature has not attempted to measure the impact of these skills on post-school outcomes.” “Therefore, NSTTAC began examining correlational research on in-school predictors of post-school success.” Lesson Plans

Have used rigorous research designs Have demonstrated a record of success for improving student outcomes Have undergone systematic review process using quality indicators to evaluate level of evidence Evidence-Based Practices Research-Based Practices Developed based on research Have demonstrated limited success for effectiveness Have used a ‘weak’ research design Promising Practices Are not based on research No data have been collected Based on Anecdotal evidence and professional judgment exists Unestablished Practices Evidence-Based Practices Have used rigorous research designs Have demonstrated a record of success for improving student outcomes Have undergone systematic review process using quality indicators to evaluate level of evidence Research-Based Practices Promising Practices Developed based on research Have demonstrated limited success for effectiveness Have used a ‘weak’ research design Unestablished Practices Are not based on research No data have been collected Based on Anecdotal evidence and professional judgment exists (Helsel, Hitchcock, Miller, Malinow, & Murray, 2006; Lembke & Stormont, 2005; Twyman & Sota, 2008)

Let’s examine a few of these practices Using Self-Advocacy Strategy to teach Student involvement in the IEP Self-Advocacy Strategy (SAS) consists of 5 steps taught over a series of seven acquisition and generalization stages: I – Inventory P – Provide L – Listen and Respond A – Ask Questions N – Name your goals

Let’s examine a few of these practices Using Whose Future is it Anyway? to increase self-determination skills and transition planning knowledge Whose Future is it Anyway? is a student directed transition planning curriculum comprised of six sections: Having self-awareness and disability awareness Decision making about transition-related outcomes Identifying and securing community resources to support transition services Writing and evaluating goals and objectives Communicating effectively in small groups Developing skills to become and effective team member, leader, or self-advocate Available: http://www.ou.edu/content/education/centers-and- partnerships/zarrow/self-determination-education-materials/whos-future-is-it- anyway.html

How does EBP Apply to Transition Planning and Instruction? OAVSNP, 2010 Keynote How does EBP Apply to Transition Planning and Instruction? Provides teachers information about secondary transition evidence-based practices for teaching students with disabilities Can be used to: Support IEP goals and objectives Support skill development Deanne Unruh, Ph.D., dkunruh@uoregon.edu

Resources Jane Slade Rhode Island Department of Education National Post-School Outcomes Center www.psocenter.org National Secondary Transition Technical Assistance Center www.nsttac.org National Dropout Prevention Center for Students with Disabilities www.ndpc-sd Facilitator's Notes: These are three federally funded technical assistance and dissemination centers charged with helping States and LEAs address post-school outcomes (NPSO), effective transition planning (NSTTAC), and graduation and dropout prevention (NDPC-SD).   Contact any of the centers for additional information and resources.

Dawn A. Rowe 541.346.8412 drowe3@uoregon.edu