1. What is Indicator 14 and why is it important?
Dawn A. Rowe, NPSO Coordinator
Rhode Island Advanced Transition Training
Providence, RI
January 26, 2011
National Post-School Outcomes Center

2. Session Objectives To learn what I-14 is and it’s utility
To learn about federal data collection and reporting efforts for post-school outcomes
To learn about Rhode Island’s data collection and reporting efforts for post-school outcomes
To introduce strategies that have evidence to support positive outcomes for youth with disabilities

3. The National Post-school Outcomes Center [NPSO]
A national technical assistance & dissemination center funded by the U.S. Department of Education Office of Special Education Programs. We help State Education Agencies develop practical, yet rigorous data collection systems to describe the further education and competitive employment experiences of youth with disabilities as they transition from high school to adult life.
Akzidenz-Grotesk BQ Reg

4. Purpose of IDEA
To ensure that all children with disabilities have available to them a free and appropriate public education that emphasizes special education and related services designed to meet their unique needs and prepare them for further education, employment, and independent living.
IDEA Regulations §300.1(a)

5. Federal Reporting Requirements
Under IDEA, States are required to submit:
State Performance Plan (SPP)
Annual Performance Report (APR)

6. Federal Requirements
State’s performance plan and annual report are based on 20 Part B indicators 4 specific to secondary transition: 1
Percent (%) of youth who graduate from high school 2
Percent (%) of youth who drop out of high school 13
Percent (%) of youth with IEPs age 16 or above with an IEP that includes specified transition components (e.g., postsecondary goals) 14
Percent (%) of youth who achieve post-school outcomes
Akzidenz-Grotesk BQ Reg

7. Critical Interrelationships for Achieving PSO
Staying in school
(Indicator 2)
Quality IEPs
(Indicator 13)
Positive post-school outcomes
(Indicator 14)
Graduating
(Indicator 1)
Kohler (NSTTAC), 2007

8. Part B SPP/APR Requirements for Indicator 14
Percent of youth who are no longer in secondary school, had IEPs in effect at the time they left school, and were:
Enrolled in “higher education”
In “competitive employment”
Enrolled in “some other postsecondary education or training”
In “some other employment”

9. Competitive Employment Other Postsecondary Education or Training
PSO Definitions
full- or part-time
community college (2-year program)
college/university (4- or more year program)
one complete term
Higher Education
pay at or above the minimum wage
setting with others who are nondisabled
20 hours a week
for at least 90 days (includes military)
Competitive Employment
at least 1 complete term
education or training program (e.g., Job Corps, adult education, workforce development program, vocational technical school which is less than a 2-year program)
Other Postsecondary Education or Training
pay or been self-employed
at least 90 days.
This includes working in a family business (e.g., farm, store, fishing, ranching, catering services, etc.)
Other Employment

10. Indicator 14 for Federal Reporting
Click to edit Master text styles
States must report a percentage for each A, B, and C in the SPP/APR and to the public. States must also provide the actual numbers for each 1, 2, 3 and 4.
States are not required to report those former students who were “doing something other than” or “not engaged” in 1, 2, 3, or 4. However, those leavers will be part of the “total respondents”.

11. Findings from state data collection efforts are used to:
Nationally
Rhode Island
Report at the national, state, and local levels
Guide and improve transition programs for transition age youth with disabilities
Continue improvement in data collection focusing on improving representativeness
Develop capacity to improve outcomes for specific populations of youth who are not engaged at the same rate as others after high school (e.g., youth with ED, AA, Hispanic) . (FFY 2010 SPP)

12. Collecting Post-school Outcomes Data
Who are the data collected on?
What data are collected?
How are the data collected?
When are the data collected?
Who collects the data?

13. Who are data collected on?
Nationally
Rhode Island
Those with IEPs who leave high school:
With a diploma – regular or modified
With a certificate
By aging out
By leaving early /dropping out
Youth with IEPs who leave high school by:
Graduating with a regular diploma
Age out
Left school early (i.e., dropped out)
Akzidenz-Grotesk BQ Reg
Challenge: Finding leavers one year out of school

14. What data are collected?
Nationally
Rhode Island
In school:
Demographic data
Disability
Gender
Race/ethnicity
Age
Method of exit
One year out:
Higher education
Competitive employment
Other postsecondary education or training
Other employment
In school:
Demographic data
Disability
Gender
Race/ethnicity
Age
Method of exit
Student Contacts
One year out:
Higher education
Competitive employment
Other postsecondary education or training
Other employment
Akzidenz-Grotesk BQ Reg

15. How are data collected nationally?
Census v. Sample
Method of Collecting
Every district over the life of the SPP
ADM > 50K
Representative sample Disability
Gender
Race/ethnicity
Age
Survey
Phone
Mailed
Face-to-face
Web- or Internet-based
Extant database
Akzidenz-Grotesk BQ Reg
Challenge: Finding leavers one year out of school

16. How are data collected in Rhode Island?
Census
Method of Collecting
Attempt to contact every leaver in every district over the life of the SPP
Use unique identifiers
Survey
Combination of Phone and
Online survey (district personnel complete)
Akzidenz-Grotesk BQ Reg
Challenge: Finding leavers one year out of school

17. When are data collected?
Nationally
Rhode Island
April through September
When youth have been out of school for at least one year
Same
Akzidenz-Grotesk BQ Reg
Challenge: Finding leavers one year out of school

18. Who collects these data?
Nationally
Rhode Island
SEA or LEA staff
Teachers and support staff
Administrators
Outside contractor
University
Survey Center
Student’s last known case manager (certified special education teacher)
LEA Census Clerks
Special Education Administrators
(Regional training provided via WebEx)
Akzidenz-Grotesk BQ Reg
Challenge: Finding leavers one year out of school

19. Results of State’s Efforts: Outcomes for Students with Disabilities as measured by Indicator 14
United States Department of Education, Office of Special Education Programs (2011).
Part B State Performance Plan/ Annual Performance Report 2011 Indicator Analyses.

20. Rhode Island’s Post-school Outcomes Data

21. Rhode Island’s PSO Data These data represent youth who left school during the 2009-2010 school year.
Of the 1579 leavers 1101 (70%) responded to the PSO survey.
The responders were found to represent youth across the state based on disability category, gender, and minority status.
Dropouts were slightly underrepresented.
Of those who responded, 62% reported being enrolled in higher education, competitively employed, enrolled in some other postsecondary education or training program, or other employment since leaving high school.

22. How well do those who responded represent all leavers in 2009-2010?
Overall Response Rate: 70%
Is our sample size large enough to draw meaningful conclusions?
States are asked to describe in the SPP how well their data obtained from those who respond to the survey represent the total leavers in the state. For RI in , of the 1579 total leavers (census), 1101 responded to the survey, for a 70% response rate.
Remember, the question we want to answer is how well do those who responded represent all leavers in ? To answer this question we look beyond the response rate and compare the proportion of respondents to leavers on key characteristics: disability, gender, minority status, and method of exit. A difference of +-3% in the proportions indicates an importance difference between those who responded and the total leavers.
One category exceeds that 3% difference in proportions: youth of who dropout.
Differences greater than ± 3% are important

23. Challenges Finding leavers 1-year out of school
Contacting leavers 1-year out of school

24. Rhode Island’s Method of Exit Of the 1100 youth who responded across the state…
Facilitator's Notes:
(Purpose #2)
This chart shows what percentage of state youth who responded to the survey exited school by graduating, dropping out, aging out or other means of exit.
If numbers do not total 100%, you should use a chart with columns (i.e., bars), not a pie chart.
In most cases, the number of youth who graduate and dropout will not total 100% of the youth who left school in a year for several reasons:
Differences in the definitions
Differences in the data used to calculate graduation and dropout rates
Some youth die, or move out of State, and may not be categorized as either.
To reach 100% on the pie chart, you may need to include categories of other methods of exit; for example, youth who reach maximum age or age out, or Other, as defined by the State or district.
Numbers in the chart are place holders.
Change the numbers in the graph by doing the following:
-Double click on a slice of the pie graph to open the Excel file
-When the Excel spreadsheet opens:
-Click on the tab labeled “Sheet 1”
-Change the numbers you want to change
-Click on the tab labeled Pie Chart
-Close the spreadsheet by clicking outside of the Excel box
Data Source: RI PSO SY

25. Rhode Island’s Engagement Rate: Of the 1100 respondents
Convert percents to numbers
Change data
RI PSO SY

26. Let’s look at examples of how data can be disaggregated.
So What??
You are probably asking:
What do these data tell me?
What use are these data?
How can these data be used to improve programs for youth with disabilities?
Engagement rate alone is insufficient to inform program decisions.
The engagement rate should be disaggregated by subgroups of students based on key characteristics, such as demographic data, geographic location, or other relevant information.
Let’s look at examples of how data can be disaggregated.

27. Are Males & Females engaged at the same or similar rate?
Brainstorm questions to answer about the engagement rates of males and females. Do we do better with certain disability groups or genders?
Facilitator's Notes:
(Purpose #3)
Ask the stakeholder group to brainstorm the questions they would like to answer about the engagement rates of males and females.
You may ask people to write questions on sticky notes and read them aloud or display them on chart paper.
Place the group’s questions on chart paper. Using the data, answer the questions.
Examples of questions:
Are males and females competitively employed at approximately the same rate? If not, which group is employed at a higher rate?
What do you think contributes to the higher employment rate for some groups?
What could the district do to promote competitive employment for groups with a lower employment rate?

28. Percent of Males & Females Engaged
Engagement Rate
Facilitator's Notes:
(Purpose #3)
Take a couple of minutes to make observations about what’s displayed.
Record observations on chart paper.
Of the questions the group wants to answer, do the data answer any of the group’s questions?
Numbers in the chart are place holders.
Change the numbers in the graph by doing the following:
Right click on a bar in the graph
Select “edit data”
When the Excel spreadsheet opens:
Change the numbers you want to change
Push the return key on the keyboard
Close the spreadsheet 28
Data Source: RI PSO SY 28

29. Are youth with various disabilities engaged at the same or similar rate?
Brainstorm questions to answer about the engagement rates of youth with different disabilities. Do we do better with certain disability groups or genders?
Facilitator's Notes:
(Purpose #3)
Ask the stakeholder group to brainstorm the questions they would like to answer about the post-school outcomes for youth of different disability categories.
You may ask people to write questions on sticky notes and read them aloud or display them on chart paper.
Place the group’s questions on chart paper.
Using the data, answer the questions.

30. Engagement Rate by Disability Categories
AO = All Other Disabilities
Engagement Rate
Facilitator's Notes:
(Purpose #3)
Take a couple of minutes to make observations about what’s displayed.
Record observations on chart paper.
Of the questions the group wants to answer, do the data answer any of the group’s questions?
Numbers in the chart are place holders.
Change the numbers in the graph by doing the following:
Right click on a bar in the graph
Select “edit data”
When the Excel spreadsheet opens:
Change the numbers you want to change
Push the return key on the keyboard
Close the spreadsheet 30
Data Source: RI PSO SY 30

31. Are youth from various race or ethnic groups engaged at the same or similar rate?
Brainstorm questions to answer about the engagement rates of youth from different backgrounds. Do we do better with certain groups of youth?
Facilitator's Notes:
(Purpose #3)
Ask the stakeholder group to brainstorm the questions they would like to answer about the post-school outcomes for youth of different disability categories.
You may ask people to write questions on sticky notes and read them aloud or display them on chart paper.
Place the group’s questions on chart paper.
Using the data, answer the questions.

32. Engagement Rate by Ethnicity Categories
AO = All Other Disabilities
Engagement Rate
Facilitator's Notes:
(Purpose #3)
Take a couple of minutes to make observations about what’s displayed.
Record observations on chart paper.
Of the questions the group wants to answer, do the data answer any of the group’s questions?
Numbers in the chart are place holders.
Change the numbers in the graph by doing the following:
Right click on a bar in the graph
Select “edit data”
When the Excel spreadsheet opens:
Change the numbers you want to change
Push the return key on the keyboard
Close the spreadsheet 32
Data Source: RI PSO SY 32

33. How can disaggregated PSO data be used?
What are other states doing?

34. How other states are using PSO data to inform transition programs
NC revised their PSO survey to collect more programmatic data to examine whether in-school programs and services correlate with better post- school outcomes.
MD is sharing PSO data with the state Interagency Council to inform and improve the cross-agency collaborative services provided while youth are in-school.

35. How other states are using PSO data to inform transition programs
SD compared outcomes of students who participated in certain transition programs to those who did not to determine program effectiveness.

36. Let’s Look Closer: South Dakota
Question: Are youth who participate in 4 state specific in- school programs more likely to be engaged one-year out of school than youth who do not participate in these programs?
Programs: Catch The Wave, Youth Leadership Forum, Self-Advocacy, & Project Skills
Used an odds ratio to measure effect size, describing the strength of association between two dichotomous values: youth engaged (i.e., working or enrolled) versus youth not engaged (i.e. not working or enrolled).

37. OVERALL for 08-09: Odds Ratio of engagement for students who participated in Catch The Wave, Youth Leadership Forum, Self-Advocacy, or Project Skills
# engaged
# not engaged
Participate in CTW, YLF, Self-Advocacy, or Project Skills
177 19
Did not participate in CTW, YLF, Self-Advocacy, or Project Skills
120 26
177/19
120/26 = 9.316/4.615 OR of 2.03
n= 342
Those who participated in these programs were 2.03 times more likely to be engaged in work or school than those who did not participate in these programs.
177/19
120/26 = 9.316/4.615 OR of 2.03

38. Empirical Evidence
South Dakota has empirical evidence that youth who participated in Catch The Wave, Youth Leadership Forum, Self-Advocacy, or Project Skills were consistently more likely to be positively engaged 1-year out of high school than youth who did not participate in these programs.
With sufficient data, you could do something similar to examine whether the programs offered in RI or your district contribute to positive post-school outcomes.

39. The Logic of Using PSO Data for Program Improvement
Data Collection
Positive Post-school Outcomes
Data Collection
Data Use
Positive Post-school outcomes

40. Thinking about Transition Related Program Improvement
In what areas are youth with disabilities doing well?
What areas need improvement?
What is the state/district doing well?
What does the state/district need to improve?
Facilitator's Notes:
(Purpose #4)
Place each question on individual sheets of chart paper. Ask the group to answer each question.
Depending on group size, you may want them answer on sticky notes, to allow individual think time, or go one by one around the group.
The goal with this slide is to start to look critically at the district or school program and identify how the program contributes to positive post-school outcomes and how it does not.
If you have several areas that need improvement, you may need to prioritize the needs. There are several criteria that can be used to prioritize needs. Examples of criteria include:
Resource investment (people, money, or equipment/materials): no additional resources, few resources, or significant resources
time investment: changes can be addressed immediately, in the short-term (defined as ….), or in the long-term (defined as …)
Degree of change anticipated
Number of youth/families affected

41. How do you make the connection between post-school outcomes of youth with disabilities and practices implemented in the district, school, or classroom?

42. Evidence-Based (Special)Education…
…is a decision-making process for deciding what to teach based on:
frameworks or principles
best available research evidence
professional judgment
student needs and values
A third response has been to look at EBP as a more generic, decision-making process…two examples…
1. See TEC, Jan/Feb, 2010, pages 42-53, “Knowledge-to-Action Guides”
(Buysse, Wesley, Snyder, & Winton, 2006; Cook, 2010 Detrich, Spencer, & Slocum, 2009;Turnbull et al., 2010; Cook, 2010)

43. Create Individualized Interventions Adapt Interventions
Frameworks Provide a Systematic Basis for using Professional Judgment to:
Select Interventions
Create Individualized Interventions
Adapt Interventions
(Spencer, 2009)

44. Student-Focused Planning Student Development Family Involvement
A Framework for Secondary Transition: The Taxonomy for Transition Programming
Student-Focused Planning
Student Development
Family Involvement
Interagency Collaboration
Program Structure
Provides a framework to guide secondary transition instructional planning and practices
(Kohler, 1996)

45. Need to Consider Inputs Outputs Outcomes OAVSNP, 2010 Keynote
Deanne Unruh, Ph.D.,

46. Inputs Outputs Outcomes
OAVSNP, 2010 Keynote
What resources do we have in place or need?
Inputs
What strategies/ evidence-based practices will we use?
Outputs
What student outcomes do we need to improve?
Outcomes
Deanne Unruh, Ph.D.,

47. Focus on What you Can Control
Post-School Outcomes
Predictors
Practices
Lesson Plans

48. PREDICTORS Post-School Outcomes Independent Living Employment
Education
Independent Living

49. In-School Predictors by Post-School Outcome Area
Predictors/Outcomes
Education
Employment
Independent Living
Academic/General Education x
Career Awareness
Community Experiences
Exit Exam Requirements/High School Diploma Status
Interagency Collaboration*
Occupational Courses
Paid Work Experience*
Parental Involvement
School Integration*

50. In-School Predictors by Post-School Outcome Area
Predictors/Outcomes
Education
Employment
Independent Living
Self-Advocacy/Self-Determination* x
Self-Care/Independent Living
Social Skills
Student Support*
Transition Program*
Vocational Education*
Work Study*

51. Linking Post-School Outcome Data to Evidence-based Practices and Predictors
How do I do this?

52. How do Predictors Apply to Transition Planning and Instruction?
OAVSNP, 2010 Keynote
How do Predictors Apply to Transition Planning and Instruction?
Provides practitioners information about secondary transition program characteristics that have been empirically linked to improved post- school success for students with disabilities
Can be used to:
develop and expand programs
evaluate existing programs
Can help IEP teams design annual IEP goals and transition services that are more likely to help students achieve their stated post-school goals
Deanne Unruh, Ph.D.,

53. Predictors of Post-School Success for Students with Disabilities
Youth Transition Program (YTP)
Transition planning focused on post-school goals
Instruction in academic, vocational, independent living, and personal social areas
Paid job training while in the program
Assistance to secure employment or enter postsecondary education upon leaving the program
Follow up support for up to 2 years after leaving the program
Transition to independence Process System (TIP)
Relies on a transition facilitator/specialist
Youth-centered and focused on youth engagement
Transition plans developed that access various service agencies based on unique needs of individual

54. Sample State’s Engagement Rates Of the 400 youth who responded to the interview/survey across the state…
Facilitator's Notes:
(Purpose #2)
This chart shows what percentage of state youth who responded to the survey were in each engagement category (i.e., higher education, competitive employment, postsecondary education or training, and other employment).
If numbers do not total 100%, you should use a chart with columns (i.e., bars), not a pie chart.
In most cases, the categories of higher education, competitive employment, other employment, and postsecondary education or training will not total 100% of youth who left school. To reach 100%, you may need to include an additional category representing the remaining percentage of youth surveyed. In this example, we label the category not engaged to represent those youth who are not counted in one of the other categories. The state may prefer to use labels such as Other or Neither. If these categories are used in the State, define who is represented in these categories.
For example, Other may include youth who are incarcerated. Neither may include youth who are not engaged in any positive post-school outcome.
Numbers in the chart are place holders.
Change the numbers in the graph by doing the following:
-Double click on a slice of the pie graph to open the Excel file
-When the Excel spreadsheet opens:
-Click on the tab labeled “Sheet 1”
-Change the numbers you want to change
-Click on the tab labeled Pie Chart
-Close the spreadsheet by clicking outside of the Excel box
Data Source: Sample PSO

55. TIP Youth Transition Program Interagency Collaboration
Increased Competitive Employment
TIP
Youth Transition Program
Interagency Collaboration

56. Rhode Island’s Engagement Rate: Of the 1100 respondents
Convert percents to numbers
Change data
RI PSO SY

57. What is Rhode Island doing that could lead to higher engagement after high school?
Community-based Training (e.g., The Sports Exchange)
Post-School Engagement
Rhode Island Transition Academies
Career Development Project
Rhode Island Guides for Students in Transition
Rhode Island “My Transition” website
Check out the new Rhode Island web site for students
Being informed is the best way to advocate for the services and programs you may need to effectively transition from high school to employment, post-secondary education & training and community living. This page is divided into sections that may help you navigate the transition process and meet with success in the transition from school to adult life
Understanding the Transition Process
The Rhode Island Department of Education works closely with school districts, public adult services agencies, parent organizations, universities and colleges to provide current and useful information. One tool that many young adults have found useful is and on-line tool called Youthhood. At Youthhood you will find tools to assist you in planning for your life after high school. It is a great place to start.
The RI Transition Academies are a unique educational program that offers students with disabilities, 18 to 21 years old,  the opportunity to complete their high school education on a college campus and/or in various employment and community settings. It is geared toward the student who has completed or nearly completed the academic courses necessary for graduation and would benefit from an additional one or two years to acquire the functional life skills, vocational skills and social skills necessary to be better prepared for adult life. Potential student candidates must be in special education with an Individual Education Plan (IEP), be currently enrolled in school and near completion of their academic program. Candidates must also be eligible for services through the Office of Rehabilitation Services and must have a current application with the Office of Rehabilitation Services.
College Resources & planing: This directory helps students who receive supports through Individual Education
Programs (IEPs), or 504 Plans, to learn about post-secondary education opportunities. This guide will help to:
- Plan and prepare for post-secondary education
- Identify available supports to aid access to post-secondary education
- Find disability-related information about colleges in Rhode Island
Employment Resources and Planning: Designed for students in transition, this guide walks them through the steps of planning a career and conducting an employment search. The workbook format encourages students to enter their own information for future use. This guide complements School to Career training programs and curricula.
Living & particpating in your Community: this guide will help students plan and access the supports they need to live and participate in their communities.
Rhode Island Transition Council
Regional Transition Centers

58. Predictors Post-School Outcomes Evidence-based Practices Lesson Plans
“While the evidence-based practices were designed to teach students specific transition-related skills, to date, the experimental literature has not attempted to measure the impact of these skills on post-school outcomes.”
“Therefore, NSTTAC began examining correlational research on in-school predictors of post-school success.”
Lesson Plans

59. Have used rigorous research designs
Have demonstrated a record of success for improving student outcomes
Have undergone systematic review process using quality indicators to evaluate level of evidence
Evidence-Based Practices
Research-Based Practices
Developed based on research
Have demonstrated limited success for effectiveness
Have used a ‘weak’ research design
Promising Practices
Are not based on research
No data have been collected
Based on Anecdotal evidence and professional judgment exists
Unestablished Practices
Evidence-Based Practices
Have used rigorous research designs
Have demonstrated a record of success for improving student outcomes
Have undergone systematic review process using quality indicators to evaluate level of evidence
Research-Based Practices
Promising Practices
Developed based on research
Have demonstrated limited success for effectiveness
Have used a ‘weak’ research design
Unestablished Practices
Are not based on research
No data have been collected
Based on Anecdotal evidence and professional judgment exists
(Helsel, Hitchcock, Miller, Malinow, & Murray, 2006; Lembke & Stormont, 2005; Twyman & Sota, 2008)

60. Let’s examine a few of these practices
Using Self-Advocacy Strategy to teach Student involvement in the IEP
Self-Advocacy Strategy (SAS) consists of 5 steps taught over a series of seven acquisition and generalization stages:
I – Inventory
P – Provide
L – Listen and Respond
A – Ask Questions
N – Name your goals

61. Let’s examine a few of these practices
Using Whose Future is it Anyway? to increase self-determination skills and transition planning knowledge
Whose Future is it Anyway? is a student directed transition planning curriculum comprised of six sections:
Having self-awareness and disability awareness
Decision making about transition-related outcomes
Identifying and securing community resources to support transition services
Writing and evaluating goals and objectives
Communicating effectively in small groups
Developing skills to become and effective team member, leader, or self-advocate
Available: partnerships/zarrow/self-determination-education-materials/whos-future-is-it- anyway.html

62. How does EBP Apply to Transition Planning and Instruction?
OAVSNP, 2010 Keynote
How does EBP Apply to Transition Planning and Instruction?
Provides teachers information about secondary transition evidence-based practices for teaching students with disabilities
Can be used to:
Support IEP goals and objectives
Support skill development
Deanne Unruh, Ph.D.,

63. Resources Jane Slade Rhode Island Department of Education
National Post-School Outcomes Center
National Secondary Transition Technical Assistance Center
National Dropout Prevention Center for Students with Disabilities
Facilitator's Notes:
These are three federally funded technical assistance and dissemination centers charged with helping States and LEAs address post-school outcomes (NPSO), effective transition planning (NSTTAC), and graduation and dropout prevention (NDPC-SD).
Contact any of the centers for additional information and resources.

64. Dawn A. Rowe