1. When You Wish You’d Done a Pretest: Comparing Pre/Post and Retrospective Pre/Post Surveys of Knowledge and Skills
Catherine Callow-Heusser, Ph.D.
Audrey Hoffmann, M.S., BCBA
EndVision Research and Evaluation

2. Think-Write-Share
What have you done in the past when you haven’t collected baseline data and you wish you had?

3. Transition Specialist Master’s Degree Program
The Utah State University Transition Specialist Program (TSP) is funded by the U.S. Department of Education - Office of Special Education Programs, Grant # K.
Project staff synthesized relevant research to create a list of competencies deemed essential for teachers exiting the TSP.
The competencies were organized into six domains and included evidence-based transition practices that met current needs in the field of transition.
The list of competencies helped ensure that graduates of the TSP would be competent in the relevant and essential areas for a transition specialization.

4. Pre/Post Survey of Transition Competence
Twenty TSP students completed a pre/post survey based on the transition competencies. They were asked to rate on a 5-point scale from novice to expert their knowledge and skills based on transition competencies.
Nine Cohort I students completed the survey upon graduation, including a retrospective report of their initial knowledge and skills.
Eleven Cohort II students completed the survey immediately prior to beginning coursework and again upon graduation from the TSP.

5. Timelines Cohort 1 Starts, Fall 2011
Cohort 1 Graduates, Spring/Summer Cohort 2 Starts, Fall 2013
Cohort 2 Graduates, Spring/Summer Cohort 3 Starts, Fall 2015
Transition Competencies Developed
Transition Competencies Survey Administered
Summer 2013 Cohort 1: Retrospective Pre, Post
Cohort 2: Pre
Spring 2015
Cohort 2: Post

6. Transition Competency Survey
Assessment and Evaluation
Level of Competence
Novice Expert
Competency
Operational Definition/Examples
⚪️ 🔘 ⚪ ⚪ ⚪
5. Matching/aligning strengths, preferences, and interests to skills and/or demands required by post secondary, employment, residential, and other community-based settings
Matching the outcomes from assessments to jobs and community-based opportunities, e.g., preference assessment results indicate a young adult prefers working inside and with children, and the young adult’s skills include talking with others, reading, providing instruction in a small group, so possible employment could be working in a day care or child care setting; results from assessments indicate a young adult would like to receive post- secondary training related to welding and the young adult possess the necessary skills to enroll in a post- secondary setting, so a possible option would be an applied technology college in a welding program.
🔘 ⚪️ ⚪ ⚪ ⚪
6. Interpreting results of transition assessments for students, families, and other professionals
Interpreting and explaining assessment results, answering questions related to assessment results, using language that is understandable when talking about assessment results, and ensuring all members of the transition planning team, including parents, understand results and potential implications.

7. Means and Standard DeviationsSD
Cohort 1 Pre
2.23
0.40
Cohort 2 Pre
2.35
0.47
Cohort 1 Post
4.11
0.31
Cohort 2 Post
4.40
0.24

8. Correlations Item Averages (n = 67) Correlation
Cohorts 1 and 2 Pretest Correlation
0.79*
Cohorts 1 and 2 Posttest Correlation
0.59*
Cohort 1 Pre-Post Correlation
0.63*
Cohort 2 Pre-Post Correlation
0.45*
* p < 0.01

9. Cohort Comparisons
The two cohorts were very similar at the beginning the program…
And t-tests showed the groups were not statistically significantly different at pretest (p > 0.05).

10. Cohort Comparisons
The two cohorts were very similar at the end of the program…
while the t-tests showed the groups were statistically significantly different at posttest (p < 0.01), the range of average values for the items is quite restricted.

11. Findings Statistically significant differences between
Pre/post ratings
Cohorts 1 and 2 post ratings
of transition competence

12. Assessment and Evaluation
Applying results of student assessments to transition plans
Using a variety of formal and informal post secondary, career, vocational, and transition assessment methods to identify transition needs
Assessing young adults’ interests/preferences related to post secondary goals and educational and vocational experiences
Selecting assessments with consideration for cultural and linguistic diversity and family values
Matching/aligning strengths, preferences, and interests to skills and/or demands required by post secondary, employment, residential, and other community-based settings
Interpreting results of transition assessments for students, families, and other professionals
Coordinating/facilitating assistive technology needs assessments
Using student data to evaluate effectiveness of transition teaching
Observing/assessing post secondary, vocational or employment settings for potential placement opportunities
Assessing natural support systems (i.e., available in existing environment) in a variety of settings
Implementing strategies for shared use of student assessment data across different agencies
Assessing young adults’ work- related skills in the actual training environment or workplace

13. Transition Planning
Including appropriate team members in the transition planning process
Using transition planning and communication strategies to facilitate input from team members
Coordinating and facilitating IEP meetings/transition planning meetings
Developing the young adult’s skills to participate to the level of their ability in the development of their IEP
Developing transition goals and objectives based on the young adult’s interests and preferences
Developing transition IEP goals and objectives based on strengths and needs identified from career, vocational, and transition assessments
Developing transition-related goals and objectives that focus on post- secondary education or training, integrated employment, continuing and adult education, independent living, and/or community participation
Including transition services and courses of study in the IEP that will reasonably enable the young adult to meet his or her post secondary goals
Aligning IEP goals and objectives with state and local core standards
Aligning post secondary goals with instructional activities
Including assistive technology in the IEP as needed
Identifying and addressing young adults’ non academic personal issues that impact transition
Providing information about school, post secondary, and community services/agencies available to specific populations of individuals with exceptional needs
Facilitating referrals related to school, post secondary, and community services/agencies
Knowing about and applying Department of Labor regulations

14. Instructional Planning
Knowing and applying models or philosophies of transition programs and practices
Knowing how the history of national transition initiatives affects current directions
Selecting and/or developing appropriate job placements
Selecting and/or developing appropriate community-based educational programs
Developing natural support systems that facilitate transition to post secondary environments
Identifying supports needed for young adults to be successful in work experiences and training opportunities
Facilitating appropriate accommodations/adaptations across a variety of settings
Planning/developing educational experiences that correspond with IEP post secondary goals and objectives

15. Curriculum and Instruction
Being familiar with and evaluating transition education methods and curricula
Implementing evidence based transition practices
Coordinating instructional activities related to young adult’s post secondary goals
Providing community-based instruction
Teaching self-advocacy and self- determination skills
Teaching appropriate job readiness skills based on developmental needs and functional limitations
Teaching daily living skills
Teaching vocational/work-related skills
Teaching social skills
Teaching learning strategies and study skills
Providing experiences and teaching leisure and recreational skills
Implementing behavior management strategies in academic, community and/or work environments
Teaching young adults how to access and use assistive technology in academic, work, and community environments if needed
Teaching job retention skills identified by employers as essential for successful employment
Providing experiences to increase awareness and knowledge of careers
Ensuring young adults with disabilities have the same career and vocational development opportunities as their peers without disabilities

16. Communication and Collaboration
Providing transition-related resources and materials to students, parents, educators, service providers, and employers
Facilitating and/or providing training on transition-related topics
Monitoring participation in transition planning and implementation
Knowing and implementing procedures and requirements for referring young adults to community service agencies and other post-secondary organizations
Resolving differences that may arise in collaborative relationships and interagency agreements
Training, managing, coordinating and evaluating job coaches and school-based paraprofessionals working with young adults in vocational and community settings
Collaborating with professionals (e.g., general educators, administrators, group home providers, employers, etc.) to improve transition outcomes
Facilitating partnerships among service providers within the school and community to maximize transition outcomes
Specifying roles and responsibilities of each team member and/or agency

17. Family Involvement
Ensuring transition processes and outcomes are consistent with families’ cultures, beliefs, practices, and values
Including and collaborating with parents and family members in planning, decision-making, and implementing transition services
Providing information to families about transition services, programs, and post- secondary options for their young adult
Identifying family service needs related to transition outcomes
Assisting families to network with extended family, neighbors, church, employers, etc., to promote transition outcomes
Developing relationships to ensure commitment of family members or other caretakers to assist in transition
Working with family systems and the dynamics of individual families and understanding how systems impact the transition process

18. Effect Size Differences
The Hedge’s g (IES Procedures and Standards Handbook, ver. 3.0, 2013) measure of effect size is calculated by dividing the difference in means by the pooled standard deviation.

19. Effect Size Differences
Hedge’s g
Cohorts 1 and 2 Pretest Comparison
0.27
Cohorts 1 and 2 Posttest Comparison
1.05*
Cohort 1 Pre-Post Comparison
5.29*
Cohort 2 Pre-Post Comparison
5.56*
* p < 0.01

20. Retrospective Surveys
In what cases can retrospective surveys of knowledge and skills work?
When will they NOT work?

21. Discussion & Questions
Cathy Callow-Heusser Secondary Math Specialist State Systemic Improvement Plan
Multi-Tiered System of Supports Project Utah State Office of Education
President EndVision Research and Evaluation, LLC