Zarrow Center TAGG: A New On-Line Transition Assessment Jim Martin Dept. of Educational Psychology Zarrow Center University of Oklahoma

Zarrow Center Agenda 1.Steps to Prioritize Behaviors for Annual Transition Goals 2.TAGG Development 3.TAGG Overview 4.Validity Evidence 5.TAGG Demonstration 6.Follow-Up Study Details 7.Predictors of Post-School Education and Employment

Zarrow Center TAGG Developed with a Grant from the National Center for Special Education Research and OU Zarrow Center Funds THANKS TO NATIONAL CENTER FOR SPECIAL EDUCATION RESEARCH

Zarrow Center TAGG OVERVIEW

Zarrow Center TAGG Purpose Assess non-academic skills associated with and predictive of post-school further education and employment To provide IEP team student strengths, needs, a written summary, and annual transition goals matched to common core standards to facilitate writing I-13 compliant IEPs

Zarrow Center Who? TAGG Designed to Assess – Secondary-aged students with IEPs who plan to be competitively employed and/or enrolled in further education after graduation Each TAGG set includes 3 versions – Student – Family – Professional

Zarrow Center Versions and Format On-line written English – May be printed and taken by hand, but item scores must be entered into website to produce results – In next few months a TAGG versions in Spanish, will be added If needed now we have a paper version Users may choose to listen to audio or watch ASL videos for TAGG instructions and items

Zarrow Center Reading Level Professional 10.4 grade level Family 5.7 grade level Student 4.8 grade level

Zarrow Center TAGG Web-Generated Results Profile Graphic results by constructs Written summary Relative and greatest strengths Relative and greatest needs Annual transition goals Components may be copied and pasted into IEP

Zarrow Center TAGG DEVELOPMENT

Zarrow Center Standards for Educational and Psychological Assessment Guided TAGG Development

Zarrow Center Development of TAGG Items TAGG items derived from research studies that identified behaviors of former students with disabilities engaged in post-high school employment and/or further education The research team initially used the research studies to develop – 10 construct definitions – Items developed from constructs 15 iterative TAGG versions were created before field testing began

Zarrow Center Initial Structure: Ten Initial Constructs Knowledge of strengths and limitations Actions related to strengths and limitations Disability awareness Employment Goal setting and attainment Persistence Proactive involvement Self-advocacy Supports Utilization of resources

Zarrow Center Establish Initial TAGG Structure Users from six states completed the initial test-version TAGG – 349 high school students with disabilities – 271 family members – 39 professionals Applied various factor analyses statistics Went from 10 constructs to 8 Went from 75 items to 34

Zarrow Center After FA: Professional and Family TAGG Stayed 1.Strengths and Limitation 2.Disability Awareness 3.Persistence 4.Interacting with Others 5.Goal Setting and Attainment 6.Employment 7.Student Involvement in IEP 8.Support Community Dropped 1.Actions Related to Strengths and Limitation 2.Utilization of Resources TAGG-P: (    df=499, RMSEA=.058, CFI=.92, TLI=.91, RMSR=.0597) TAGG-F: (  2 =862.74, df=499, RMSEA=.057, CFI=.91, TLI=.90, RMSR=.058)

Zarrow Center Student Version Constructs After FA After FA Constructs 1.Strengths and Limitations & Support Community 2.Disability Awareness 3.Persistence 4.Student Involvement in IEP 5.Interacting with Others 6.Goal Setting and Attainment 7.Employment Dropped Constructs 1.Actions Related to Strengths and Limitation 2.Utilization of Resources Combined Constructs 1.Strengths and Limitations 2.Support Community TAGG-S: (  2 =819.00, df=505, RMSEA=.047, CFI=.89, TLI=.88, RMSR=.064)

Zarrow Center Replicate Factor Structure Two additional independent studies from TAGG users across the country Results confirmed factor structure

Zarrow Center Three Years of Data Collection 2,556 participants from 42 states and 162 schools – 1,291 secondary students with disabilities who had postsecondary further education and/or competitive employment goals – 172 professional completed TAGG on 7 to 8 of their students – 847 family members completed TAGG on their child

Zarrow Center IN BRIEF: VALIDITY EVIDENCE

Zarrow Center Internal Reliability Generally a score between.7 and.8 is considered “good” – Each TAGG version has great overall internal consistency and satisfactory subscale consistency (ranging from α =. 89 to α =. 95)

Zarrow Center Test-Retest Reliability Scores of.7 or higher represent good or satisfactory test-retest reliability – 14 weeks after the first TAGG was completed, same users completed the TAGG again. – A large correlation was found between the first and the second administration.80 for professional TAGG.70 for family TAGG.70 for student TAGG

Zarrow Center Fairness Validity Evidence: Gender Do differences exist by gender? – No overall difference by gender on TAGG-P, TAGG-S – On TAGG-F small overall differences – Some construct differences exist. On TAGG-S – females rated themselves higher on student involvement than males – Males rated higher on employment

Zarrow Center Fairness Validity Evidence: Disability Categories Construct Differences were as expected but small – On most constructs OHI and LD higher than Autism, ED, and ID Students with ED and Autis lower scores on interacting with others

Zarrow Center Measurement Invariance Structure held across all disability categories The three TAGG versions are appropriate for students regardless of: – Students’ time in general education classes – Number of transition education classes completed – High School grade level – Disability category

Zarrow Center Free/reduced lunch eligibility No significant differences for construct scores on TAGG- P or TAGG-S. Only small differences for TAGG-F scores. Family employment No significant differences for construct scores Family education Significant differences due to family education were found on the TAGG-F scores. Almost all of the differences occurred between students whose family member was at the highest level of education (Master’s/Ph.D./other professional degree) and those at the lowest level of education (less than HS). Fairness Validity Evidence: SES

Zarrow Center How Close Are Students, Professionals, and Family TAGG Scores? How closely do the different TAGG versions assess the same student? – Medium correlations across Parent, Educator, and Student versions when assessing the same student. – This is excellent for this type of assessment

Zarrow Center TAGG & AIR Self-Determination Assessment Same users completed TAGG and AIR Self- Determination Assessment – Medium Correlation This implies the TAGG addresses some self- determination skills and assesses other skills, too.

Zarrow Center Predictive Validity Process Follow-up of 297 former high school students who completed the TAGG while in high school Logistic regressions examined relations between TAGG non-academic behavior constructs and postsecondary education and employment

Zarrow Center Constructs Predicting Further Education Interacting with Others Student Involvement in the IEP Support Community Goal Setting and Attainment

Zarrow Center Constructs Predicting Employment Employment Student Involvement in IEP Support Community Interacting with Others

Zarrow Center Item Response Theory Advantages of IRT include – The ability to scale different item types – Provides a common metric for scales with different number of items – Weights items differentially by their validity for assessing the construct of interest

Zarrow Center We Used a Four-Step IRT Algorithm 1.Placed each scale onto a common score metric 2.Projected item characteristics (e.g. item difficulty) onto the scale score metric 3.Conducted a within-student comparison of scale scores across constructs to determine relative strengths and weaknesses 4.Conducted a within-construct comparison of a student’s scale score to item responses (e.g. difficulty) to generate appropriate goals for identified weaknesses

Zarrow Center Overall Score The overall score is a weighted combination of all items. The overall score is not an average of all the construct scores.

Zarrow Center TAGG DEMONSTRATION

Zarrow Center OBTAIN THE TAGG

Zarrow Center Minimal Cost – $3 per set (Professional, Student, Family versions) – Pay with Purchase Order or Credit Card – All funds handled by the OU Office of Technology Development – Used to pay for on-going TAGG development and operational costs TAGG profiles saved for 7 years Data kept on OU high-speed secure cloud servers Purchased credits may be transferred to other registered TAGG users Unused credits refunded for one-year after purchase

Zarrow Center TAGG Web Site Location 1.The OU Zarrow Center’s Web Page 2.The TAGG Section of the ZC Web Page

Zarrow Center The On-Line TAGG and Result Profile SAMPLE TAGG SCREEN SHOTS

Zarrow Center

Disability Awareness Profile

Zarrow Center Combined Score Profile

Zarrow Center Greatest and Relative Strengths

Zarrow Center Areas of Greatest and Relative Need

Zarrow Center Summary Statement for IEP Chad Bailey’s skills were assessed using the TAGG, a norm-referenced assessment with research-based items known to be associated with post-school employment and education. Compared to similar students, Chad’s scores are average. Results indicate greatest strengths are in the areas of Goal Setting and Attainment. Chad’s relative strengths include Disability Awareness and Student Involvement in the IEP. Greatest needs are in the area of Strengths and Limitations, with Employment being a relative need.

Zarrow Center Suggested Annual Transition Goals To prepare for success in employment, the student will write an essay describing three situations where the student used his or her strengths with 90% grammar and context accuracy by the end of the essay writing unit.

Zarrow Center Contact Information Professor Jim Martin Phone: (405) Website: