Steven W. Evans, Christine Brady, Lee Kern, Christiana Andrews and the CARS Research Team Measurement Development and Inclusion Criteria: Developing Meaningful Standards

Defining the Population No consistent standard that is applied across districts that would allow us to rely on school district determined labels. ED OHI (ADHD) LD Other Criteria High school students with social, emotional behavior problems Cognitive ability in the normal range Significant impairment in school functioning Placement in special education

Key Aspects of Definition Social, emotional or behavior problems Broad band measure At risk range Significant impairment in school functioning Limited assessment tools for use with high school aged students Academic Competence Evaluation Scales (ACES): 73 items with norms for adolescents Classroom Performance Scale: 24 items with no norms or factors Placement in special education Cognitive ability in the normal range FSIQ estimate equal to or greater than 75 according to most recent evaluation in school records

Research Plan Collect CPS and BASC data for students in the pilot study with open eligibility criteria Compare data to BASC norms Gather normative data with the CPS Identify factors (if any) Establish norms Compare data to findings from CPS Identify items/factors that assess most problematic areas and issues for which we have interventions Determine eligibility criteria

BASC Means & SD for CARS Participants MeanSD BASC External BASC Internal

BASC Externalizing

BASC Internalizing

Number Who Met BASC Criteria

Classroom Impairment: Method CARS investigators were asked to recruit staff at high schools for the study Measurement packets were created for each teacher and placed in their mailbox Packets contained a cover letter, student selection form, teacher demographic sheet, and six copies of the classroom performance survey (CPS), disruptive behaviors disorders scale (DBD), and impairment rating scale (IRS) Teachers were asked to complete the CPS, DBD, and IRS for three randomly selected boys and girls from their first period classroom using the student selection form

Method Those who completed packets were entered in a drawing to win one of three monetary prizes ($75, $50, $25) Drawing were conducted within school to increase participation Teachers were allowed two weeks to complete and return the packets

Classroom Performance Survey (CH.A.D.D., 1996) Teacher report 20 items on a 5 point Likert-type scale (1 = “always” to 5 = “never”) Items regarding homework, note-taking, test-taking, interpersonal skills, communication, and attention behaviors AlwaysSometimesNever Brings necessary items to class.12345

Teacher Demographics Overall Sample 139 teachers Race Caucasian83.60% African American3.70% Asian10.40% Other2.20% Gender Male29.10% Female70.90% Subject Language Arts24.60% Math26.60% Social Studies19.40% Science18.70% Other10.70% Type of Teacher General Ed86.80% Special Ed13.20% # of Years Teaching11.53

Student Demographics Overall Sample Grade 9th33.70% 10th27.90% 11th19.40% 12th18.90% Sex Male46.80% Female53.20% Race Caucasian39.90% African American45.30% Hispanic11.40% Asian0.70% Other2.70%

Analysis Plan Given our sample size (139 teachers and 833 students), we decided to run an exploratory factor analysis with a portion of the sample (263 students) and a confirmatory factor analysis with the others (570 students) Data were randomly assigned to an analysis by teacher within school

Exploratory Factor Analysis Results indicated the presence of two factors accounting for 78.6% of the variance Factor 1: Academic (14 items) included items regarding homework, tests, attending to instruction, and completion of longer term assignments Factor 2: Interpersonal (6 items) included items regarding relation to peer, relation to teachers, accepting assistance, and respecting property

Confirmatory Factor Analysis Two models were compared: the two factor model found in the EFA and a one factor model Results indicated some degree of fit, but not enough to justify the use of either model Items on the CPS should be treated individually

Mean Data on the CPS Scores on every item included the full range of scores (1 to 5) The two most commonly endorsed problematic behaviors were recording assignments (M = 2.68, SD = 1.36) and completing homework assignments (M = 2.65, SD = 1.26) The two least endorsed items were respecting property (M = 1.69, SD = 1.02) and relating positively to teachers (M = 1.84, SD = 1.06)

CPS Correlations CPS1CPS2CPS3CPS4CPS5CPS6CPS7CPS8CPS9CPS10CPS11CPS12CPS13CPS14CPS15CPS16CPS17CPS18CPS19CPS20 DBD-IA DBD-HI DBD-ODD IRS1 (Peers) IRS2 (Teacher) IRS3 (Academics) IRS4 (Classroom) IRS5 (Overall) All correlations were significant at p =.001

CPS Correlations CPS1CPS2CPS3CPS4CPS5CPS6CPS7CPS8CPS9CPS10CPS11CPS12CPS13CPS14CPS15CPS16CPS17CPS18CPS19CPS20 DBD-IA DBD-HI DBD-ODD IRS1 (Peers) IRS2 (Teacher) IRS3 (Academics) IRS4 (Classroom) IRS5 (Overall) All correlations were significant at p =.001

CPS Descriptives ItemNormativeCARS MeanSDMeanSDAverage Z-ScoreT-testSignificance Brings necessary items to class Completes class assignments Completes homework on time Records assignments consistently Turns in completed work Completes long-term assignments * Attends to instructions in class * Arrives to class on time * Cooperates/Participates in class Demonstrates skills in reading assigned tests and materials Demonstrates adequate spelling and writing skills in work Takes notes in class to study Performs satisfactorily on tests Completes assigned work with accurate computation/detail Completes assignments legibly Relates positively to teacher(s) Demonstrates respect for property Relates positively to peers * Communicates own needs or asks questions Accepts assistance when needed or offered *

CPS Descriptives ItemNormativeCARS MeanSDMeanSDAverage Z-ScoreT-testSignificance Brings necessary items to class Completes class assignments Completes homework on time Records assignments consistently Turns in completed work Completes long-term assignments * Attends to instructions in class * Arrives to class on time * Cooperates/Participates in class Demonstrates skills in reading assigned tests and materials Demonstrates adequate spelling and writing skills in work Takes notes in class to study Performs satisfactorily on tests Completes assigned work with accurate computation/detail Completes assignments legibly Relates positively to teacher(s) Demonstrates respect for property Relates positively to peers * Communicates own needs or asks questions Accepts assistance when needed or offered *

Average Z-Score by Item

4 - Records Assignments Consistently

6 – Completes Long-Term Assignments

7 – Attends to Instructions in Class

12 – Takes Notes in Class to Study

14 – Completes Assigned Work with Accurate Computation/Detail

18 – Relates Positively to Peers

19 – Communicates Own Needs or Asks Questions

20 – Accepts Assistance when Needed or Offered

How are Students Distributed?

Number of Times Teacher Endorsed a z-score of 0.5 or Greater on Any Item

Establishing Initial Criterion Who are students with means better than normative sample? Who are students with very few items rated half of a sd worse than normative sample? Talk to CARS clinicians about students. Criterion – 2 or fewer items rated less than.5 z-score Best match with clinicians descriptions Possible to have better than average mean with some areas of significant impairment

Criteria Applied to Students Participating in Pilot We have parent BASC and CPS data for 32 participating students at two sites Met neither criteria – 2 Met one criteria – 19 Met both criteria - 11

Implications Possible reasons some students did not meet academic impairment criteria In a setting where nothing is being expected of them Special education teachers tend to rate problems as less severe than regular education teachers Other consideration Many of the most problematic students are not attending public high schools Modifications Consider BASC reports from both parent and special education teacher (use “or” criteria) Gather CPS data from all teachers

Criteria Social, emotional or behavior problems 1 sd from mean on either internalizing or externalizing Significant impairment in school functioning One-half sd from mean on 3 or more items on the CPS Cognitive ability within average range (broadly defined) IQ > 75 In special education In special education for something other than a Pervasive Developmental Disorder

Next Steps Keep eligibility criteria during next phase of pilot broadly defined (6 sites; 8 schools) In special education for social, emotional or behavior problems Exclude Pervasive Developmental Disorders and IQ less than 75 Adjust data collection and use “or” criterion BASC – gather from parent and primary special education teacher CPS – gather from all teachers Analyze eligibility criteria in relation to students’ responses to interventions and other evaluation data