Presentation is loading. Please wait.

Presentation is loading. Please wait.

Dichotomizing Children’s Behavior Problems: Carving Nature at Its Joints Sandy Braver, Jenessa Shapiro & Amy Weimer SPRI and Quant Seminar Presentation.

Similar presentations


Presentation on theme: "Dichotomizing Children’s Behavior Problems: Carving Nature at Its Joints Sandy Braver, Jenessa Shapiro & Amy Weimer SPRI and Quant Seminar Presentation."— Presentation transcript:

1 Dichotomizing Children’s Behavior Problems: Carving Nature at Its Joints Sandy Braver, Jenessa Shapiro & Amy Weimer SPRI and Quant Seminar Presentation

2 2 Statisticians Have Strongly Argued Against Dichotomizing Inherently Continuous Variables (MacCallum et al., 2002) Very Common Practice Derives from a “Group” ANOVA Mentality Medical Model:Categories of Disease, Not Continua Can Handle Through Regression Instead, But Not All Researchers Know This A Number of Statistical and Inferential Problems Result

3 3 Statistical Problems “Dichotomization is rarely defensible and often will yield misleading results.” Lower Statistical Power Cutpoints Are Typically Arbitrary Other Cutpoints Might Not Replicate Result Cutpoint Itself (e.g. median) Won’t Replicate Forces one into nonsensical positions:  1 and 62 (both below cutpoint) are the SAME, but  62 and 63 (one below and one above cutpoint) are DIFFERENT

4 4 But Real World Often Demands Dichotomization Selection Problems Screening At Risk Insurance DSMIV (But see Fall 2005 Special Issue of Journal of Abnormal Psychology, which argues for replacing categorical approach of DSMIV with a DIMENSIONAL Approach in Forthcoming DSMV)

5 5 Dichotomization Natural and Appropriate for Some Medical Problems Cancer For Other Problematic Conditions, Dichotomization More Arbitrary High Blood Pressure Diabetes Depression and Most Mental Health Conditions Pregnancy Death

6 6 NIMH: “Congress is Interested in Us Preventing or Treating Serious Mental Health Problems” “I Need to Show Them We Can Prevent Schizophrenia” Our Primary Dependent Variable at PRC: Child Well-Being or Mental Health “Caseness” Desired By Policy-Makers; Detecting Children Who Are Seriously Mentally ill

7 7 Current Standard: Child Behavior Checklist (CBCL) Achenbach E.g., The 2001 Annual Report to Congress on the Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program says: “The CBCL (Achenbach, 1991a) has been identified as the most reliable and valid parent report measure currently available for assessing children's emotional and behavioral problems (Reitman, Hummel, Franz, & Gross, 1998).”

8 8 Characteristics of the CBCL 133 items: “Argues A Lot”; “Depressed, Withdrawn” not true=0, somewhat or sometimes true =1, very true or often true=2 Parent Report, Youth Self-Report, Teacher Report Total Problems Score; Broad Band (Internalizing, Externalizing); Narrow Band (e.g., Attention Problems/Hyperactivity, Oppositional Defiant, and Somatization) Raw Score, T-Score Clinical Cutoff: “Internalizing, Externalizing, and Total Problems scale T-scores are considered in the clinical range if they are above 63, while scores from 60 to 63 are borderline. Scores in the clinical range indicate a need for clinical care.” High Reliability (Alpha and Test-Retest) Excellent National Norms Lots of Validation Studies Thousands of studies use the CBCL and report the percent of their study group in the clinical (and borderline clinical) range, reifying the arbitrary clinical range cutoff

9 9 Alternatives to the CBCL Rutter DISC (Takes several hours to administer, many hours to train and certify testors, matches Psychiatrists’ DSM Diagnosis) Short Form of CBCL, the Behavior Problem Index (BPI) 32 items Not Copyrighted—Free We Use BPI

10 10 Validation Studies Need a Criterion CBCL’s Criterion is referred to as “Referral Status” That is, A Clinic Sample vs A “Matched” Non- Clinic Sample Were Assessed Lots of differences are found between the two “status groups” on various CBCL variables, which establishes validity of scale

11 11 Cutoff Determination ALSO requires a criterion variable Choice of Cutoff Value for Caseness Determination can and should use the same criterion variable as for Validity studies This is a very common problem in medical settings (e.g., high blood pressure, diabetes) and human resources settings (e.g., hire, no hire) a technology (ROC) for distinguishing “normal” from “case” has developed and received acceptance

12 12 Brief Primer on Cutoff Determination: ROC (Receiver Operating Characteristic, Signal Detection) Analysis Diagnosis -+ Tests’ Value Relative to Proposed Cutoff Below False Negative(FN) At or Above False Positive(PN)

13 13 ROC Analysis Uses Constructs of Sensitivity & Specificity SE: {Sensitivity}: Is the proposed cutoff value sensitive? Does it detect most of the Positive Cases? SP: {Specificity}: Is the proposed cutoff value specific to the positive cases? Does it correctly indicate the cases that are NOT POSITIVE For a good test, with a well selected cutoff, both should be very high.

14 14 Sensitivity & Specificity Example Diagnosis -+ Test’s Value Relative to Proposed Cutoff BelowA=500B=200A+B=700 At or Above C=100D=200C+D=300 A+C=600B+D=400T=1000 =A+B+C+D SE=Sensitivity=Prob of pos test given pos diag= SP=Specificity= Prob of neg test given neg diag= 1-Specificity= Prob of POS test given neg diag = Specificity Should be LOW

15 15 Ideal point Best Cutoff 1-SP too high SE too low ROC Graph (SPSS): Sensitivity & 1-Specificity Are Calculated Repeatedly, for Each Potential Cutoff Value A CONVINCING Cutoff Should Be Noticeably Better Than Its Neighbors Potential Cutoffs 15

16 16 A CONVINCING Cutoff Should Be Noticeably Better Than Its Neighbors Otherwise, Cutoff Is Arbitrary Flat ROC Curves Provide No Compelling Rationale For Choosing One Cutoff Value vs Another For This Important Real World Choice Depends on Emphasis on SE or SP

17 Alternatives to Sensitivity & Specificity (Helena Kraemer) Diagnosis -+ Test’s Value Relative to Proposed Cutoff BelowA=500B=200A+B=700 At or Above C=100D=200C+D=300 A+C=600B+D=400T=1000=A+B+C+D PVP=predictive value of a positive test =(D=200)/((C+D=300) =.67 PVN=predictive value of a negative test =(A=500)/((A+B=700)=.71 Quality PVP=κ(0,0)=(PVP-P)/(1-P)=(.67-.4)/(1-.4)=.44, weights avoiding False Positives P=Prevalence (of a Pos Diag) =(B+D=400)/(T=1000)=.40 Quality PVN=κ(1,0)=(PNP-(1-P))/P=(.71-.6)/.4=.29,weights avoiding False Negatives Cohen’s Kappa [κ(.5,0)]=.35, weights FN & FP equally Weighted Kappa [κ(r,0)] [e.g. κ(.8,0)]=.31, weights FN & FP relatively, by r Efficiency=EFF=Overall Prob of Correct Class=(A+D= =700)/(T=1000)=.70 PHI coefficient (Φ) r =W FN /(W FN +W FP ), W is weight; r of.8 means that FN are 4 times worse than FP 4/(4+1) 17

18 18

19 19 Example 1 r pbi Note that it’s>Φ

20 20

21 21 Example 2 r pbi Φ>r pbi

22 22 Example 3 r pbi Φsubstantially>r pbi

23 23 Example 4 r pbi Φsubstantially>r pbi

24 24 Only A Peaked ROC Curve Provides Strong Rationale for A Specific Value For Cutoff, Whatever the Index Chosen Chooses Same Cutoff By Any Criterion No Need to Defend Choice of SE & SP, vs PVP & PVN, vs Quality version, vs Kappas No Need to Defend Weights for FN vs FP in Kappas No Need to Know Prevalence A Peaked ROC curve will result only if the raw data has a “Joint”, Joint on all curves at same point Want to “carve” (choose as cutoff) right at Nature’s joint

25 25 Applying ROC Concepts to CBCL Needs a True Dichotomous Criterion i.e., the “Diagnosis” Achenbach Uses Referral Status, the Same Variable Used as Criterion in Validity Studies, As Recommended Referred vs Not Referred i.e., Clinic Cases vs Normal The cutoff of 63 (Clinical Range) was chosen by Achenbach in order to meet the following Criterion: Minimizing Error (=ERR=FN+FP=1-Efficiency) (i.e., Maximize EFF) Difficulties with this Approach:  Doesn’t Provide ROC Curves, so Can’t Determine Whether ROC is Peaked, Whether that Cutoff Is Noticeably Better Than Neighbors  Is Equal Weighting of FN & FP Appropriate For Caseness?

26 26 Our Biggest Issue with the CBCL: The Validity Criterion Chosen They Say It’s “Referred vs Non-Referred”, But It’s Really Not It’s Really Clinic vs Non-Clinic Not Known WHO Referred Or If “Referred” By Anyone At All This Leads to the Question of Which Kids Get to Clinic Empirical Research on Parents Taking Kids to Clinic Shows Many Factors Are Influential, Only One of Which is Kid’s Mental State (Lobitz and Johnson, 1975)  Parents’ Labeling  Parent’s Belief in Efficacy of Clinical Therapy  Insurance Issues  Parent’s (Mother’s) Own Clinical Levels

27 27 Their Method Prevents Estimation of Prevalence, Needed For Some Indexes What Is prevalence of serious mental illness in Adolescents? Major Review Article of 52 Studies (Roberts,1998) Lots Of Issues What’s Criterion, What’s Population? 3 to 54% Median: About 15% This “Feels About Right” to Diagnosticians Matches Prevalence of Other Mental Health Problems CBCL Clinical Range T Score of 63 = About 15%

28 28 Is There Another Easy To Use True Dichotomous Criterion? Our Idea Teacher Report of Referral Actual Referral, Not Whether Being Seen Matches Notions About NEED For Services, Rather Than Use Of Services Teachers Are More Neutral Than Parents Teachers Have Better Frame Of Reference For Disturbed Behavior

29 29 Alternative: Peer Report of Psychopathology Data Suggests Peers of Adolescents Are Excellent Judges of Pathology Sociometrics Ratings of Peers Predict Better Than Any Other Variable Later Problem Behavior Sociometrics Cumbersome to Acquire Teachers Good Detectors of Sociometric Data Teacher Report on Peers Proxy for Peer Report

30 30 PAYS Data Set About 400 families All with 7 th Graders Recruited from schools in AZ and Riverside, CA Will be 3 Waves Mother, Father, Child Report on BPI Internalizing and Externalizing 2 Teacher Reports on our adapted version of BPI

31 31 * p <.001 Correlations of Teacher 1, Teacher 2, Child, Mom and Dad on BPI Teacher 1.57*.33*.26*.32* 2. Teacher 2.39*.30*.39* 3. Child's CBCL.35*.28* 4. BPI (Mom).47* 5. BPI (Dad)

32 8 True dichotomies (yes/no): Teacher Battery Added Criterion Questions

33 33 Correlations of 8 dichotomous items; Percent of Teachers that said “Yes” for each item * p<.05

34 34 Correlations between individual dichotomies (N=700) with teacher BPI * p<.001

35 35 Teacher Thought About Contacting Parents

36 36 Teacher Actually Contacted Parents Better, but still no real joint

37 37 Combine Dichotomies: 1 if ANY, 0 if NONE

38 38 Thought Abouts

39 39 Actuals

40 40 Peers Nature at the Joint?

41 41 So, pick a score of 6 on BPI as Cutpoint with Peers as Criterion SP.91 1-SP.09 SE.70 PVP.50 PVN.96 EFF.89 ERR.11 κ(0,0).44 κ(.2,0).47 κ(.4,0).51 κ(.5,0)=Cohen's.52 κ(.6,0).54 κ(.8,0).59 κ(1,0).64 phi.53 But why not just ask teacher the dichotomous questions: Forget the BPI


Download ppt "Dichotomizing Children’s Behavior Problems: Carving Nature at Its Joints Sandy Braver, Jenessa Shapiro & Amy Weimer SPRI and Quant Seminar Presentation."

Similar presentations


Ads by Google