Presentation on theme: "The Survey of Well-being of Young Children (SWYC)"— Presentation transcript:
1 The Survey of Well-being of Young Children (SWYC) Ellen C. Perrin, M.D.R. Christopher Sheldrick, Ph.D.So I’m going to tell you today about an instrument for surveillance that we have been working on for the past several years.
2 We have no relevant financial relationships with the manufacturer(s) of any commercial product(s) and/or provider(s) of commercial services discussed in this CME activity. We do not intend to discuss an unapproved/investigative use of a commercial product/device in this presentation.
3 Historical perspective Rosie D. lawsuitCBHIAvailable InstrumentsLengthyProprietaryScoring complexFragmentedSo most of you remember the lawsuit brought by a few families of mentally ill young adults, attesting that the state had fallen down in its obligation to screen, detect, and treat their psychiatric illnesses while they were younger. When the state lost that lawsuit, the solution involved the creation of the CBHI. It had many charges, but the first one it took on was developing a program for behavioral screening of all children from birth onwards. As you know, it studied the available screening tools and propogated a list of 8 that the committee flet were the best validated, and asked all pedatricians an family practitioners to adminsiter one of these instruments to every child they saw for well child care.When we looked at that list we worried about the practical aspects of the measures that were available for children below 5. They were lengthy, many were proprietary, all cost money, scroing was somewhat complex, and they only addressed behavior, not other aspects of development.
4 Our goals Surveillance Instrument (birth to 5) First level screening Integrate “behavior” and “development”Include family context/risksQuick to fill out, score & interpret“Developmental growth curve”Free and in public domainAmenable to electronic formatComputerTelephoneInternetSo we were just naïve enough to think we could build a better mousetrap. We wanted to create an instrument that oculd be used at every well child visit, essentially a surveillance instrument. This would be a first level screen, it would integrate cognitive, language, motor, and socail-emotional development, as well as assessing family risks and also autism at the appropriate age. We knew it would have to be short, and easy to adminsiter and to score, and that it would have to be free and easily available. We also wanted to make sure that it would be easy eventially to transfer it onto a computer platform.
5 Survey of Well-being of Young Children SWYCDevelopmentBehaviorWe called our new instrument the “Survey of Well-being of Young Children” or SWYC,Family Context5
6 Development Behavior Family Risk Preschool Pediatric Symptom Checklist (PPSC)DevelopmentBehaviorInternalizingExternalizingBaby Pediatric Symptom Checklist (BPSC)For behavior, we considered the Pediatric Symptom Checklist, or PSC, to be a model. The PSC is short, easy to score with only one total score, and it is freely available. Based on a host of validation studies, the PSC has recently been provisionally accepted by the National Quality Center to assess medical outcomes—it is one of only two psychiatric measures to do so (the other is the PHQ9). Unfortunately, the PSC is only valid for children 5 and up, but fortunately, Drs. Mike Murphy and Mike Jellinik, the creators of the PSC, agreed to collaborate on a downward extension. Thus, we created the Preschool PSC for children 18 months – 5 years, and the Baby PSC for children under 18 months.For the domain of emotional/behavioral development, we took the Pediatric Symptom Checklist as the model, and worked with Michale Murphy and Michael Jellinek to create a downward extension of that instrument. As you probably know, the PSC is free and in the public domain, and is widely respected and used as a screening tool in pediatric and other settings. Se we are calling our instruments the Baby PSC, for children from 1 to 18 months, and the Preschool PSC, for children from 18 months to 5 years.Family Risk6
7 Development Behavior Family Risk Milestones Preschool Pediatric Symptom Checklist (PPSC)Parent’s Observations of Social Interactions (POSI)DevelopmentDelaysAutismBehaviorInternalizingExternalizingBaby Pediatric Symptom Checklist (BPSC)For cognitive, language and motor development, we planned two measures. The Milestones was conceptualized as a way to help structure physicians’ surveillance of key milestones across developmental domains at each well-child visit. The POSI was created as a brief autism-specific screening tool STOP to be administered between 18 and 36 months,Family Risk7
8 Development Behavior Family Risk FAMILY RISK FACTORS Depression PHQ-2 Substance AbuseAlcoholTobaccoOther drugsParental DiscordHungerFor family risk factors, we were able to identify a number of 1-2 questions assessments of problems that affect child well-being and are potentially amenable to intervention by the pediatrician, STOP including the PHQ2 for parent depresseion, screening questions for parental substance abuse, 2 questons about marital discord, and a single-item screen for hungerFamily RiskFAMILY RISK FACTORS8
9 Four Domains Social-emotional-behavioral Cognitive, Language and Motor BPSC (Birth to 18 months) – 12 itemsPPSC (18 months to 5 years) – 18 itemsCognitive, Language and MotorMilestones (each WCV) – 10 itemsAutismPOSI (18 to 36 months) – 7 itemsFamily RisksEach WCV - 11 itemsSo, just to reiterate, the whole instrument includes a social-emotional-behavioral screen, a developmental screen, an autism screen, and a family risk screen. Obviously the specific questions are different for different age children, so this either needs to be converted to a computer or telephone format, or you have to have a whole bunch of stacks of papers for the receptionist to hand out to families. One practice has put this up on a patient protal, and asks families to fill it out from home the night before their appointment.So what we have is a set of instruments that together, we hope, describe a child’s overall well-being. We have completed an iniital validation of the measures and I’m going to let my colleague Chris Sheldrick describe that process to you.
10 SWYC Psychometrics! Latent variable modeling Comparative effectiveness I know that nothing gets people excited like psychometrics.The CDC actually invited us to give a talk about the SWYC, and after regaling them for about 45 minutes with all of the ins and outs of the SWYC’s psychometrics, several audience members politely suggested that I try to be a bit more succinct the next time. So here it goes.There are two main types of analyses we have done with the SWYC that are worth knowing about:Latent variable modeling andComparative effectiveness analyses
11 Latent variable modeling item clustersPPSC bifactor modelReliabilityItem 1f1Item 2Differential Item FunctioningItem 3Item 4general factorItem 5Item 6Item 7Social/ emotional problemsf2Item 8Item 9Item 10Item 11For example, here is a depiction of the bifactor model we developed for the BPSC. It fit the data well, and it is important for a couple reasons. The first is reliability.The model tells us that the items function well together, and that it makes sense to combine them to score a single total scale. Replicating these findings across multiple samples gives us greater faith in its reliability.The second is differential item functioning, which is a way of testing whether the items perform as well across different populations.For example, we tested items by SES (i.e., family income and parent education).We know expected that social/emotional problems would be more common among low-SES children than higher SES children, as depicted here.The question was whether parents from different SES groups answered PPSC questions differently, even after controlling for their child’s overall level of social/emotional problems. Through latent variable modeling, we were able to eliminate items that displayed high levels of DIF with respect to a range of factors, including SES and race/ethnicity. This increases our confidence that the SWYC will work well across diverse populations.Item 12Item 13f3Item 14Item 15Item 16?Item 17SESItem 1811
12 PRIMARY CARE SAMPLE Sensitivity Specificity 1.0 1.0 0.8 0.8 0.6 0.6 PPSCASQ-SE1.0PPSCASQ-SE0.80.80.60.60.40.4In a separate set of analyes, we compared the how well the SWYC predicted key outcomes to how well other validated screenign instruments predicted key outcomes. On the left, the blue lines depict the PPSC’s sensitivity in predicting CBCL total scores, internalizing scores, and externalizing scores. The red lines are for the ASQ-SE, an instrument that is already approved by CBHI. You can see that the confidence intervals overlap, suggesting no significant difference. Specificity is on the right.0.20.20.00.0Total Internalizing ExternalizingCBCL 1.5-5Total Internalizing ExternalizingCBCL 1.5-5
13 CLINIC SAMPLE Sensitivity Specificity 1.0 1.0 0.8 0.8 0.6 0.6 0.4 0.4 PPSCASQ-SE1.0PPSCASQ-SE0.80.80.60.60.40.4Here you can see the same set of analyses in a clinical sample, enrolled mostly from DBP and psychiatry clinics. Again, the PPSC performs at least as well as the ASQ-SE.0.20.20.00.0Total Internalizing ExternalizingCBCL 1.5-5Total Internalizing ExternalizingCBCL 1.5-513
14 Sensitivity Specificity 1.00.80.60.4In the combined sample, we looked at how well the PPSC predicted parent reports of diagnoses, including ADHD, anxiety, and other conduct problems. Again, the PPSC performed about as well as the ASQ-SEPPSCASQ-SECBCL 1.5-50.20.0ADHD Anxiety Behavior or Conduct Any diagnosisParent report of existing diagnosisSpecificity14
15 Other SWYC studies POSI v MCHAT Milestones v ASQ-3 BPSC v ??? We have done similar studies for other parents of the SWYC. For example, the POSI has displayed higher sensitivity than the MCHAT in two studies. In a DBP clinic, the POSI detected 89% of children with confirmed diagnoses of autism, while the MCHAT only detected 71%. In a separate study, the difference in sensitivities was even greater, although the POSI’s specificity of 75% was somewhat lower than the MCHAT’s at 84%.In an initial study, the Milestone’s 10 items detected developmental delays as well as the ASQ-3, and a replication study is nearing completion.The BPSC has been the toughest one to validate because there are no well accepted criteria for behavioral disorders for kids under 18 months. Therefore, we know that the BPSC has good factor structure and that its items don’t display strong DIF, and we also know that it correlates well with the three forms of the ASQ-SE that are available for this age group.
16 Social/ Emotional Behavior Example SWYC FormsMilestones12-Month24-MonthSocial/ Emotional Behavior
17 Page 2 (on the back)Autism12-Month24-MonthFamily Risk Questions
18 Summary Surveillance/first level screening instrument Includes Cognitive, language, motor, and social-emotional developmentAutism spectrum disordersFamily context/risksCompares well with older instrumentsFree, public domainEnglish and SpanishConvertible to computer platformTailored for child’s ageLink to follow-up questionnaires