The Survey of Well-being of Young Children (SWYC) Ellen C. Perrin, M.D. R. Christopher Sheldrick, Ph.D. So I’m going to tell you today about an instrument for surveillance that we have been working on for the past several years.

We have no relevant financial relationships with the manufacturer(s) of any commercial product(s) and/or provider(s) of commercial services discussed in this CME activity. We do not intend to discuss an unapproved/investigative use of a commercial product/device in this presentation.

Historical perspective Rosie D. lawsuit CBHI Available Instruments Lengthy Proprietary Scoring complex Fragmented So most of you remember the lawsuit brought by a few families of mentally ill young adults, attesting that the state had fallen down in its obligation to screen, detect, and treat their psychiatric illnesses while they were younger. When the state lost that lawsuit, the solution involved the creation of the CBHI. It had many charges, but the first one it took on was developing a program for behavioral screening of all children from birth onwards. As you know, it studied the available screening tools and propogated a list of 8 that the committee flet were the best validated, and asked all pedatricians an family practitioners to adminsiter one of these instruments to every child they saw for well child care. When we looked at that list we worried about the practical aspects of the measures that were available for children below 5. They were lengthy, many were proprietary, all cost money, scroing was somewhat complex, and they only addressed behavior, not other aspects of development.

Our goals Surveillance Instrument (birth to 5) First level screening Integrate “behavior” and “development” Include family context/risks Quick to fill out, score & interpret “Developmental growth curve” Free and in public domain Amenable to electronic format Computer Telephone Internet So we were just naïve enough to think we could build a better mousetrap. We wanted to create an instrument that oculd be used at every well child visit, essentially a surveillance instrument. This would be a first level screen, it would integrate cognitive, language, motor, and socail-emotional development, as well as assessing family risks and also autism at the appropriate age. We knew it would have to be short, and easy to adminsiter and to score, and that it would have to be free and easily available. We also wanted to make sure that it would be easy eventially to transfer it onto a computer platform.

Survey of Well-being of Young Children SWYC Development Behavior We called our new instrument the “Survey of Well-being of Young Children” or SWYC, Family Context 5

Development Behavior Family Risk Preschool Pediatric Symptom Checklist (PPSC) Development Behavior Internalizing Externalizing Baby Pediatric Symptom Checklist (BPSC) For behavior, we considered the Pediatric Symptom Checklist, or PSC, to be a model. The PSC is short, easy to score with only one total score, and it is freely available. Based on a host of validation studies, the PSC has recently been provisionally accepted by the National Quality Center to assess medical outcomes—it is one of only two psychiatric measures to do so (the other is the PHQ9). Unfortunately, the PSC is only valid for children 5 and up, but fortunately, Drs. Mike Murphy and Mike Jellinik, the creators of the PSC, agreed to collaborate on a downward extension. Thus, we created the Preschool PSC for children 18 months – 5 years, and the Baby PSC for children under 18 months. For the domain of emotional/behavioral development, we took the Pediatric Symptom Checklist as the model, and worked with Michale Murphy and Michael Jellinek to create a downward extension of that instrument. As you probably know, the PSC is free and in the public domain, and is widely respected and used as a screening tool in pediatric and other settings. Se we are calling our instruments the Baby PSC, for children from 1 to 18 months, and the Preschool PSC, for children from 18 months to 5 years. Family Risk 6

Development Behavior Family Risk Milestones Preschool Pediatric Symptom Checklist (PPSC) Parent’s Observations of Social Interactions (POSI) Development Delays Autism Behavior Internalizing Externalizing Baby Pediatric Symptom Checklist (BPSC) For cognitive, language and motor development, we planned two measures. The Milestones was conceptualized as a way to help structure physicians’ surveillance of key milestones across developmental domains at each well-child visit. The POSI was created as a brief autism-specific screening tool STOP to be administered between 18 and 36 months, Family Risk 7

Development Behavior Family Risk FAMILY RISK FACTORS Depression PHQ-2 Substance Abuse Alcohol Tobacco Other drugs Parental Discord Hunger For family risk factors, we were able to identify a number of 1-2 questions assessments of problems that affect child well-being and are potentially amenable to intervention by the pediatrician, STOP including the PHQ2 for parent depresseion, screening questions for parental substance abuse, 2 questons about marital discord, and a single-item screen for hunger Family Risk FAMILY RISK FACTORS 8

Four Domains Social-emotional-behavioral Cognitive, Language and Motor BPSC (Birth to 18 months) – 12 items PPSC (18 months to 5 years) – 18 items Cognitive, Language and Motor Milestones (each WCV) – 10 items Autism POSI (18 to 36 months) – 7 items Family Risks Each WCV - 11 items So, just to reiterate, the whole instrument includes a social-emotional-behavioral screen, a developmental screen, an autism screen, and a family risk screen. Obviously the specific questions are different for different age children, so this either needs to be converted to a computer or telephone format, or you have to have a whole bunch of stacks of papers for the receptionist to hand out to families. One practice has put this up on a patient protal, and asks families to fill it out from home the night before their appointment. So what we have is a set of instruments that together, we hope, describe a child’s overall well-being. We have completed an iniital validation of the measures and I’m going to let my colleague Chris Sheldrick describe that process to you.

SWYC Psychometrics! Latent variable modeling Comparative effectiveness I know that nothing gets people excited like psychometrics. The CDC actually invited us to give a talk about the SWYC, and after regaling them for about 45 minutes with all of the ins and outs of the SWYC’s psychometrics, several audience members politely suggested that I try to be a bit more succinct the next time. So here it goes. There are two main types of analyses we have done with the SWYC that are worth knowing about: Latent variable modeling and Comparative effectiveness analyses

Latent variable modeling item clusters PPSC bifactor model Reliability Item 1 f1 Item 2 Differential Item Functioning Item 3 Item 4 general factor Item 5 Item 6 Item 7 Social/ emotional problems f2 Item 8 Item 9 Item 10 Item 11 For example, here is a depiction of the bifactor model we developed for the BPSC. It fit the data well, and it is important for a couple reasons. The first is reliability. The model tells us that the items function well together, and that it makes sense to combine them to score a single total scale. Replicating these findings across multiple samples gives us greater faith in its reliability. The second is differential item functioning, which is a way of testing whether the items perform as well across different populations. For example, we tested items by SES (i.e., family income and parent education). We know expected that social/emotional problems would be more common among low-SES children than higher SES children, as depicted here. The question was whether parents from different SES groups answered PPSC questions differently, even after controlling for their child’s overall level of social/emotional problems. Through latent variable modeling, we were able to eliminate items that displayed high levels of DIF with respect to a range of factors, including SES and race/ethnicity. This increases our confidence that the SWYC will work well across diverse populations. Item 12 Item 13 f3 Item 14 Item 15 Item 16 ? Item 17 SES Item 18 11

PRIMARY CARE SAMPLE Sensitivity Specificity 1.0 1.0 0.8 0.8 0.6 0.6 PPSC ASQ-SE 1.0 PPSC ASQ-SE 0.8 0.8 0.6 0.6 0.4 0.4 In a separate set of analyes, we compared the how well the SWYC predicted key outcomes to how well other validated screenign instruments predicted key outcomes. On the left, the blue lines depict the PPSC’s sensitivity in predicting CBCL total scores, internalizing scores, and externalizing scores. The red lines are for the ASQ-SE, an instrument that is already approved by CBHI. You can see that the confidence intervals overlap, suggesting no significant difference. Specificity is on the right. 0.2 0.2 0.0 0.0 Total Internalizing Externalizing CBCL 1.5-5 Total Internalizing Externalizing CBCL 1.5-5

CLINIC SAMPLE Sensitivity Specificity 1.0 1.0 0.8 0.8 0.6 0.6 0.4 0.4 PPSC ASQ-SE 1.0 PPSC ASQ-SE 0.8 0.8 0.6 0.6 0.4 0.4 Here you can see the same set of analyses in a clinical sample, enrolled mostly from DBP and psychiatry clinics. Again, the PPSC performs at least as well as the ASQ-SE. 0.2 0.2 0.0 0.0 Total Internalizing Externalizing CBCL 1.5-5 Total Internalizing Externalizing CBCL 1.5-5 13

Sensitivity Specificity 1.0 0.8 0.6 0.4 In the combined sample, we looked at how well the PPSC predicted parent reports of diagnoses, including ADHD, anxiety, and other conduct problems. Again, the PPSC performed about as well as the ASQ-SE PPSC ASQ-SE CBCL 1.5-5 0.2 0.0 ADHD Anxiety Behavior or Conduct Any diagnosis Parent report of existing diagnosis Specificity 14

Other SWYC studies POSI v MCHAT Milestones v ASQ-3 BPSC v ??? We have done similar studies for other parents of the SWYC. For example, the POSI has displayed higher sensitivity than the MCHAT in two studies. In a DBP clinic, the POSI detected 89% of children with confirmed diagnoses of autism, while the MCHAT only detected 71%. In a separate study, the difference in sensitivities was even greater, although the POSI’s specificity of 75% was somewhat lower than the MCHAT’s at 84%. In an initial study, the Milestone’s 10 items detected developmental delays as well as the ASQ-3, and a replication study is nearing completion. The BPSC has been the toughest one to validate because there are no well accepted criteria for behavioral disorders for kids under 18 months. Therefore, we know that the BPSC has good factor structure and that its items don’t display strong DIF, and we also know that it correlates well with the three forms of the ASQ-SE that are available for this age group.

Social/ Emotional Behavior Example SWYC Forms Milestones 12-Month 24-Month Social/ Emotional Behavior

Page 2 (on the back) Autism 12-Month 24-Month Family Risk Questions

Summary Surveillance/first level screening instrument Includes Cognitive, language, motor, and social-emotional development Autism spectrum disorders Family context/risks Compares well with older instruments Free, public domain English and Spanish Convertible to computer platform Tailored for child’s age Link to follow-up questionnaires

eperrin@tuftsmedicalcenter.org rsheldrick@tuftsmedicalcenter.org