Developing Evidence-Based Products Using the Systematic Review Process Session 3/Unit 9: Inclusion and Exclusion Criteria Screening and Coding Studies Julia Lavenberg October 30, 2007 NCDDR training course for NIDRR grantees

2 Outline Study selection vs. data extraction Requirements for each Exploration of screening and coding via a worked example Application of inclusion and exclusion criteria and codes to planned reviews

3 Study Selection vs. Data Extraction The process by which one chooses studies for inclusion in a systematic review. Accomplished by instituting specific and detailed inclusion and exclusion criteria. The process by which one locates and transcribes information from a primary study. Accomplished by applying specific and detailed criteria to the information in a primary study.

4 Eligibility criteria Provide readers with an idea of the research domain of interest Aid the systematic reviewer in applying consistent and objective standards throughout the selection process Clearly circumscribe the review

5 Inclusion and Exclusion Criteria: Categories Publication types Study design Population Intervention Outcomes Measures

6 Publications Identify the types of reports to be included Determine any geographic or linguistic limitations “Studies eligible for this review may be published or unpublished reports (e.g., dissertations theses, government reports, school district reports, etc.) of school-based interventions conducted in any country and reported in any language.” (Lavenberg, 2007)

7 Study design Specify the research designs to be included Also identify the research designs to be excluded Address the rationale for inclusion and exclusion Consider theoretical framework Consider available evidence

8 Population Indicate the desired target population Stipulate any required characteristics Additionally, distinguish characteristics that would make the target population ineligible “Studies of interventions that target children and youth who are enrolled in kindergarten through grade 12 (or the international equivalents) at public, private, parochial, or alternative schools and are between the ages of 4 and 20 years will be included in this review. Persons identified as attending ‘preschool’ or ‘college’ will not be included, even if they are within the acceptable age range.” (Lavenberg, in process)

9 Intervention Detail the required characteristics of the intervention Provide definitions List examples of what would be included and what would be excluded

10 Outcomes Indicate outcome variables of interest “At least one quantitative measure of aggressive behavior must be reported in each study….[citations to literature here]… Therefore, both physical and verbal aggressive behaviors will be considered acceptable outcomes.” (Lavenberg, in process)

11 Measures Address acceptable measures “Standardized measures of aggressive behavior (e.g., Child Behavior Checklist – Teacher Report Form….) and unstandardized measures with adequate face validity (e.g., local administrative records…) will both be considered acceptable forms of reporting aggressive behavior. Measures reported in the Buros Institute of Mental Measures Yearbook…will be considered standardized measures; all others will be considered unstandardized measures.” (Lavenberg, in process)

12 Example: Application to a review with > 900 records retrieved (Lavenberg, 2007) Two phase screening process developed Phase I – title and abstract Goal: differentiate potential studies from clearly inappropriate studies Evaluate each study for topical relevance, setting, participants, and general study design Phase II – method section Goal: create the pool of eligible studies Evaluate each study for intervention, design details, and outcome

13 Selection process for including reports Potentially relevant reports identified (n>900) Reports retrieved for further review (n=155) Reports progressing to next level (n=109) Reports used to code studies for inclusion (n=50) Reports excluded at Phase II screening (n=59) Reports excluded at Phase I screening (n=46) Reports excluded by title and abstract review (n=750)

14 Data Extraction (i.e., “coding”) Initiated after identifying appropriate pool of studies for the systematic review Accomplished by means of a coding form and codebook Goal: ensure reliable and orderly extraction of information from each study report

15 Forms… May be either paper-based or computer- based Are best completed by trained coders Should have items organized such that the order reflects the manner in which information is presented in the study report Contain the same categories of information as inclusion and exclusion criteria

16 Coding categories That is, the data extracted will address: Publication type Design Participants Interventions Outcomes Measures

17... in coding, much more detail is required. The data that is extracted from each study report is the basis of all subsequent analyses. BUT:

18 Overarching principle: Conserve as much of the original information as possible. Fewer coder judgments = fewer errors

19 Publication characteristics Document publication year Distinguish between types Create mutually exclusive categories Must assign numerical values to these categorical variables

20 Study design and methodological characteristics Consider identifying the recruitment pool to provide additional context Describe the unit of assignment (e.g., individual, family, school, clinic, workplace) Address the mechanism of allocation (e.g., random assignment, matching on pretest measures of outcome variables, etc.) List unit of analysis Note whether unit of assignment matches unit of analysis

21 Participant characteristics Age May need to capture data in multiple ways (e.g., school grade, chronological age, mental age, age range) Gender Race/ethnicity Socioeconomic status Specific characteristics relevant to review topic (e.g., official diagnoses, developmental categories, baseline level of behavior) Co-morbidity

22 Intervention characteristics Describe important aspects of the intervention Characteristics or features Total length of intervention, number of sessions, length of each session Method of delivery Personnel implementing intervention Fidelity of implementation Identify comparison group conditions

23 Outcome and measure characteristics Identify outcome variables of interest Allow for and document variations Identify outcome informant or source of information Determine if informant was blind to assignment Identify personnel administering measure Specify outcome measures used Indicate time of outcome assessment

24 Effect size characteristics Extract information required to calculate an effect size Sample size (intervention and control groups) Pretest and posttest Attrition Means, standard deviations d index F value Others: Chi-square, p value, t value, etc.

25 Worked example #1 Approaches to Parent Involvement for Improving the Academic Performance of Elementary School Age Children Nye, Turner, & Schwartz (2006) doc-pdf/Nye_PI_Review.pdf

26 Definition of parent involvement “…the parent has a direct interaction with the child in either the delivery or monitoring of the program or intervention.” (page 11)

27 Inclusion criteria Design: limited to randomized controlled trials (RCTs) Rationale: “…in order to provide the least biased estimate of the effect of parent involvement on student achievement and to control for variation in threats to internal validity.” (pg. 12)

28 Inclusion criteria, continued Participants: described in detail in the coding categories section of the review (pg. 13) Age Grade Gender SES Ethnicity

29 Inclusion criteria, continued Interventions: “…studies reported the following characteristics of the intervention program: 1) Parent involvement with their child in academic support activities outside of school (e.g., reading or completing supplemental math problems with the child), and 2) Parent involvement [as defined earlier] for a minimum of 20 days” (pg. 11)

30 Inclusion criteria, continued Outcomes: “Included studies reported the following outcomes on children’s academic performance in: reading, mathematics, spelling, writing, language arts, or science. (pgs )

31 Inclusion criteria, continued Measures: not specified as an inclusion criteria in Nye, et al. Again, specification of this characteristic will depend on the topic and decisions made by the review team e.g., depression as measured by Beck inventory

32 Nye, Turner, & Schwartz Coding categories and coding sheet (unique elements) Report characteristics Subject characteristics Treatment and comparison groups separately addressed Sample source – types of schools Classroom assignments – high achieving, underachieving, average, academically at-risk Developmental categories – emotionally disabled, physically disabled, hearing impaired, speech/language impaired, etc. 1NyeTurnerSchwartz_PI_CodingForm_2006.doc

33 Coding categories, continued Intervention characteristics Coder description required, in addition to numerical data (e.g., total length of intervention program, length of time per day of PI activity, etc.) Fidelity of implementation Outcome measures – norm referenced vs criterion referenced vs rating scale vs survey Design characteristics Recruitment pool Specific method of randomization and identification of who implemented the process Effect size characteristics

34 Review topics Weight loss intervention to reduce physical disability in older adults Improving the ambulatory transition of pre- and post-adolescents with cerebral palsy Improving cultural competency to increase the employment of persons with disabilities

35 Review topics, continued Use of assistive technology and employment supports in the employment of working age people with disabilities Health promotion programs that improve primary access for adults with disability

36 Contact information Julia Lavenberg

37 SUPPLEMENTAL SLIDES

38 Worked example #2 Effects of School-Based Cognitive- Behavioral Anger Interventions: A Meta-Analysis (Lavenberg, 2007)

39 Primary Research Question: Are school-based cognitive-behavioral anger interventions effective in reducing child and adolescent aggressive behavior in the school setting?

40 Secondary Research Questions… Involved looking at whether the following were related to the effect size: Methodological issues (e.g., random assignment) Setting Geographic location (urban, suburban, rural) Delivery personnel Teachers vs. counselors Stand-alone vs. embedded anger interventions Participant characteristics (e.g., gender, age, baseline level of aggressive behavior)

41 Define and operationalize critical elements: Aggression Anger Cognitive-behavioral Theory Intervention Anger management intervention School-based intervention

42 Screening Forms and Manual Copy sent to Joann Starks for posting 3Lavenberg_2007_Screening_Coding_FormsManuals.doc

43 Coding Form and Manual Copy sent to Joann Starks for posting 3Lavenberg_2007_Screening_Coding_FormsManuals.doc