1 Session 3: Framework for a Coding Guide October 27, :00-3:30 PM (EDT) Instructor: Chad Nye

2 Dr. Chad’s Wonder Drug Life’s Elixir of Youth, Beauty, Wisdom & Painless Coding

3 A wholesome blend of natural extracts of thirty different herbs and root vegetables that promotes and revitalizes the capacity for learning and memory, strengthens the nerves, and effectively wards off cataleptic neuroplexy AND coding anxiety. cataleptic neuroplexy

4 Inclusion and Exclusion Criteria Reduce Reviewer Bias: Criteria are determined before beginning Information Retrieval ‘Standardize’ the process and resulting decisions

5 Inclusion and Exclusion Criteria: Potential Categories Study Design Population Intervention Outcomes Measures Publication Sources

6 Coding for Fun and Profit Dr. Chad’s Wonder Drug taken 2 times a day will result in the complete life experience of the life well lived!!!

7 Coding is a 2 Step Process Stage 1 – Title and Abstract Eliminate studies that are clearly inappropriate Used a limited set of criteria

8 Inclusion Criteria Abstract & Title Stage Studies will meet at least one of the following criteria: 1. Participant The participant sample of study must include only adults 18 years or older, with a diagnosis of Autism Spectrum Disorder. 2.Intervention The focus of the study intervention must center on the topic of employment.

9 Inclusion Criteria Abstract & Title Stage Studies can be advanced to the next stage if: 1. They meet the criteria OR 2. If these criteria are not clear from the title or abstract

10 Coding is a 2 Step Process Stage 1– Title and Abstract Eliminate studies that are clearly inappropriate Used a limited set of criteria Stage 2– Full Text Identify the studies that will be included in the review itself Use full set of criteria

11 Inclusion Criteria Full Text Stage Research Design Studies use an experimental (RCT), quasi- experimental design (QED), and single subject experimental design (SSED) Participants Individuals with an autism spectrum disorder who are 18 years of age or older and not enrolled in a secondary school program (e.g., high school)

12 Inclusion Criteria Full Text Stage a. Research Design b. Participants c. Intervention The treatment must be directed towards the provision of vocationally oriented services intended to produce employment outcomes for individuals with autism spectrum disorders.

13 Inclusion Criteria Full Text Stage a. Research Design b. Participants c. Intervention d.Outcome Measures The study must report subsequent attainment of an employment placement and provide specific data about the duration and/or retention of that placement. Employment will be considered to encompass full or part-time (25% time or more) placements. Interventions of any length duration will be included provided adequate description is provided.

14 Inclusion Criteria Full Text Stage a. Research Design b. Participants c. Intervention d. Outcome Measures e. Publication Status Eligible studies may be either published or unpublished, written in any language, and conducted in any country.

15 Inclusion/Exclusion Decision Tree Potentially relevant reports identified (n=583) Reports progressing to next level (n=60) Reports used to code studies for inclusion (n=20) Reports excluded at Phase II screening (n=40) Reports excluded by title and abstract review (n=523)

16 Study Coding* Purpose of Coding 1. Identify all those variables that you believe might make difference in the observed results. 2. Organize those individual variables into categories of characteristics in order to evaluate the quality of the study 3. Try to determine what is really important about the study 4. Quantify both text and data for analysis

17 Coding Form 5. Guide for locating and recording the who, what, where, how, when, and how much of a study (aka-variables) 6. A record of the ‘interpretation’ of the variables that can be independently replicated (aka-a monitor on the accuracy of judgments) 7. Provides the basis for the data arrangement at the analysis stage

18 Study Coding Categories of Study Characteristics Design Participant Treatment Outcome

19 Design Characteristics Type of Design RCT QED Pre-Experimental Single Subject Blinding Attrition Sampling Method

20 Participant Characteristics Information is collected for BOTH Experimental AND Control groups. Number of Participants Age Gender Severity Language Setting

21 Treatment Characteristics Type of Tx Method Length of Tx Program Tx Mode: Group-Individual Number of Tx Sessions Length of Tx Sessions Dosage

22 Outcome Characteristics Outcome Variable(s) Outcome Measure(s) Assessor of Outcome Assessor Qualifications Time Post Tx to Outcome Measure Immediate Follow-Up

23 CodeBook A dictionary of all the terms used in the Coding Form A guide for all coders to follow to reduce error and bias A reference source for accurate replication

24 A. Publication Source 1. Journal2. Dissertation 3.Thesis 4. Unpublished Paper 5. Book 6. Other: ________ B. Subject Characteristics (page(s): ____________) Group Pretest Posttest Attrition Age Grade(s) Males (n) (n) (n) (n)(n) (n) Tx: _22___ _22__ ______ ______ ______ _____ Cp: ______ _____ ______ ______ ______ _____ Co: _22___ __18_ __ 4___ ______ ______ _____

25 C. Socioeconomic Status (page(s):___________ 1.Low 2. Middle3. Upper 4. Low-Middle 5. Middle Upper6. Labeled Mixed 7.Unclear8. ___________9. NR Comments:__________________________________

26 D. Sample Source (page(s)_________) 1.Public School 2. Secular Private School 3. Religious Private School 4. Other:_________________ 5. NR Comments

27 E. Study Setting (page(s) ______) 1.Urban2. Suburban 3. Rural Geographic Setting: ______________________ Comments:

28 F. Author’s Labeling of Participants’ Achievement Levels (page _____) 1.Average Achieving 2. High Achieving 3. Underachieving 4. At-Risk 5. Unclear 6. Other:___________________7. NR G. Race/Ethnicity (page ______) 1.African American 2. American Indian3. Asian 4. Hispanic/Latino 5. White 6. Other: ____________________7. NR

29 H. Intervention Characteristics 1. Type of intervention (page __________________) 2. Total length of intervention program: __________ or NR (page _____) 3. Length of time PER intervention session:________ or NR (page _____) 4. Number of sessions: ________ per ________ or NR (page _____) 5. Describe the intervention(s) intended for delivery to the study groups including how and when it was administered.______________________________________

30 6.Fidelity of implementation data reported? (page(s)______) 0=No 1=Yes If yes, describe the intervention(s) intended for delivery to the study groups including how and when it was administered: ________________________________________________ ________________________________________________ 7. Intervention implemented as described? (page(s) _____) 1=No 2=Yes 3= Somewhat 9=NR

31 I. Outcome Measure(s) Type of Measure Instrument Variable(s)Who Administered the Measurement Instrument? 1. Norm Referenced Test (page____) _________ _________ _________ a.Parent b. Teacher c. Researcher d. Other e. NR 2. Criterion Referenced Test (page____) _________ _________ ________ a. Parent b. Teacher c. Researcher d. Other e. NR 4. Rating/Survey (page____) _________ a. Parent b. Teacher c. Researcher d. Other e. NR 5. Class Grade (page____) _________ _________ _________ a. Parent b. Teacher c. Researcher d. Other e. NR

32 J. Design Characteristics Recruitment Pool (page_____) 1.Referral 2. Criterion3. Norm Reference Test 4. Existing Group5. Cohort 6. Volunteer 7.Other ___________________ 99. NR Research Design Type (pg____) 1. Group (Cluster, Place-based) Randomized Design 2. Individual Randomized Design 3. Quasi-Experiment: Interrupted Time Series Design 4. Quasi-Experiment: Regression Discontinuity 5. Quasi-Experiment: Nonequivalent Comparison Group Design 6. Single Subject Design 7. Other Designs: _____________________________________

33 K. Assignment Procedures (If applicable, otherwise skip to section I) Subject Assignment (page(s) _____) 1.Individual Random 2.Group Random 3.Individual Match Random 4.Quasi-random

34 2. Method of Random Allocation (page(s) _____) Method of Random AllocationWho Implemented the Random Allocation? 1. Random Numbers (page____)a.Parent b. Teacher c. Researcher d. Other e. NR 2. Coin Flip (page____)a. Parent b. Teacher c. Researcher d. Other e. NR 4.Other:_______(page____)a. Parent b. Teacher c. Researcher d. Other e. NR

35 Was concealment of allocation used as part of the randomization procedures? 1=no 2= yes 3=NR a. If yes, were the allocation procedures concealed from any of the following people? 1. Researcher (pg _____) 2. Participant (pg _____) 3. Intervener (pg _____) 4. Assessor (pg _____)

36 I.Quasi-Experimental Assignment Procedures (If applicable, otherwise skip to section J) 1. Was any of the following research design techniques used to equate groups prior to the intervention? a. Matching on pretest proxy (pg _____) b. Matching on a pretest (pg _____) c. Matching on other variables (pg _____) d. No Matching (pg _____) 2. Was statistical equating used prior to the intervention? 1=no 2=yes3.NR Comments:________________________________________

37 J. Effect Size Characteristics Comparisons: Group 1: _TX___vs. Group 2:_Ctl___ Outcomes _Reading___ __________ ___________ Groups __1__ _2_ __1__ __2__ __1_ __2__ Mean _22__ __18_ _____ _____ ____ _____ SD _3.55 _4.22_ _____ _____ ____ _____ N at Posttest _____ _____ _____ _________ _____ F value _____ __________ __________ ____ Chi-square _____ _____________ __________ ____ p value ______________________________ t value ______________________________

38 Method of Analysis (pg _____) a. Intention to Treat: b. Treated Participants Only

39 Excerpt from Master Coding Guide for Study Coding Code the Fantuzzo Study

40 Data Extraction Practice