1. Coding Manual and Process

2. Illustration, and Introduction of metafor
Teacher expectancy data (Raudenbush and Bryk)
Reading data into R
Acquiring the metafor package
Simple analyses using metafor
What is metafor actually doing?

3. The Coding Manual
Begin by keeping a list of possible covariates you might be interested in
But eventually narrow because meta-analytic sample sizes tend to be small, with insufficient degrees of freedom to support too much data.
Begin preparing a draft coding manual
Consider a web-based coding manual
Saves confusing multiple electronic files that coders give you periodically
Allows coders to work at home
Automatically saves and forwards data to you.

4. Components of Coding Manual
The goal is codes that are useful in describing and predicting effect sizes.
An ID system (to be discussed shortly)
Substantive codes, e.g.,
Treatment subtype
Methodological Codes, e.g.,
Kind of comparison group (TT v TC)
Effect size codes (effect size and sample size per group)
How to deal with missing data
Plausible Guesses
Distinguishing “No” from “Unknown”
Code missing data as NA for analysis in R
Partial example next page, full example CROPS

6. Structure of Coding Manual
In meta-analysis, the effect size is the basic unit being described
The structure of the manual depends on whether your studies are group comparisons or correlational.

7. Typical Structure for Group ComparisonsID
Variables that are constant in a study
e.g., publication status, year published, methods like use of random assignment
Variables associated with Group 1
E.g., characteristics of participants, of treatment, of treatment providers
Variables associated with Group 2
All of Group 1 codes, plus an indicator for whether the comparison is Treatment-Treatment or Treatment-Control
Variables associated with the outcome measure
E.g., self-report or not, training of observers, masking, construct being measured
Variables associated with effect size
E.g., method of computation, sample size.

8. Typical Structure for Correlational StudiesID
Variables that are constant in a study
Variables associated with Variable 1
E.g., what is it measuring, how measured, who measured, etc.
Variables associated with Variable 2
Variables associated with effect size

9. An ID System I keep ID bibliography in Microsoft Word
Purpose: to locate particular places in the data set
To correct errors (e.g., found when cleaning data)
To recode (e.g., to change a particular study)
Components:
Study ID number
Comparison ID number (effect sizes are always on comparisons)
Dependent Variable ID number
Any other information you want to record
To record reasons for excluding studies initially used.
Sample ID system in Handouts (Part on next slide)

11. Training of Coders Crucial to train coders, especially
If large numbers of studies (so you can’t do it all yourself)
If inferences are complex (e.g., should alternating assignment be coded as random?)
If coders are undergraduates (or at that level)
Random spot checks on all codes by PI

12. The Coding Process
Do effect size coding separate from all other coding
Inter-Rater Reliability (IRR)
Decide whether to code some or all studies more than once (advisable if you are not the coder).
Some or all studies? If some, at least 10-20?
Can use third coder (PI?) to resolve disagreements among the first two coders.
Report IRR analysis (kappa; see handout)

13. The Hierarchical Structure of Meta-Analytic Data
In between-group treatment outcome studies,
Some data are constant over the whole study (e.g., year of publication, publication status), and so only needs to be coded once
Some data vary for each treatment condition (e.g., length of treatment, dosage).
Some data vary for each outcome measure (e.g., self- vs other report, reliability)
Some data vary for each effect size (e.g., n per group, computation method), and so needs to be coded for every effect size
So:

14. Square vs Hierarchical Data Sets
The effect size is the basic unit (each line)
Square Data Sets
What we commonly think of as a data set
Every variable coded for each effect size even if redundant
Efficient if number of studies (k) or effect sizes is small.
Hierarchical Data Sets
Code study level variables once for each study
Code treatment variables once for each treatment
Code outcome variables once for each outcome
Code effect size variables once for each effect size
Then use “match files” to duplicate all the missing codes from what would have been the square data set
Most efficient when k or number of effect sizes is large, but more complex to do.

15. Data Entry Depends on square or hierarchical data sets
If square, one file
If hierarchical, one file for each “level”
Option #1: First do paper codes, then enter into computer
More room for transcription errors
Option #2: Direct data entry into, say, SPSS Data Editor, or an Excel Worksheet with a manual as a guide. Or directly into a web-based manual.

16. Example of Excel Coding Sheet