Chapter 5: Improving and Assessing the Quality of Behavioral Measurement Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Indicators of Trustworthy Measurement Validity Directly measures a socially significant behavior Measures a dimension of the behavior relevant to the question Ensures the data are representative Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Indicators of Trustworthy Measurement Accuracy Observed values match the true values of an event Reliability Measurement yields the same values across repeated measurement of the same event Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Threats to Measurement Validity Indirect measurement Measuring a behavior other than the behavior of interest Requires inferences be made about the relationship between those behaviors Must provide evidence that the behavior measured is directly related to behavior of interest Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Threats to Measurement Validity Measuring a dimension that is irrelevant or ill suited to the reason for measuring behavior Measurement artifacts Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Measurement artifacts Misleading data that result from the way behavior is measured: Discontinuous measurement Poorly scheduled observations Insensitive or limiting measurement scales Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Threats to Measurement Accuracy and Reliability Human error Poorly designed measurement systems Cumbersome Difficult to use Complex Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Threats to Measurement Accuracy and Reliability Inadequate observer training Explicit and systematic Careful selection Train to competency standard On-going training to minimize observer drift Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Threats to Measurement Accuracy and Reliability Unintended influences on observers Observer expectations of what the data should look like Observer reactivity when she/he is aware that others are evaluating the data Measurement bias Feedback to observers about how their data relates to the goals of intervention Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Assessing the Accuracy and Reliability of Behavioral Measurement First, design a good measurement system Second, train observers carefully Third, evaluate extent to which data are accurate and reliable Measure the measurement system Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Assessing the Accuracy of Measurement Accuracy means the observed values match the true values of an event No one wants to base research conclusions or treatment decisions on faulty data Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Assessing the Accuracy of Measurement Four purposes of accuracy assessment: Determine if data are good enough to make decisions Discovery and correction of measurement errors Reveal consistent patterns of measurement error Assure consumers that data are accurate Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Accuracy Assessment Procedures Measurement is accurate when observed values match true values Accuracy determined by calculating correspondence of each data point with its true value Process for determining true value must differ from measurement procedures Accuracy assessment should be reported in research Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Assessing the Reliability of Measurement Measurement is reliable when it yields the same values across repeated measures of the same event Not the same as accuracy Reliable application of measurement system is important Requires permanent products for re-measurement Low reliability signals suspect data Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Using Interobserver Agreement to Assess Behavioral Measurement The degree to which two or more independent observers report the same values for the same events Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Benefits of Interobserver Agreement (IOA) Determine competence of new observers Detect observer drift Judge clarity of definitions and system Increase believability of data Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Requisites for IOA Observers must: Use the same observation code and measurement system Observe and measure the same participants and events Observe and record independently of one another Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Methods for Calculating IOA Percentage of agreement is most common Event Recording methods compare: Total count recorded by each observer Mean count-per-interval Exact count-per-interval Trial-by-trial Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Methods for Calculating IOA Timing recording methods: Total duration IOA Mean duration-per-occurrence IOA Latency-per-response Mean IRT-per-response Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Methods for Calculating IOA Interval recording and Time sampling: Interval-by-interval IOA (Point by point) Scored-interval IOA Unscored-interval IOA Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Considerations in IOA During each condition and phase of a study Distributed across days of the week, time of day, settings, observers Minimum of 20% of sessions, preferably 25-30% More frequent with complex systems Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Considerations in IOA Obtain and report IOA at the same levels at which researchers will report and discuss in study results For each behavior For each participant In each phase of intervention or baseline Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Considerations in IOA More conservative methods should be used Methods that will overestimate actual agreement should be avoided If in doubt, can report more than one calculation Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Considerations in IOA Believability of data increases as agreement approaches 100% History of using 80% agreement as acceptable benchmark Depends upon the complexity of the measurement system Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Considerations in IOA Reporting IOA Narrative form Table Graphs In all formats, report how, when, and how often IOA was assessed Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition

Assessing the Quality of Measurement Indicators of the quality of data include: IOA Accuracy Reliability Can report multiple indices to assess data quality Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition