1. Bowden, Shores, & Mathias (2006): Failure to Replicate or Just Failure to Notice.
Does Effort Still Account for More Variance in Neuropsychological Test Scores than TBI Severity?
Martin L. Rohling George J. Demakis
UNIVERSITY OF SOUTH ALABAMA - MOBILE
UNIVERSITY OF NORTH CAROLINA - CHARLOTTE
INTRODUCTION
Green et al. (2001) claimed that effort accounts for more variance in test scores than does severity of traumatic brain injury (TBI). In fact, they reported a correlation of .73 between a composite score of 5 effort measures and a composite score of 43 cognitive measures (i.e., Overall Test Battery Mean or OTBM). Furthermore, they stated that effort accounted for 53% of the variance in cognitive test scores, which was over 25 times the variance accounted for by TBI severity. Finally, they noted that failing an effort test suppressed the OTBM by 1.20 SDs as compared to those passing effort tests.
Bowden et al. (2006) claimed to have failed to replicate Green et al. (2001). Bowden et al. suggested that the validity of the WMT, and cognitive symptom validity tests (SVT) in general, is in doubt. They cautioned clinicians against making diagnoses based on scores from such tests, particularly the WMT. They concluded that SVTs like the WMT are no different from other memory tests and do not assess a unique construct that has been referred to as “effort.” In fact, Bowden et al. (2006) stated that:
“…the correlations between WMT scores and measures of intelligence or overall test battery performance (OTBM: Green et al., 2001) may simply indicate that all of the WMT scores measure verbal memory (pg. 868).”
Because of obvious differences between these 2 studies & the very different conclusions drawn by the two author groups, we re-examined results of Green et al. (2001) & Bowden et al. (2006) to see what might have caused the two sets of authors to reach such disparate conclusions regarding the WMT.
RESULTS & DISCUSSION
With respect to cognition (i.e., OTBM), no effect of age in either sample. None expected due to age-corrected scores. Education had effect in both samples. Significance effect of PTA in Green sample but not Bowden sample. GCS had an effect in both samples. LOC had an effect in Green sample. Contrary to prediction, no effect of CT/MRI findings and OTBM in Green sample.
Related to the WMT-IR, small effect in Green sample but no effect in Bowden sample. No effect of education in either sample. No effect of PTA in either sample. No effect of GCS in either sample. No effect of LOC in Green sample. Effect of CT/MRI findings opposite of prediction, with abnormalities associated with higher scores.
Summarizing, TBI severity associated with poorer cognition but unrelated to WMT scoes. Across both samples, the majority of analyses found effect of variables known to impact cognition evident in these samples (i.e., 90%). However, 88% of these same analyses found no effect on WMT-IR. Results indicate that the WMT measures a construct other than cognition.
Cognitive Test Score Variance Accounted for by Effort: We repeated the analysis of Green et al. (2001), which generated a r between WMT-IR and OTBM of .47 (p < .0001; Rho = .49). The same analyses of Bowden sample resulted in an r of .55 (p < .0001; Rho = .59). Thus, effort accounted for 23% to 33% of the variance in OTBM.
Influence of TBI Severity on Effort Test Scores: Looking at the WMT-IR, in the Green sample, PTA accounted for only 1.7% of the variance and just .1% in the Bowden sample. GCS accounted for < .1% of the variance in Green sample and 2.0% in the Bowden sample, but in opposite direction. LOC in Green sample accounted for .5% of the variance and CT/MRI findings accounted for just .4% of the variance.
Influence of TBI Severity on Cognitive Test Scores: In the Bowden sample, PTA accounted for 2.6% of the variance in OTBM scores. In the Green sample, PTA accounted for just .3% of the variance in OTBM scores. For the Bowden sample, GCS accounted for 15.2% of the variance in the OTBM. For the Green sample, GCS accounted for 3.2% of the variance in OTBM scores. Finally, for the Green sample, LOC accounted for 9.6% of the variance in OTBM scores, but CT/MRI abnormalities accounted for less than .1% of the variance in the OTBM scores. Failing WMT-IR suppressed the OTBM by -.86 (Green) and (Bowden) SD units.
Looking for an Interaction between Cognitive Ability, TBI Severity, & Effort
For PTA in Bowden sample, main effect of repeated measure (p = .002), no effect of PTA, and no interaction. However, for in Green sample, there was a main effect of repeated measure (p = .001), small effect of PTA (p = .11), and an interaction (p = .002).
Looking at GCS, in Bowden sample, main effect of repeated measure (p = .01), but no effect of GCS, and no interaction. However, for the Green sample, there was a main effect of repeated measures (p = .003), no effect of GCS, but a trend for an interaction (p = .30).
In the Green sample with LOC, there was not a main effect of repeated measure, no effect of LOC, but the interaction was significant (p < .05). For CT/MRI findings, there was a main effect of repeated measure (p < .0001), a main effect of CT/MRI abnormalities (p = .007), and an interaction (p < .0001).
Summarizing, the results with LOC and CT/MRI abnormalities are consistent with those obtained using PTA and GCS for the Green sample, indicating that patients with more severe TBIs performed relatively better on effort measures than on cognitive measures. Those with milder TBIs scored more poorly on effort measures than they did on cognitive measures. Such findings are inconsistent with the WMT being a measure of cognition.
Thus, averaged across samples and measures, TBI severity accounted for 5% of the variance in OTBM scores, but none of the variance in WMT-IR scores. The variance accounted for by effort on the OTBM was estimated to be 25%. This means that effort accounted for 5.0 times the variance in cognitive test scores than TBI severity. This is similar to what was reported by Green et al. (2001), which was that effort accounted for 4.5 times the variance in cognitive test scores than TBI severity. We found more similarities than differences between the Green and Bowden samples. Our results support the construct validity of the WMT as an effort test.
ABSTRACT
Several studies have reported that TBI has a very small effect on test scores, as compared with the effect of poor effort on neuropsychological performance. As a consequence, many authors have concluded that effort needs to be measured routinely and that it is necessary to control for poor effort when measuring the effects of brain disease or injury on performance. Recently, however, Bowden et al. (2007) have challenged these notions. They argued that the Immediate Recognition subtest of the Word Memory Test (Green & Flaro, 2003), an effort measure, is simply another test of memory. In this study, we re-examine the data from Bowden et al. (2007) and Green et al. (2001) to identify differences between the two studies that might account for their contradictory conclusions. In both sets of data, reanalysis showed that effort explains approximately five times more of the variance in composite neuropsychological test scores than TBI severity. Moreover, scores on the Word Memory Test – Immediate Recognition (WMT-IR) were not found to correlate with major variables known to be measuring ability (e.g., years of education). These findings challenge the conclusions offered by Bowden et al. (2006).
METHOD (continued)
Differences with Respect to TBI Severity – After excluding non-TBI referrals from the Green sample, the comparison between the two samples found the Bowden sample to be more impaired with respect to GCS (d = -.57, p = .02) and PTA (d = -.45, p < .0001).
Differences with Respect to Education – Green detailed each person’s years of education; whereas, Bowden classified patients (<12, = 12, >12). When examining just TBI referrals, Green’s sample was better educated than the Bowden sample (d = .47, p = .004).
Differences with Respect to Effort – Examining just WMT-IR, the Green sample did better than the Bowden sample (M = 91% vs 88%, respectively, 21% vs. 29% failures).
Procedure:
Participants: Only TBI referrals and adults were included in both samples. No adjustments were made to the samples based on age, education, or TBI severity.
Measures: Test scores were adjusted to be consistent with WAIS-R (1981) normative scores. VIQ, PIQ, & General Memory Index (GMI) were averaged into the Overall Test Battery Mean (OTBM).
Hypotheses
Bowden et al. (2006) claimed the WMT simply measured verbal memory. If this were the case, then variables that have been shown to effect scores on verbal memory measures should have similar associations to scores from the WMT. Such variables would include age, education, and severity of injury as indexed by the GCS, PTA, LOC, and brain abnormalities identified by CT or MRI scans. To investigate this point, comparisons were made between the effects of these variables on measures of cognitive ability, particularly memory, and measures of effort as operationally defined by the WMT.
Alternatively,
Differences might have been caused by differences in the populations sampled.
Differences might have been caused by differences in methods of measurement.
Differences might have been caused by differences in the interpretations made by the respective author groups despite the fact that the two samples revealed similar findings.
RESULTS
METHOD
When attempting to replicate a particular study’s findings, it is important that the two datasets be as comparable as possible. The differences between the Green et al. (2001) and Bowden et al. (2006) studies, in terms of sampling, measurement techniques, and statistical procedures, were of a sufficient magnitude that we were concerned they might have prohibited the authors from coming to similar conclusions. Therefore, we thought it necessary to adjust the datasets by using similar sampling procedures, measurement techniques, and statistical analysis, such that comparisons between the two might be more meaningful.
Listed below are the differences we considered relevant
Differences between Studies in Measures of Effort – Green et al. used all 3 WMT effort measures plus 2 others; whereas, Bowden et al. used just the WMT-Immediate Recognition score.
Differences between Studies in Selected Measures of Cognitive Ability – Green et al. used an overall test battery mean of 43 measures; whereas, Bowden et al. used only WAIS-based PIQ & WMS-based GMI. We generated an OTBM for Bowden that included 16 subtest scores and a similar composite for Green that only included 17 test scores. The included measures were from IQ and Memory tests only.
Difference with Respect to Age – Green included only adults (>18 y/o); whereas 16% of the Bowden sample were children who were under the age of 18. After excluding children, the samples still differed with respect to age, with the Green sample being older (M = 39.8 vs 36.3, d = .30).
Differences with Respect to Diagnoses – Green included many different neurological and psychiatric diagnoses; whereas, Bowden included only individuals who allegedly suffered from a TBI.