Bowden, Shores, & Mathias (2006): Failure to Replicate or Just Failure to Notice. Does Effort Still Account for More Variance in Neuropsychological Test.

Slides:



Advertisements
Similar presentations
Exaggeration Affects Test Scores Far More Than Severe Brain Injury Paul Green, Ph.D. Neurobehavioural Associates, Edmonton, Alberta, CA Martin L. Rohling,
Advertisements

Method Participants 184 five-year-old (M age=5.63, SD=0.22) kindergarten students from 30 classrooms in central Illinois Teacher ratings The second edition.
Examining the Relationship Between Confrontational Naming Tasks & Discourse Production in Aphasia Leila D. Luna & Gerasimos Fergadiotis Portland State.
Neuropathology and Cognitive Scores Workgroup The role of vascular and Alzheimer’s Disease pathology in differential cognitive impairment among older adults.
Developing and validating a stress appraisal measure for minority adolescents Journal of Adolescence 28 (2005) 547–557 Impact Factor: A.A. Rowley.
Signed Paired Associates Test (SPAT). SPAT Structure Similar to WMS “paired associates” subtest 14 sign pairs – 7 easy & 7 hard Based on sign associate.
The Effects of Increased Cognitive Demands on the Written Discourse Ability of Young Adolescents Ashleigh Elaine Zumwalt Eastern Illinois University.
Meta-analysis & psychotherapy outcome research
Linguistic Demands of Preschool Cognitive Assessments Glenna Bieno, Megan Eparvier, Anne Kulinski Faculty Mentor: Mary Beth Tusing Method We employed three.
Barbara A. Wilson, Eve Greenfield, Linda Clare, Alan Baddeley, Janet Cockburn, Peter Watson, Robyn Tate, Sara Sopena, Rory Nannery & John Crawford (2008)
Participants and Procedure  Twenty-five older adults aged 62 to 83 (M = 70.86, SD = 5.89).  Recruited from St. John’s and surrounding areas  56% female.
S14: Analytical Review and Audit Approaches. Session Objectives To define analytical review To define analytical review To explain commonly used analytical.
Introduction The Relationship between Neurocognitive Functioning and Childhood Abuse among Persons with SMI: Mediating Proximal and Distal Factors L. Felice.
Experimental Research Methods in Language Learning Chapter 10 Inferential Statistics.
Issues concerning the interpretation of statistical significance tests.
The Self-Compassion Scale (SCS) is the primary measure of self- compassion in both social/personality psychology and clinical research (Neff, 2003). It.
Personally Important Posttraumatic Growth as a Predictor of Self-Esteem in Adolescents Leah McDiarmid, Kanako Taku Ph.D., & Aundreah Walenski Presented.
Analytical Review and Audit Approaches
VALIDITY, RELIABILITY & PRACTICALITY Prof. Rosynella Cardozo Prof. Jonathan Magdalena.
Chapter 9: Introduction to the t statistic. The t Statistic The t statistic allows researchers to use sample data to test hypotheses about an unknown.
Project VIABLE - Direct Behavior Rating: Evaluating Behaviors with Positive and Negative Definitions Rose Jaffery 1, Albee T. Ongusco 3, Amy M. Briesch.
The Clinical Utility of the LANSE- A and LANSE-C Jennifer L. Harrison, M.A., Megan Pollock, M.A., Amy Mouanoutoua, M.A. Ashley Brimager, M.A., & Paul C.
Wedded Bliss: Dual Incomes and Shared Finances Amanda Swope & Dr. April Phillips Northeastern State University, Broken Arrow Introduction There are many.
Abstract Recent studies assessing the neuropsychological effects of PTSD and/or remote mTBI suggest that there is negligible long-term cognitive dysfunction.
Methods of Presenting and Interpreting Information Class 9.
Association of Body Mass Index (BMI) and Depression Severity
Chapter 8 Introducing Inferential Statistics.
Rachel L. Fazio, Psy. D. , Allison N. Faris, Psy. D. , Karim Z
Predicting Brain Size Samantha Stanley Jasmine Dumas Christopher Lee.
Hypothesis Tests l Chapter 7 l 7.1 Developing Null and Alternative
Logic of Hypothesis Testing
Alison Burros, Kallie MacKay, Jennifer Hwee, & Dr. Mei-Ching Lien
Discussion & Conclusion
Step 1: Specify a null hypothesis
Academic Writing Skills
1University of Oklahoma 2Shaker Consulting
Theoretical issues Traits capture relatively stable individual differences. They are assumed to be relatively stable over time. They are also assumed to.
Kimron Shapiro & Frances Garrad-Cole The University of Wales, Bangor
Experimental Psychology
The Relationship Between Emphasis of Cell-phone Use on Performance and Anxiety: Classroom Implications Jordan Booth, Leah Cotton, Jeni Dillman, Kealey.
ERFCON th International Conference of the Faculty of Education and Rehabilitation Sciences University of Zagreb 17 – 19 May 2017, Zagreb SOCIAL COMPETENCE.
Background/Objective
Parental Status and Emergency Preparedness:
FUNDING ACKNOWLEDGEMENT
Hypothesis Testing and Confidence Intervals (Part 1): Using the Standard Normal Lecture 8 Justin Kern October 10 and 12, 2017.
SAINT LOUIS UNIVERSITY
Key research: Van Leeuwen et al
National Academy of Neuropsychology
The Intergenerational Transmission of Attachment:
Suboptimal Performance: When Do Methods & Mood Matter?
Associations Between Regional Brain Volumes and the NIH-Toolbox Cognition Battery Fluid and Crystallized Composite Scores in Cognitively Healthy Older.
University of South Alabama Neurobehavioural Associates
Evidence for gender bias in interpreting online professor ratings
Background/Objective
Self-discrepancies in the Social Role of Mother: Associations between Self-discrepancies and Negative Affect Nicole J. Holmberg, Laura D. Pittman, Emily.
Background/Objective
12 Inferential Analysis.
Theoretical issues Traits capture relatively stable individual differences. Traits are assumed to be relatively stable over time. Traits are also assumed.
Method Separate subheadings for participants, materials, and procedure (3 marks in total) Participants (1 mark) Include all info provided in the assignment.
A 4 Step Process (Kind of…)
Clinical Assessment and Diagnosis
12 Inferential Analysis.
Introduction to Hypothesis Testing
Inferential Statistics
Chapter 10: Intelligence & Testing
Chapter 9 Hypothesis Testing: Single Population
COMPARING VARIABLES OF ORDINAL OR DICHOTOMOUS SCALES: SPEARMAN RANK- ORDER, POINT-BISERIAL, AND BISERIAL CORRELATIONS.
ANalysis Of VAriance Lecture 1 Sections: 12.1 – 12.2
MGS 3100 Business Analysis Regression Feb 18, 2016
Evidence Based Diagnosis
Presentation transcript:

Bowden, Shores, & Mathias (2006): Failure to Replicate or Just Failure to Notice. Does Effort Still Account for More Variance in Neuropsychological Test Scores than TBI Severity? Martin L. Rohling George J. Demakis UNIVERSITY OF SOUTH ALABAMA - MOBILE UNIVERSITY OF NORTH CAROLINA - CHARLOTTE INTRODUCTION Green et al. (2001) claimed that effort accounts for more variance in test scores than does severity of traumatic brain injury (TBI). In fact, they reported a correlation of .73 between a composite score of 5 effort measures and a composite score of 43 cognitive measures (i.e., Overall Test Battery Mean or OTBM). Furthermore, they stated that effort accounted for 53% of the variance in cognitive test scores, which was over 25 times the variance accounted for by TBI severity. Finally, they noted that failing an effort test suppressed the OTBM by 1.20 SDs as compared to those passing effort tests. Bowden et al. (2006) claimed to have failed to replicate Green et al. (2001). Bowden et al. suggested that the validity of the WMT, and cognitive symptom validity tests (SVT) in general, is in doubt. They cautioned clinicians against making diagnoses based on scores from such tests, particularly the WMT. They concluded that SVTs like the WMT are no different from other memory tests and do not assess a unique construct that has been referred to as “effort.” In fact, Bowden et al. (2006) stated that: “…the correlations between WMT scores and measures of intelligence or overall test battery performance (OTBM: Green et al., 2001) may simply indicate that all of the WMT scores measure verbal memory (pg. 868).” Because of obvious differences between these 2 studies & the very different conclusions drawn by the two author groups, we re-examined results of Green et al. (2001) & Bowden et al. (2006) to see what might have caused the two sets of authors to reach such disparate conclusions regarding the WMT. RESULTS & DISCUSSION With respect to cognition (i.e., OTBM), no effect of age in either sample. None expected due to age-corrected scores. Education had effect in both samples. Significance effect of PTA in Green sample but not Bowden sample. GCS had an effect in both samples. LOC had an effect in Green sample. Contrary to prediction, no effect of CT/MRI findings and OTBM in Green sample. Related to the WMT-IR, small effect in Green sample but no effect in Bowden sample. No effect of education in either sample. No effect of PTA in either sample. No effect of GCS in either sample. No effect of LOC in Green sample. Effect of CT/MRI findings opposite of prediction, with abnormalities associated with higher scores. Summarizing, TBI severity associated with poorer cognition but unrelated to WMT scoes. Across both samples, the majority of analyses found effect of variables known to impact cognition evident in these samples (i.e., 90%). However, 88% of these same analyses found no effect on WMT-IR. Results indicate that the WMT measures a construct other than cognition. Cognitive Test Score Variance Accounted for by Effort: We repeated the analysis of Green et al. (2001), which generated a r between WMT-IR and OTBM of .47 (p < .0001; Rho = .49). The same analyses of Bowden sample resulted in an r of .55 (p < .0001; Rho = .59). Thus, effort accounted for 23% to 33% of the variance in OTBM. Influence of TBI Severity on Effort Test Scores: Looking at the WMT-IR, in the Green sample, PTA accounted for only 1.7% of the variance and just .1% in the Bowden sample. GCS accounted for < .1% of the variance in Green sample and 2.0% in the Bowden sample, but in opposite direction. LOC in Green sample accounted for .5% of the variance and CT/MRI findings accounted for just .4% of the variance. Influence of TBI Severity on Cognitive Test Scores: In the Bowden sample, PTA accounted for 2.6% of the variance in OTBM scores. In the Green sample, PTA accounted for just .3% of the variance in OTBM scores. For the Bowden sample, GCS accounted for 15.2% of the variance in the OTBM. For the Green sample, GCS accounted for 3.2% of the variance in OTBM scores. Finally, for the Green sample, LOC accounted for 9.6% of the variance in OTBM scores, but CT/MRI abnormalities accounted for less than .1% of the variance in the OTBM scores. Failing WMT-IR suppressed the OTBM by -.86 (Green) and -1.06 (Bowden) SD units. Looking for an Interaction between Cognitive Ability, TBI Severity, & Effort For PTA in Bowden sample, main effect of repeated measure (p = .002), no effect of PTA, and no interaction. However, for in Green sample, there was a main effect of repeated measure (p = .001), small effect of PTA (p = .11), and an interaction (p = .002). Looking at GCS, in Bowden sample, main effect of repeated measure (p = .01), but no effect of GCS, and no interaction. However, for the Green sample, there was a main effect of repeated measures (p = .003), no effect of GCS, but a trend for an interaction (p = .30). In the Green sample with LOC, there was not a main effect of repeated measure, no effect of LOC, but the interaction was significant (p < .05). For CT/MRI findings, there was a main effect of repeated measure (p < .0001), a main effect of CT/MRI abnormalities (p = .007), and an interaction (p < .0001). Summarizing, the results with LOC and CT/MRI abnormalities are consistent with those obtained using PTA and GCS for the Green sample, indicating that patients with more severe TBIs performed relatively better on effort measures than on cognitive measures. Those with milder TBIs scored more poorly on effort measures than they did on cognitive measures. Such findings are inconsistent with the WMT being a measure of cognition. Thus, averaged across samples and measures, TBI severity accounted for 5% of the variance in OTBM scores, but none of the variance in WMT-IR scores. The variance accounted for by effort on the OTBM was estimated to be 25%. This means that effort accounted for 5.0 times the variance in cognitive test scores than TBI severity. This is similar to what was reported by Green et al. (2001), which was that effort accounted for 4.5 times the variance in cognitive test scores than TBI severity. We found more similarities than differences between the Green and Bowden samples. Our results support the construct validity of the WMT as an effort test. ABSTRACT Several studies have reported that TBI has a very small effect on test scores, as compared with the effect of poor effort on neuropsychological performance. As a consequence, many authors have concluded that effort needs to be measured routinely and that it is necessary to control for poor effort when measuring the effects of brain disease or injury on performance. Recently, however, Bowden et al. (2007) have challenged these notions. They argued that the Immediate Recognition subtest of the Word Memory Test (Green & Flaro, 2003), an effort measure, is simply another test of memory. In this study, we re-examine the data from Bowden et al. (2007) and Green et al. (2001) to identify differences between the two studies that might account for their contradictory conclusions. In both sets of data, reanalysis showed that effort explains approximately five times more of the variance in composite neuropsychological test scores than TBI severity. Moreover, scores on the Word Memory Test – Immediate Recognition (WMT-IR) were not found to correlate with major variables known to be measuring ability (e.g., years of education). These findings challenge the conclusions offered by Bowden et al. (2006). METHOD (continued) Differences with Respect to TBI Severity – After excluding non-TBI referrals from the Green sample, the comparison between the two samples found the Bowden sample to be more impaired with respect to GCS (d = -.57, p = .02) and PTA (d = -.45, p < .0001). Differences with Respect to Education – Green detailed each person’s years of education; whereas, Bowden classified patients (<12, = 12, >12). When examining just TBI referrals, Green’s sample was better educated than the Bowden sample (d = .47, p = .004). Differences with Respect to Effort – Examining just WMT-IR, the Green sample did better than the Bowden sample (M = 91% vs 88%, respectively, 21% vs. 29% failures). Procedure: Participants: Only TBI referrals and adults were included in both samples. No adjustments were made to the samples based on age, education, or TBI severity. Measures: Test scores were adjusted to be consistent with WAIS-R (1981) normative scores. VIQ, PIQ, & General Memory Index (GMI) were averaged into the Overall Test Battery Mean (OTBM). Hypotheses Bowden et al. (2006) claimed the WMT simply measured verbal memory. If this were the case, then variables that have been shown to effect scores on verbal memory measures should have similar associations to scores from the WMT. Such variables would include age, education, and severity of injury as indexed by the GCS, PTA, LOC, and brain abnormalities identified by CT or MRI scans. To investigate this point, comparisons were made between the effects of these variables on measures of cognitive ability, particularly memory, and measures of effort as operationally defined by the WMT. Alternatively, Differences might have been caused by differences in the populations sampled. Differences might have been caused by differences in methods of measurement. Differences might have been caused by differences in the interpretations made by the respective author groups despite the fact that the two samples revealed similar findings. RESULTS METHOD When attempting to replicate a particular study’s findings, it is important that the two datasets be as comparable as possible. The differences between the Green et al. (2001) and Bowden et al. (2006) studies, in terms of sampling, measurement techniques, and statistical procedures, were of a sufficient magnitude that we were concerned they might have prohibited the authors from coming to similar conclusions. Therefore, we thought it necessary to adjust the datasets by using similar sampling procedures, measurement techniques, and statistical analysis, such that comparisons between the two might be more meaningful. Listed below are the differences we considered relevant Differences between Studies in Measures of Effort – Green et al. used all 3 WMT effort measures plus 2 others; whereas, Bowden et al. used just the WMT-Immediate Recognition score. Differences between Studies in Selected Measures of Cognitive Ability – Green et al. used an overall test battery mean of 43 measures; whereas, Bowden et al. used only WAIS-based PIQ & WMS-based GMI. We generated an OTBM for Bowden that included 16 subtest scores and a similar composite for Green that only included 17 test scores. The included measures were from IQ and Memory tests only. Difference with Respect to Age – Green included only adults (>18 y/o); whereas 16% of the Bowden sample were children who were under the age of 18. After excluding children, the samples still differed with respect to age, with the Green sample being older (M = 39.8 vs 36.3, d = .30). Differences with Respect to Diagnoses – Green included many different neurological and psychiatric diagnoses; whereas, Bowden included only individuals who allegedly suffered from a TBI.