1. Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington, DC: June, 2011

2. Disclosure Dr. Echemendia is a paid consultant to: The National Hockey League Major League Soccer US Soccer Federation

3. Historical Background Neuropsychological assessment has been a key component in evaluating/managing TBI for decades. Neuropsychological evaluations have become a key tool in the evaluation of MTBI since mid 70s. Focus on functional vs. structural assessment. NP assessment of sports concussions began in mid 80s.

4. Background Sports Laboratory Assessment Model Pre-post injury evaluations Paper & Pencil Assessment Batteries Computerized Batteries

5. Psychometric Considerations Validity vs. Reliability Validity Does the test measure what it is supposed to measure? Construct Validity –Operationalization of the construct, measurement of the construct Cognitive tests DO NOT measure concussion! Assess cognitive domains that are believed to be affected by concussion Convergent/Divergent Validity Does the test correlate with other known measures of the construct? Does the test not correlate with domains that it is not supposed to measure Face Validity Face Validity Does the test appear to be measuring what it is supposed to measure

6. Psychometric Considerations Sensitivity vs. Specificity Sensitivity: Is the measure sensitive in detecting the construct being measured? Do concussed individuals score lower than non-concussed individuals? What proportion of concussed individuals are identified by the test? Positive Predictive Power What proportion of Non-concussed individuals are identified by the test? Negative Predictive Power

7. Abnormal NP ScoresNormal NP Scores 0 0 150 300 Concussed Normal 150

8. Abnormal NP ScoresNormal NP Scores 30/ 20% 150 300 Concussed Normal 150 120/ 80% 90/ 60% 60/ 40% True Positive False Negative False Positive True Negative

9. Psychometric Considerations Sensitivity vs. Specificity Specificity: Is the measure specifically measuring the identified construct and not others? Concussion vs ADHD, Depression, Anxiety, Fatigue, etc. Ideal: 100% Sensitivity – 100% Specificity Real: NOT! Sensitivity vs Specificity trade-off Typically as sensitivity increases, specificity decreases The user must decide on what side to err.

10. ROC Curve

11. Psychometric Considerations Reliability – Estimating Error Can we expect to get the same results when the test is given…. To the same person an infinite number of times? (Confidence Intervals) To the same person under different circumstances? (Sideline vs. Office) To the same person at different points in time? To the same person by different test administrators? In different/alternate forms?

12. Tests Used in Sports TBI….

13. Paper & Pencil Tests Selected from tests already in use and validated for assessing MTBI Test Requirements: Short Easily administered by non-NPs Easily scored Sensitive to changes in specified domain Acceptable psychometrics Alternate forms. Assess several cognitive domains. Typical Tests: HVLT. BVMT. Trails, Digit Span, SDMT, Verbal Fluency, LNS, Cancellation tests.

14. Paper & Pencil Tests Cont. Advantages: Direct observation of player behavior Direct monitoring of effort/motivation & task performance Superior assessment of memory Broad normative db with multiple samples. Disadvantages: Labor intensive (Expensive) Variability in standardized admin Less reliable assessment of info proc speed. Tests must be hand scored

15. Computerized Batteries Two approaches: Adapt existing P&P measures into computer platform Measure multiple cognitive functions (memory, proc speed, reaction time, etc.) Develop tests de novo for computers Limited assessment of domains (largely RT, Info Proc Speed) New Instruments are being developed rapidly Existing Instruments: ImPACT, ANAM Sports, HeadMinders CRI, CNS Vital Signs, CogSport.

16. Computerized Tests Cont. Advantages: Can be used in groups (Less costly) Standardized admininistration Immediate scoring/data capture Finer assessment of RT/Proc Speed (altho other problems) Multiple languages Theoretically infinite alternate forms

17. Computerized Tests Cont. Disadvantages: Less complete assessment of memory Loss of observational data Less control of effort/motivation Diminished ability to assess understanding/track task performance Computer-specific measurement issues Data loss

18. Hybrid Approaches Combine strengths of both P&P and Computerized tests Once adequate normative data have been established…. Computer for baseline testing Computer plus P&P post-injury

19. Sports MTBI specific issues Phase of Evaluation Baseline Acute – On Field Management Sub acute – Office/Training Room Return to Play/Work/School Long Term Follow-up

20. Sports MTBI specific issues Baseline Testing – Is it necessary/helpful? Theoretically designed to minimize error due to intra-individual comparison Does it do this? Does it increase sensitivity/specificity? Does it introduce additional sources of error? What about multiple post-injury tests? How often should it be repeated? Age dependent?

21. Sports MTBI specific issues Test – Retest Reliability Significant variability across studies What test-retest intervals should be used? Reliability estimates for multiple tests

22. Sports MTBI specific issues Practice Effects Content Practice vs. Procedural Use of Alternate Forms Reliability/Validity of Alternate Forms Are practice effects bad? How to control for / appropriately use practice effects? RCI, regression based estimates

23. Sports MTBI specific issues Motivation/Effort Unique aspect of sports was the assumption that players are motivated to perform well. Different motivation as different phases of assessment Baseline vs. Post-injury Individual vs. Group Identification of poor effort Test specific indices Outlier identification

24. User Qualifications Although not the responsibility of the FDA, clear delineation of user qualifications for administering and interpreting tests is critical. Failure to do so may lead to scope of practice challenges, medico-legal quandaries and poor practice.

25. Thank you!