Designing a Battery for Clinical Trials: the MATRICS Project for Schizophrenia Richard Keefe, PhD Department of Psychiatry and Behavioral Sciences Duke University Medical Center NeuroCog Trials, Inc. Durham, North Carolina ASENT February 21, 2014 Special Thanks to Michael Green, Bob Kern and Keith Nuechterlein

FDA registration targets DSM disorders “No fundamental objection to syndrome-based clinical targets (fever, pain, agitation)” “We will not accept a new clinical endpoint for the convenience of any drug company” NIMH can use its convening authority as independent scientific entity to define new and valid clinical endpoints Slide from W. Fenton

NIMH – MATRICS (Measurement and Treatment Research to Improve Cognition in Schizophrenia) Goals and Products Create Standardized Measure for use in Clinical Trials Define Optimal Experimental Designs Establish path to FDA Approval Attract large pharmaceutical companies to focus efforts on this important clinical target Required involvement of: NIMH, FDA, pharmaceutical industry, and academia

1)A focus on level of cognitive domains versus global composite scores; 2)Neutrality on the relative importance of computerization of the battery; 3)Inclusion of the relatively new domain of social cognition in the MCCB; 4)Neutrality on inclusion of tests from clinical neuropsychology vs cognitive psychology and cognitive neuroscience; 5)Clear emphasis on psychometrics and tolerability/practicality; 6)Emphasis on norms / co-norms MATRICS Critical Decisions for Measurement

How important is it for the selected battery to reliably assess the major separable cognitive factors in schizophrenia? Importance of Assessment at Domain Level Kern et al. 2004

Importance of Computerized Assessment Do you favor a battery that involves paper-and-pencil or computerized test administration? Kern et al. 2004

CATIE Baseline Neurocognition Completion Rates Percentage of Cases Keefe et al., 2006; total n=1427 Computerized tests

1)A focus on level of cognitive domains versus global composite scores; 2)Neutrality on the relative importance of computerization of the battery; 3)Inclusion of the relatively new domain of social cognition in the MCCB; 4)Neutrality on inclusion of tests from clinical neuropsychology vs cognitive psychology and cognitive neuroscience; 5)Clear emphasis on psychometrics and tolerability/practicality; 6)Emphasis on norms / co-norms MATRICS Critical Decisions for Measurement

Steps to MATRICS Consensus Cognitive Battery Subgroup of NCC* & survey of experts NCC, based on survey of experts Survey of experts NCC MATRICS Team RAND Panelists NCC, based on ratings of Panelists *NCC: MATRICS Neurocognition Committee **PASS: MATRICS Psychometric and Standardization Study 1. Identify cognitive domains 2. Select key criteria for test selection 3. Solicit nominations for cognitive tests 4. Narrow tests to 6 or less per domain 5. Create data base on criteria for candidate tests 6. Evaluate tests on criteria with RAND Method 7. Select 2-5 tests per domain for beta battery PASS** group NCC and PASS group 8. Psychometric study with beta battery 9. Final battery of 1-3 tests per domain 10. Co-norming of tests on community sample PASS group

RAND / UCLA Appropriateness Method Method established in the 1980s to enable the measurement of the overuse and underuse of medical and surgical procedures “appropriateness” refers to relative weight of benefits vs. harms associated with a medical or surgical intervention Data from randomized clinical trials often not available for multitude of conditions seen in patients in everyday clinical practices Method needed that combined best available scientific evidence with the collective judgment of experts to yield a statement concerning the appropriateness of a procedure Since then it has been applied more broadly to help increase agreement among persons representing differing stakeholder groups Process involves creating a panel of experts (i.e., key stakeholders) Conducting a literature review and creating a database of the best available scientific evidence Expert panel performs ratings in two rounds: 1 st round – no interaction 2 nd round – in-person meeting

MATRICS RAND Panel Process A database was created that included data on the five selection criteria from published and unpublished studies for each test (July – August, 2003) A RAND panel of experts was assembled Cognitive deficits in schizophrenia Clinical neuropsychology Clinical trials methodology Cognitive science Neuropharmacology Clinical psychiatry Biostatistics and psychometrics

MATRICS RAND Panel Process The RAND panelists reviewed the database and rated the 36 tests on each selection criteria prior to the in-person meeting These preconference ratings (n = 180) were reviewed by the MATRICS team to identify any that reflected a lack of consensus At the in-person meeting, the expert panel discussed each of the ratings showing lack of consensus (n = 20) and rated them again (September, 2003) Dispersion decreased with nine median ratings changing

Steps to MATRICS Consensus Cognitive Battery Subgroup of NCC* & survey of experts NCC, based on survey of experts Survey of experts NCC MATRICS Team RAND Panelists NCC, based on ratings of Panelists *NCC: MATRICS Neurocognition Committee **PASS: MATRICS Psychometric and Standardization Study 1. Identify cognitive domains 2. Select key criteria for test selection 3. Solicit nominations for cognitive tests 4. Narrow tests to 6 or less per domain 5. Create data base on criteria for candidate tests 6. Evaluate tests on criteria with RAND Method 7. Select 2-5 tests per domain for beta battery PASS** group NCC and PASS group 8. Psychometric study with beta battery 9. Final battery of 1-3 tests per domain 10. Co-norming of tests on community sample PASS group

Selection of Beta Battery Based on the results from RAND panel review, the Neurocognition Committee selected 20 tests Two to five tests were selected per cognitive domain These 20 tests comprised the beta battery and moved forward for evaluation in a prospective study with persons with schizophrenia and schizoaffective disorder (Psychometric And Standardization Study; PASS)

Steps to MATRICS Consensus Cognitive Battery Subgroup of NCC* & survey of experts NCC, based on survey of experts Survey of experts NCC MATRICS Team RAND Panelists NCC, based on ratings of Panelists *NCC: MATRICS Neurocognition Committee **PASS: MATRICS Psychometric and Standardization Study 1. Identify cognitive domains 2. Select key criteria for test selection 3. Solicit nominations for cognitive tests 4. Narrow tests to 6 or less per domain 5. Create data base on criteria for candidate tests 6. Evaluate tests on criteria with RAND Method 7. Select 2-5 tests per domain for beta battery PASS** group NCC and PASS group 8. Psychometric study with beta battery 9. Final battery of 1-3 tests per domain 10. Co-norming of tests on community sample PASS group

Psychometric and Standardization Study PASS: Phase I Aim: To collect psychometric data for beta version of the battery -- provide basis for selection of final battery Individual tests from beta battery were evaluated on previously identified selection criteria: Test-retest reliability Repeatability (e.g., practice effects) Relationship to social and work functioning, independent living Practicality and tolerability 176 schizophrenia/schizoaffective disorder patients were tested at baseline; 167 were available for follow-up at 4 weeks

PASS Phase I and II Research Sites Sites Duke University Duke University Harvard University Harvard University University of Kansas University of Kansas University of Maryland University of Maryland UCLA UCLA

Steps to MATRICS Consensus Cognitive Battery Subgroup of NCC* & survey of experts NCC, based on survey of experts Survey of experts NCC MATRICS Team RAND Panelists NCC, based on ratings of Panelists *NCC: MATRICS Neurocognition Committee **PASS: MATRICS Psychometric and Standardization Study 1. Identify cognitive domains 2. Select key criteria for test selection 3. Solicit nominations for cognitive tests 4. Narrow tests to 6 or less per domain 5. Create data base on criteria for candidate tests 6. Evaluate tests on criteria with RAND Method 7. Select 2-5 tests per domain for beta battery PASS** group NCC and PASS group 8. Psychometric study with beta battery 9. Final battery of 1-3 tests per domain 10. Co-norming of tests on community sample PASS group

Test- Retest (r) Test- Retest (ICC) WAIS-III Block Design BACS Tower of London NAB Mazes0.83 Reasoning and Problem Solving - Test-retest Reliability -

T 1 Mean (SD) T 2 Mean (SD) # of Scores at Low / High Limit t-test, p-value Effect Size (d) WAIS-III Block Design 29.4 (13.3) 32.5 (13.7)1/ (p<.0001) 0.22 BACS Tower of London 12.7 (5.1) 13.6 (5.2)4/ (p=.01) 0.17 NAB Mazes 11.8 (6.9) 12.4 (7.1)11/ (p=.07) 0.08 Reasoning and Problem Solving - Utility as a Repeated Measure -

Global (r) Social (r) Independent Living (r)Work (r) WAIS-III Block Design 0.10 (0.08) 0.01 (0.04) (-0.18) 0.19 (0.18) BACS Tower of London 0.25 (0.27) 0.11 (0.15) 0.13 (0.22) 0.20 (0.20) NAB Mazes 0.15 (0.18) 0.12 (0.18) (-0.05) 0.21 (0.17) Reasoning and Problem Solving - Relationship to Functional Outcome – overall r (median r across sites)

Practicality Mean (SD) Tolerability Mean (SD) Admin Time Mean (SD ) Missing Data N (%) Block Design 5.7 (1.2) 4.6 (1.7)11.7 (3.2) 1 (0.6%) Tower of London 5.6 (1.2) 5.0 (1.4) 6.9 (1.7) 6 (3.4%) NAB Mazes 6.2 (0.8) 4.6 (1.6) 11.2 (3.2) 2 (1.1%) Reasoning and Problem Solving - Practicality* and Tolerability* - *1 to 7 Likert scale

Test Rank WAIS-III Block Design BACS Tower of London NAB Mazes Reasoning and Problem Solving - Ranks by Neurocognition Committee

MATRICS Consensus Cognitive Battery Final version Speed of Processing Category Fluency BACS Symbol Coding Trial Making A Attention / Vigilance Continuous Performance Test - Identical Pairs version Working Memory Maryland Letter Number Span WMS-III Spatial Span Verbal Learning Hopkins Verbal Learning Test-R Visual Learning Brief Visuospatial Memory Test-R Reasoning and Problem Solving NAB Mazes Social Cognition MSCEIT Managing Emotions

Steps to MATRICS Consensus Cognitive Battery Subgroup of NCC* & survey of experts NCC, based on survey of experts Survey of experts NCC MATRICS Team RAND Panelists NCC, based on ratings of Panelists *NCC: MATRICS Neurocognition Committee **PASS: MATRICS Psychometric and Standardization Study 1. Identify cognitive domains 2. Select key criteria for test selection 3. Solicit nominations for cognitive tests 4. Narrow tests to 6 or less per domain 5. Create data base on criteria for candidate tests 6. Evaluate tests on criteria with RAND Method 7. Select 2-5 tests per domain for beta battery PASS** group NCC and PASS group 8. Psychometric study with beta battery 9. Final battery of 1-3 tests per domain 10. Co-norming of tests on community sample PASS group

Psychometric and Standardization Study PASS: Phase II Aim: To provide co-norms for the final MATRICS battery so that results can be standardized Same five research sites The final MATRICS battery was administered to 300 adults from the general community with representative demographic characteristics of the US population The results provided the basis for the MATRICS scoring program that provides age-, gender-, and education-corrected scores (T-scores, percentiles)

Why Collect Community Normative Data for the MCCB? Puts all tests on common metric (based on mean & SD of stratified community sample) Improves ability to detect “signal” in clinical trials -- reduces error variance due to bad scaling Enables valid composite scores (e.g., overall score) Enables valid comparisons between cognitive domains (to detect domain-specific effects) Provides basis for age, gender, and education corrections Additional norming needs for international trials

MCCB – What we Know After 5 Years 1.Reliability 2.Practice effects 3.Sensitivity to training interventions 4.Sensitivity to novel psychopharm interventions 5.Sensitivity to anticholinergic load 6.Sensitivity to antipsychotic medication adherence 28

MCCB Composite T-Score Test-retest Reliability is Extremely High Domain Memory/ Roche (ICC) Lurasidone (ICC) TURNS MK-0777 (Pearson r) TURNS AL-108 (ICC) Sanofi- Connect (ICC) Composite T-score Speed of processing Attention/Vigilance Working memory Verbal learning Visual learning Reasoning/Problem solving Social cognition Keefe et al. Schizophrenia Res 2011; Buchanan et al, Biological Psychiatry, 2011; Javitt et al, Schizophrenia Res, 2012; Umbricht et al, submitted

Test-Retest MCCB Composite T-Score (N=318) ICC = 0.88 Pearson’s r = 0.89 Screening Composite T-Score Baseline Composite T-Score Least Squares Fit Between screening and baseline visit (days): ● 2-5, O 6-14, □ 15-25, ◊ 26-32, ♦ Effect of time elapsed between screening and baseline visits: NS (p>0.4, interaction test for slope in least squares regression). Keefe et al, Schiz Res, in press

MCCB Practice Effects are Modest: Baseline to First Follow-up Domain Memory/ Roche (d) DSP Lurasidone v. Risperidone (d) TURNS MK-0777 (d) Sanofi- Aventis CONNECT (d) Composite T-score Speed of processing Attention/Vigilance Working memory Verbal learning Visual learning Reasoning/Problem solving Social cognition Keefe et al. Schizophrenia Res 2011; Buchanan et al, Biological Psychiatry, 2011; Javitt et al, Schizophrenia Res, 2012; Umbricht et al, submitted Confidential Effect sizes are measured in Cohen’s d

Fisher, et al. Am J. Psychiatry 2009 Using Neuroplasticity-Based Auditory Training to Improve Cognition in Schizophrenia

* * # *p<.05; # p<.10 T-score Treatment Effects of Cognitive Remediation 20 Sessions 20 Sessions (CRSTN study) Keefe et al. J Clin Psychiat. 2012; The sig. effect on global score became trend after 40 sessions with fewer subjects

Cognitive Training on MCCB Overall Composite Score Covarying for antipsychotic medication adherence (n = 46) Group X Time interaction, p =.025 Nuechterlein et al. SRP 2012, ICOSR 2013

35 EnVivo Compound versus Placebo: Cognition Endpoint (MCCB overall comosite ) 1.0 mg vs. Placebo: P = ES = 0.28 EVP mg EVP mg Placebo

Envivo compound – Effects on MCCB Global score are stronger in younger subjects 36 Confidential

37 Randomized, Double-Blind, Placebo-Controlled Add-on Treatment of Benzoate, a D-Amino Acid Oxidase Inhibitor, for Schizophrenia Lane et al. in press JAMA-Psychiatry MCCB Neurocognitive composite (Chinese) Placebo BL = 48.9 (9.6); Endpoint = 52.8 (9.6) Benzoate BL = 51.3 (10.5); Endpoint = 58.9 (12.8) p value = 0.035; between-group effect size 0.67

Serum anticholinergic activity accounted for 20% of the variance in global cognition change, independent of age, IQ, or symptom severity. Vinogradov et al Serum Anticholinergic Load and MCCB Response to Cognitive Training

Correlations between Antipsychotic Medication Adherence and MCCB Gains in First-Episode Schizophrenia (n = 57) * p <.05; + p <.10 Nuechterlein et al. SRP 2012, ICOSR 2013

MCCB – What We Know After 5 Years 1.Reliability – Extremely high test-retest across all studies 2.Practice effects – Small across all studies 3.Sensitivity to training interventions – Significant in at least 3 separate studies 4.Sensitivity to novel psychopharm interventions – Significant in at least 3 separate studies (2 of which we have permission to show in this slide set). 5.Sensitivity to anticholinergic load 6.Sensitivity to antipsychotic medication adherence – Across multiple cognitive domains 40

Steps in Translation and Cultural Adaptation of the MATRICS Consensus Cognitive Battery 41 Nuechterlein, MATRICS-CT Meeting, Bethesda, October 27, 2009 MAI attorney and seven test IP owners MAI and IP owners Professional Translators MAI and Professional Translators Experts arranged by MAI MAI = MATRICS Assessment, Inc. 1. Legal permission to translate 3. Forward translation (2) 4. Reconciliation 5. Back translation (2) 6. Iterative revision and harmonization 7.Review by end language cognitive testing professional Test IP owners Professionals arranged by MAI in each language 9. Testing of schizophrenia patients 10. Page composition and printing MAI working with a page compositor and a printer 8. Review and approval by IP owners 2. Concept and style sheets

MCCB Translation Availability 42