Measurement Science and Engineering FUNDING PROVIDED BY NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY August Monthly Call August 10, 2011 Andrew J. Buckler, MS Principal Investigator QI-Bench optimizing performance through characterization TM

August 10, 2011QI-Bench: Optimizing Performance through Characterization2 Specify context for use and assay methods. Use consensus terms in doing so. Specify Assemble applicable reference data sets. Include both imaging and non-imaging clinical data. Formulate Compose and iterate batch analyses on reference data. Accumulate quantitative read-outs for analysis. Execute Characterize the method relative to intended use. Apply the existing tools and/or extend them. Analyze Compile evidence for regulatory filings. Use standards in transfer to regulatory agencies. Package QIBO, RadLex/ Snomed/ NCIt built using Ruby on Rails caB2B, NBIA, PODS data elements, DICOM query tools MIDAS, BatchMake, Condor Grid. Built using CakePHP MVT portion of AVT, re- useable library of R scripts STDM standard of CDISC into repositories like FDA’s Janus

User Requirements -> Activities -> Lab Protocols 1.Specify a.Consider concepts, relations, or properties b.Decide if new or existing biomarker c.Specify clinical context (either as first for new biomarker or as additional for existing biomarker) d.Specify assay method (either as first for new biomarker or as additional for existing biomarker) 2.Formulate a.Infer testable hypothesis contributory for characterizing performance of an assay method for a clinical context for use b.Formulate data requirements to execute statistical analysis plans to prove testable hypotheses based on biomarker definition c.Develop queries based on data requirements d.Run queries to create study-specific Reference Data Sets 3.Execute a.Create ground truth or other reference annotation and markup b.Package algorithm or method using Batch Analysis Service API c.Initiate a run d.Sequence through tasks (potentially with help of Condor grid) e.Run interfaced algorithm on a data set 4.Analyze a.Develop analytical method to support testable hypothesis (e.g., in R scripts) b.Perform an analysis for a specific test (repeat across tests considered for group analysis) c.Annotate Biomarker DB with results for this test d.Perform an analysis to characterize performance for the group e.Annotate Biomarker DB with results of group statistic 5.Package a.Discuss qualification goal with regulator AND/OR discuss device/software clearance/approval goal with regulator b.Prepare qualification full data package AND/OR prepare 510(k) or PMA c.Approve submission (according to relevant pathway) August 10, 2011QI-Bench: Optimizing Performance through Characterization3

User Requirements -> Activities -> Lab Protocols 2.Formulate d.Run queries to create study-specific Reference Data Sets 3.Execute a.Create ground truth or other reference annotation and markup b.Package algorithm or method using Batch Analysis Service API c.Initiate a run d.Sequence through tasks (potentially with help of Condor grid) e.Run interfaced algorithm on a data set 4.Analyze a.Develop analytical method to support testable hypothesis (e.g., in R scripts) b.Perform an analysis for a specific test (repeat across tests considered for group analysis) August 10, 2011QI-Bench: Optimizing Performance through Characterization4

PickLists August 10, 2011QI-Bench: Optimizing Performance through Characterization5

ImageFormation August 10, 2011QI-Bench: Optimizing Performance through Characterization6

LesionSeeding August 10, 2011QI-Bench: Optimizing Performance through Characterization7 demonstrator15 – 156 seeds (1A read 15) demonstrator40 – 408 seeds (1A read 40) demonstrator1C – 2364 seeds (1C is set to read 66) demonstrator1187 – 7122 seeds

PhantomTruthing August 10, 2011QI-Bench: Optimizing Performance through Characterization8

Demonstrator Reference Data Sets and Number of Lesions for Evaluation demonstrator15 – 156 seeds (1A read 15) demonstrator40 – 408 seeds (1A read 40) demonstrator1C – 2364 seeds (1C is set to read 66) demonstrator1187 – 7122 seeds Available: RIDER, IDRI, MSKCC “1B”, … August 10, 2011QI-Bench: Optimizing Performance through Characterization9

GOTO ON-SCREEN DEMOS August 10, 2011QI-Bench: Optimizing Performance through Characterization10

Load Cases and Run MVT (measurement variability tool)

AIM output for selected case (from Batch Analysis Service)

Choose Analysis Type (we’ll be adding new Types)

Ground truth can be direct or indirect

Deselect RECIST and WHO measurements Choose Biomarker Measurements to Analyze (we’ll be adding Measurements)

R Script for summary statistic Some Methods exist; we will create new ones (e.g., factor analysis)

Column per measurement Columns for calculation according to the Methods

Intra- and Inter-reader comparisons Output of Analysis Type run (this example intra-reader)

Inter-reader measurements Output of Analysis Type run (this example inter-reader)

… calculated Additional outputs based on Methods

Outliers list Outlier Analysis with Drill-down is Supported (access to raw imagery)

Configure plots as needed Resulting plots – we’ll be adding plot types, e.g., box, radar, and K-M

Focus of FY12 Grant Year Specify – Integrate QIBO, Profiles, and AIM Template – Populate Biomarker DB with first biomarkers Formulate – Integrate NBIA Connector, NIST DICOM-based query, and caB2B Execute – Support black-box API harness – Support reader studies (i.e., in addition to current fully-automated reads) – Support FDG-PET August 10, 2011QI-Bench: Optimizing Performance through Characterization23 Analyze – Support Clinical Performance assessment (i.e., in addition to current Technical Performance) – Scale up Package – Converge imaging and device standards in SDTM

August 10, 2011QI-Bench: Optimizing Performance through Characterization24 QI-Bench optimizing performance through characterization TM