1. Quantitative Medical Imaging FUNDING PROVIDED BY NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY September Monthly Call September 14, 2011 Andrew J. Buckler, MS Principal Investigator QI-Bench optimizing performance through characterization TM

2. Agenda Study Description file and Data Tagging (Gary) Lab Protocol (Gary, Patrick, and Andy) QC (Alden and Ganesh) Analysis status (Andy) Year Outlook September 14, 2011QI-Bench: Optimizing Performance through Characterization2

3. Study Description File and Data Tagging September 14, 20113QI-Bench: Optimizing Performance through Characterization

4. Lab Protocols 1.Specify a.Consider concepts, relations, or properties b.Decide if new or existing biomarker c.Specify clinical context (either as first for new biomarker or as additional for existing biomarker) d.Specify assay method (either as first for new biomarker or as additional for existing biomarker) 2.Formulate a.Infer testable hypothesis contributory for characterizing performance of an assay method for a clinical context for use b.Formulate data requirements to execute statistical analysis plans to prove testable hypotheses based on biomarker definition c.Develop queries based on data requirements d.Run queries to create study-specific Reference Data Sets 3.Execute a.Create ground truth or other reference annotation and markup b.Package algorithm or method using Batch Analysis Service API c.Initiate a run d.Sequence through tasks (potentially with help of Condor grid) e.Run interfaced algorithm on a data set 4.Analyze a.Develop analytical method to support testable hypothesis (e.g., in R scripts) b.Perform an analysis for a specific test (repeat across tests considered for group analysis) c.Annotate Biomarker DB with results for this test d.Perform an analysis to characterize performance for the group e.Annotate Biomarker DB with results of group statistic 5.Package a.Discuss qualification goal with regulator AND/OR discuss device/software clearance/approval goal with regulator b.Prepare qualification full data package AND/OR prepare 510(k) or PMA c.Approve submission (according to relevant pathway) September 14, 20114QI-Bench: Optimizing Performance through Characterization

5. Lab Protocols 2.Formulate d.Run queries to create study-specific Reference Data Sets 3.Execute a.Create ground truth or other reference annotation and markup b.Package algorithm or method using Batch Analysis Service API c.Initiate a run d.Sequence through tasks (potentially with help of Condor grid) e.Run interfaced algorithm on a data set 4.Analyze a.Develop analytical method to support testable hypothesis (e.g., in R scripts) b.Perform an analysis for a specific test (repeat across tests considered for group analysis) September 14, 20115QI-Bench: Optimizing Performance through Characterization

6. Seed Point QC demonstrator40 – 408 seeds (1A read 40) pilot3A (was demonstrator15) – 156 seeds (1A read 15) study1C (was demonstrator1C) – 2364 seeds (1C is set to read 66) study1187 (was demonstrator1187) – 7122 seeds Available: RIDER, IDRI, MSKCC “1B”, … September 14, 20116QI-Bench: Optimizing Performance through Characterization

7. Analyze Status: Metrology Definitions File System Engineering\User Needs\Definitive Metrology Terms for QI 2011-09-13.docxSystem Engineering\User Needs\Definitive Metrology Terms for QI 2011-09-13.docx September 14, 2011QI-Bench: Optimizing Performance through Characterization7

8. Analyze Status: Stepwise Plan Near-term 1.Process the lesion reads on the same 40 lesions used in the 1A pivotal as a 7 th reader using 1A STATA method and compare results. [QI-Bench 1A STATA]7 th reader1A STATA method 2.Extend the STATA analysis to the 408, publish this result as the first QI-Bench publication (after QC). Mid-term 3.Process the 6 selected lesions from the MVT demonstrator using the 1A STATA method and compare results. [MVT 1A STATA] 4.Process the lesion reads on the same 6 lesions used in the MVT demo set as a 7 th reader and compare results. [QI- Bench MVT] 5.Convert 1A STATA analysis to R and compare the results on the 408. [STATA R] 6.Extend MVT to use the created R scripts (and fill other gaps). 7.Re-do analyses to verify that results come out the same. Perform and analyze other studies (e.g., 1C, 3A, 1187, other modes, etc. using STATA analysis method. Longer- term Convert to R-based, MVT-implemented analysis. September 14, 2011QI-Bench: Optimizing Performance through Characterization8

9. Analyze Status: MVT development Lesion tracking Other modalities and measures, e.g., SUV via FDG-PET Properly functioning multiple regression and N-way ANOVA Support Clinical Performance assessment (i.e., in addition to current Technical Performance) – Outcome studies – Integrated genomic/proteomic correlation studies – Group studies for biomarker qualification Serial studies / change analysis Persistent database Scale-up to handle thousands of cases (10’s thousands of lesions) Deploy as Web app September 14, 2011QI-Bench: Optimizing Performance through Characterization9

10. Year Outlook Time frame Content September STATA Analysis (Analyze) QI-Bench landing page (all user apps plus developer environment) New Mac and Unix servers October “Image Formation”: scripts and spreadsheet for provenance of Reference Data Sets Laboratory Protocol (Formulate, Execute, Analyze) November Apply QI-Bench style sheets to apps Support change analysis / serial studies (up to two time points, extensible to additional in subsequent development iterations) (Formulate, Execute, Analyze) December- January Specify working model (including QIB Ontology and AIM template builder 18 or 23 as possible) Document and solidify the API harness for execution modules of the Batch Analysis Service (Execute) February Drive reading stations by making worklist items from within Batchmake scripts (Formulate, Execute) March Support DICOM Query/Retrieve from Reference Data Set Manager (to complete the Reader Studies support package) (Execute, Analyze) April-May Extend harness API for AIM templates (including generate output from the LSTK module via AIM template) (Execute) June Formulate working model (caB2B-based, including re-worked NBIA Connector and other content as above) September 14, 2011QI-Bench: Optimizing Performance through Characterization10

11. Next Steps QC demonstrator40 and process the lesion reads on the same 40 lesions used in the 1A pivotal as a 7th reader using 1A STATA method and compare results. [QI-Bench 1A STATA] Extend the STATA analysis to the 408, publish this result as the first QI-Bench publication (after QC). New Mac and native-Unix servers QI-Bench landing page (all user apps plus developer environment) QC and re-tag pilot3A, study1C, and study1187 Will send a Doodle to find a new meeting time that works better for West coast participants, still good for Europe. September 14, 201111QI-Bench: Optimizing Performance through Characterization