1. Program Update October 11, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench WITH FUNDING SUPPORT PROVIDED BY NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

2. Agenda Summary and close-out of FY12 development iteration – Covering what’s been accomplished from multiple points of view FY13 development iteration – Deployment progress and support – Continued Progress on ISA files – Architecture – Contour-based analysis 22

3. 3 Autumn 2012 (n=54) FY 2012 (n=110) Winter 2013 (n=15) In Queue (n=6) 333 Establish overall structure Initial Specify, Formulate, and Iterate Substantial work in Execute and Analyze First work on test-beds Establish overall structure Initial Specify, Formulate, and Iterate Substantial work in Execute and Analyze First work on test-beds V&V issues in Execute Substantial work in Analyze library Deployment support Advance test-beds V&V issues in Execute Substantial work in Analyze library Deployment support Advance test-beds Introduce radiologist workstation component, including scripted reader studies First real implementation of Biomarker KB triple store Introduce radiologist workstation component, including scripted reader studies First real implementation of Biomarker KB triple store W3C-compliance in Specify Formulate using SPARQL Full realization of QI-Bench cohesive architecture Full realization of worked example test-bed W3C-compliance in Specify Formulate using SPARQL Full realization of QI-Bench cohesive architecture Full realization of worked example test-bed

4. Lab Protocol Develop and run queries based on data requirements – Use of Formulate Use of Formulate Load Reference Data into the Reference Data Set Manager – Example Pilot3A Data Processing Steps Example Pilot3A Data Processing Steps Server-Side Processing using the Batch Analysis Service – Package Algorithm or Method using Batch Analysis Service API Package Algorithm or Method using Batch Analysis Service API – Prepare Data Set Prepare Data Set Create Ground Truth or other Reference Annotation and Markup Create Ground Truth or other Reference Annotation and Markup Importing location points and other data for use Importing location points and other data for use – Writing Scripts Writing Scripts – Initiate a Batch Analysis Run Initiate a Batch Analysis Run Perform statistical analysis – Analyze Use Instructions Analyze Use Instructions Design Documents User Needs and Requirements Analysis Architecture Application-specific Design – Specify "Specify" Scope Description (ASD) "Specify" Architecture Specification (AAS) "Quantitative Imaging Biomarker Ontology (QIBO)" Software Design Document (SDD) "Quantitative Imaging Biomarker Ontology (QIBO)" Software Design Document (SDD) "Biomarker DB" (a.k.a., the triple store) Software Design Document (SDD) "Biomarker DB" (a.k.a., the triple store) Software Design Document (SDD) AIM Template Builder Design Documentation: – Formulate "Formulate" Scope Description (ASD) "Formulate" Architecture Specification (AAS) "NBIA Connector" Software Design Document (SDD) "NBIA Connector" Software Design Document (SDD) – Execute "Execute" Scope Description (ASD) "Execute" Architecture Specification (AAS) Reference Data Set Manager (RDSM) Software Design Document (SDD) Reference Data Set Manager (RDSM) Software Design Document (SDD) Batch Analysis Service Software Design Document (SDD) Batch Analysis Service Software Design Document (SDD) – Analyze "Analyze" Scope Description (ASD) "Analyze" Architecture Specification (AAS) – Package "Package" Scope Description (ASD) "Package" Architecture Specification (AAS) 4444 V&V QI-Bench Project Management Plan (PMP) QI-Bench Project Management Plan (PMP) Traceability Report QI-Bench Verification and Validation Plan QI-Bench Verification and Validation Plan QI-Bench Iteration 1 Validation Report QI-Bench Iteration 1 Validation Report QI-Bench Iteration 1 Verification Protocol QI-Bench Iteration 1 Verification Protocol QI-Bench Iteration 1 Verification Report QI-Bench Iteration 1 Verification Report Application Test Protocols, Reports, and Records: – Specify: from AIMfrom AIM – Formulate: from caB2Bfrom caB2B – Execute – From MIDAS From MIDAS – RDSM Integration Test Report RDSM Integration Test Report – Analyze – From AVT From AVT – Library Integration Test Report Library Integration Test Report – Iterate: from Tavernafrom Taverna

5. Demonstrator 15, 40, 3A Pilot and Pivotal, and Change Analysis … Investigation 1: Pilot and pivotal study are finished FDA, M. A Gavrielides et al., “A resource for the Assessment of lung nodule size estimation methods: database of thoracic CT scans of an anthropomorphic phantom”, Optics Express, vol. 18, n.14, pp. 15244- 15255, 2010. Phantom data, FDA, NIST, QI-Bench Fig. 1: Radial plot showing comparative performance on the selected descriptive statistics as well as mean of absolute percent errors. Challenge Definition: estimate absolute volumes in phantom data Explicitly indicate descriptive statistics: bias, variance. Null hypothesis: analysis software model does not have a significant effect on the bias and variance. PILOT STUDY PIVOTAL STUDY 10 participants who measured 408 nodules 12 participants who measured 97 nodules 5555

6. 6666

7. 7777

8. FY13 Development Iteration Deployment progress and support Continued Progress on ISA files Architecture Contour analysis (purpose, methods, and file formats) 88888

9. Continued Progress on ISA Files 99999 Assay (a_) and Study (s_) levels: – _dcm.csv: SUBJID, TPINDEX, SITE, ACQREP, SERIESTYPE, – _loc.csv: SUBJID, TPINDEX, ACQREP, LOCRDR, LOCTOOL, LOCREP, TARGET, locX, locY, locZ, bbX0, bbX1, bbY0, bbY1, bbZ0, bbZ1 – _rdg.csv: SUBJID, TPINDEX, ACQREP, LOCRDR, LOCTOOL, LOCREP, TARGET, SEGRDR, SEGTOOL, SEGREP, SERIESTYPE, READING, – _cov.csv: SUBJID, AGE, GENDER, HEIGHT, WEIGHT, RACE, – _chg.csv: SUBJID, TARGET, TPINDEX1ST, ACQREP1ST, VALUE1ST, TPINDEX2ND, ACQREP2ND, VALUE2ND, ARITHDIFF, PCTDIFF, PPNDIFF, – _dx.csv: SUBJID, TARGET, deltaX, X, SOURCE – _mo.csv: TYPE, INSTANCE, VALUE, MODULE Investigation (i_) level: – Works in progress, but concept is serialized triple store roll-up including aggregation analyses such as aggregate uncertainty

10. data services (e.g., MIDAS, NBIA, etc.) data services (e.g., MIDAS, NBIA, etc.) RDSM for Images and ISA files QI-Bench Stack Modified XIP Host Hibernate Applications (a generic QI-Bench template, as well as specific configurations) Cached objects: AIM/DICOM, etc Data Access Layer Web GUI High level features: GWT (or Tapestry) UI ; both desktop and web client versions RESTful service layer; need to work out details between Hibernate and Jena Implemented according to open source best practices; In such a way as to enable the enhancement roadmap; and Integrated with projects driving advanced semantics and support for regulatory e- submissions Ontologies and vocabularies Ontologies and vocabularies RDF triple store for Patient info, annotations, Collections Experiments Jena 10 XIP LIB R LIB Taverna

11. Applications (a generic QI-Bench template, as well as specific configurations) Cached objects: AIM/DICOM, etc Data Access Layer Cohort applications: Specify Formulate Execute’s RDSM and BAS Analyze/Iterate (may combine) 11 XIP LIB R LIB Taverna

12. Applications (a generic QI-Bench template, as well as specific configurations) Cached objects: AIM/DICOM, etc Data Access Layer Individual Subject (Patient) Workstation Plug-in to We provide: wrapper for ClearCanvas as example and template Data access layer: Unified worklist transactions Support for Q/R to RDSM Support for access to Biomarker KB Taverna-desktop level of capability for workflows 12 XIP LIB R LIB Taverna C++ layer, for leverage of components in C++ and support of ClearCanvas? Java core, for consistency across QI- Bench?

13. data services (e.g., MIDAS, NBIA, etc.) data services (e.g., MIDAS, NBIA, etc.) RDSM for Images and ISA files Modified XIP Host Hibernate GUI Local Host (“workstation”) configuration (thick client) Ontologies and vocabularies Ontologies and vocabularies RDF triple store for Patient info, annotations, Collections Experiments Jena 13 local remote

14. data services (e.g., MIDAS, NBIA, etc.) data services (e.g., MIDAS, NBIA, etc.) RDSM for Images and ISA files Modified XIP Host Hibernate Web GUI Web-based (thin client) Ontologies and vocabularies Ontologies and vocabularies RDF triple store for Patient info, annotations, Collections Experiments Jena 14 local remote

15. Early ideas on Technical Approach What – Re-architect UI for Analyze application – Interface XIP Host Services through REST API – Retrieve AIM data and pass to Analyze – Present analysis data in new web UI – Interface with all AIM and DICOM data through XIP Host Services How – Incorporate Hibernate object-relational mapping (ORM) for DB2, Midas Build XIP Host Services instance Retrieve all data through XIP Host Services and WADO – Select Web application Framework GWT, Tapestry, Spring, Wicket, HybridJava, etc., and build UI for XIP Host Services for Data Retrieval presenting results from MVT application interacting with MVT analysis data based on User Stories and Use Cases. – Using Hibernate, persist results using Jena KB 15

16. Contour-based Analysis Purpose Methods: – STAPLE – Meyer’s P-maps – MICCAI indices – DICE File formats: – DICOM segmentation objects – AIM 4.0 – STL – MHT 16

17. 17

18. Value proposition of QI-Bench Efficiently collect and exploit evidence establishing standards for optimized quantitative imaging: – Users want confidence in the read-outs – Pharma wants to use them as endpoints – Device/SW companies want to market products that produce them without huge costs – Public wants to trust the decisions that they contribute to By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders 18

19. Summary: QI-Bench Contributions We make it practical to increase the magnitude of data for increased statistical significance. We provide practical means to grapple with massive data sets. We address the problem of efficient use of resources to assess limits of generalizability. We make formal specification accessible to diverse groups of experts that are not skilled or interested in knowledge engineering. We map both medical as well as technical domain expertise into representations well suited to emerging capabilities of the semantic web. We enable a mechanism to assess compliance with standards or requirements within specific contexts for use. We take a “toolbox” approach to statistical analysis. We provide the capability in a manner which is accessible to varying levels of collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access. 19

20. QI-Bench Structure / Acknowledgements Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette, Kjell Johnson, Jovanna Danagoulian) Co-Investigators – Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer) – Stanford (David Paik) Financial support as well as technical content: NIST (Mary Brady, Alden Dima, John Lu) Collaborators / Colleagues / Idea Contributors – Georgetown (Baris Suzek) – FDA (Nick Petrick, Marios Gavrielides) – UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad, Yelena Yesha) – Northwestern (Pat Mongkolwat) – UCLA (Grace Kim) – VUmc (Otto Hoekstra) Industry – Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner) – Device/Software: Definiens, Median, Intio, GE, Siemens, Mevis, Claron Technologies, … Coordinating Programs – RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao) – Under consideration: CTMM TraIT (Andre Dekker, Jeroen Belien) 20

21. Statistical Validation Service for Imaging Quantitatively characterizing and optimizing performance of imaging accelerates discovery and widens the availability of new treatments to patients with unmet medical needs. We bring MVT forward as it could be beyond what it currently is; Available as a web application with thin and thick-client options with persistent database. Implemented to be generalizable to needs of RadOnc, QIN, QIBA, FNIH, C- Path, and other members of the community. Applications include: Augmenting the current genome based biomarkers with imaging based biomarkers in Transcend for Breast cancer and/or TCGA for brain cancer Community Cancer Centers Project may be synergistically pursued with the FDA imaging submission project. 21