1. Monthly Program Update January 12, 2012 Andrew J. Buckler, MS Principal Investigator WITH FUNDING SUPPORT PROVIDED BY NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

2. Agenda Monthly snapshot in Jira – (including status of installation at NIST) QIBA 3A project snapshot Theoretical development Architecture and SW stack 22

3. 3 BSD-2 license Domain is www.qi-bench.orgwww.qi-bench.org. Landing page provides Access to prototypes, Repositories for download and development, Acknowledgemen ts, Jira issue tracking, and Documentation 3

4. QIBA 3A PROJECT SNAPSHOT (recalling that this is a testbed for us) 444

5. Basic structure of the challenges 5 Pilot Pivotal Investigation 1 Train Test Pilot Pivotal Investigation Train Test Pilot Pivotal Investigation Train Test Pilot Pivotal Investigation n Train Test PrimaryPrimary SecondarySecondary Defined set of data Defined challenge Defined test set policy 555 First one: Presently in pilot phase, Using StudyDescription method Used batch scripting with reference method to aid data curation 10-12 participants (about 20 QI-Bench users) First participant data received Analysis plan using N-way ANOVA in R started Pivotal phase starting with batch assisted curation Will be transitioning to database schema for metadata (gradually away from spreadsheet) First one: Presently in pilot phase, Using StudyDescription method Used batch scripting with reference method to aid data curation 10-12 participants (about 20 QI-Bench users) First participant data received Analysis plan using N-way ANOVA in R started Pivotal phase starting with batch assisted curation Will be transitioning to database schema for metadata (gradually away from spreadsheet)

6. 666 1.Relative performance is computed according to descriptive statistics 2.We determine a group value for each of the descriptive statistics, e.g., as the mean plus 1 stdev (or as wide as we think wise). 3.Results presented using radar plots

7. Bias Variability Repeatability cross-x reproducibili ty cross-y reproducibili ty New Method7.003.0010.0011.0012.00 Group7.166.646.967.819.00 In this example, the new proposed method does not perform well enough to be considered a valid method since it falls outside the group values. 777

8. In this example, the new proposed method is seen to perform within group values and may even help pave the way for an improved claim. Bias Variability Repeatability cross-x reproducibility cross-y reproducibility New Method3.00 4.006.008.00 Group7.166.646.967.819.00 888

9. THEORETICAL DEVELOPMENT progress re: utilization of logical and statistical inference at each of two levels, technical performance of assay methods, and qualification of biomarker in specific clinical context 999

10. Another way to look at what needs to happen 10 Formulate Reference Data Sets QIBO Specify RDF Triple Store CT Volumetry CT obtained_by Tumor growth measure_of Therapeutic Efficacy Therapeutic Efficacy used_for Analyze Y=β 0..n +β 1 (QIB)+β 2 T+ e ij Execute Feedbac k

11. Specify: Establish a logical specification and setup terms for mathematical analysis 11 Functionality: Establish means to semantically labeling imaging biomarker data with emphasis on representing both the clinical context in which an imaging biomarker is used as well as the specifics of the imaging protocol used to acquire the images. Set up the logistic regression model: Precisely specify dependant variable Account for covariates Enumerate independent variables and error terms (sources of variability) Establish database for collection of terms. Method: Provide GUI to traverse the QIBO concepts according to their relationships and create statements represented as RDF triples and stored in an RDF store. Each set of RDF triples will be stored as a “profile.” Relationship strength initialized based on prior estimates (if available) QIBO Specify RDF Triple Store CT Volu metry CT obtained_by Tumor growt h measure_of Therap eutic Efficac y Therap eutic Efficac y used_for

12. Ontologies supporting Specify 12 Extend the QIBO to link to existing established ontologies 1.leverage BFO upper ontology to align different ontologies 2.convert portions of BRIDG and LSDAM to ontology models in OWL Automated conversion would done in two steps: 1.convert current Sparx Enterprise Architect XMI EMF UML format 2.export resulting EMF UML into a RDF/OWL representation using TopBraid Composer

13. Formulate: advanced query framework made possible by Specify 13 allow users to select the profiles (or set of RDF triplets) created in Specify, execute a query and retrieve the results in various forms. assemble/transform the set of RDF triples to SPARQL queries: 1.form an uninterrupted chain linking the instance of the input class from the ontology to the desired output class 2.formulate/invoke necessary SPARQL queries against the web services deployed in SADI framework. interface with the query engine and will have offline (asynchronous) query execution capability. results to be exportable as serialized objects (RDF/XML and CSV) Formulate Reference Data Sets RDF Triple Store CT Volu metry CT obtained_by Tumor growt h measure_of Therap eutic Efficac y Therap eutic Efficac y used_for

14. Data Services supporting Formulate 14 wrap existing data services such as NBIA, caArray, caTissue, AIM and PODS using Semantic Automated Discovery and Integration (SADI) this is enabled by metadata available through the UML representations of the models exposed by these services and CDE annotations available for them through caDSR. describe service I/O semantically using the extended version of QIBO service registry of SADI will help the automated composition of computer- interpretable queries by the query engine. example: “there is a service that returns Biological Subjects that has undergone certain Biological Interventions”

15. Analyze: Use annotation and image markup to support statistical inference 15 Support Clinical Performance assessment (i.e., in addition to current Technical Performance) Outcome studies Integrated genomic/proteomic correlation studies Group studies for biomarker qualification (set up a basic multiple regression analysis, e.g.) Intent to treat analysis of the primary outcome via covariance model of the general form (QIB t )=β 0..n +β 1 (QIB 0 )+β 2 T+ e ij where QIB t and QIB 0 are the QIB at a time after treatment and at randomization respectively, T is a treatment group indicator, and β 0..n, β 1, and β 2 are model parameters. β 2 represents the effect of treatment and its estimate is the difference between group means on the log scale, after adjustment for any imbalance between the groups in log QIB. The error terms in the model, e ij, are assumed mutually independent and normally distributed. Depending on the nature of the QIB, the log transformation may be used instead of the direct value. Likewise calculations may be performed using z scores with corresponding conversion with raw values.

16. Examples of output at biomarker (above the assay level) 16 From Jack 1999. Note: W-score is the relative score of the measured HC volume corrected for intracranial volume and compared to age and sex adjusted normals. Hypothetical model of dynamic biomarkers of the Alzheimer's pathological cascade Jack et al., 2010 16 To inform thresholding To substantiate surrogacy (or its weaker form of “activity”)

17. ARCHITECTURE AND SW STACK So what is a cohesive architecture that maximizes leverage of best thinking, existing touchpoints, and stays current over time? 17

18. STDM standard of CDISC into repositories like FDA’s Janus. MVT portion of AVT, re- useable library of R scripts. MIDAS, BatchMake, Condor Grid; built using Zend on PHP. caB2B, NBIA, PODS data elements, DICOM query tools. QIBO, AIM, RadLex/ Snomed/ NCIt; built using Ruby on Rails. Specify context for use and assay methods. Use consensus terms in doing so. Specify Assemble applicable reference data sets. Include both imaging and non-imaging clinical data. Formulate Compose and iterate batch analyses on reference data. Accumulate quantitative read-outs for analysis. Execute Characterize the method relative to intended use. Apply the existing tools and/or extend them. Analyze Compile evidence for regulatory filings. Use standards in transfer to regulatory agencies. Package 18

19. MVT : Reasonable framework, but many gaps 19 There are multiple possibilities to deploy it as a web application, some of which we’ve considered: 1.Re-implement the existing implementation to use GWT in place of Swing, inclusive of both the XIPHost as well as MVT components, retaining the WG23 concept. 2.Re-implement only those parts necessary to perform the needed MVT functions using GWT with enough data handling to do so but without doing everything necessary to retain the full XIPHost capability. 3.Leverage the GUI design concept but otherwise implement without starting from the Swing code. In all cases, there is the secondary design alternative of introducing a RESTful web service layer explicitly or not. (By the way, just for fun, I performed a conversion of the current Swing code to Ajax using AjaxSwing. I got most of AVT working over the web with minimal effort, but this isn’t a serious alternative because AjaxSwing has a license fee. I did it because I wanted to see how easy such a path would be. It’s an interesting capability! But irrelevant in the end.)

20. Pros: optimized for DICOM, works with workstations Cons: hard to create web apps, not optimized for semantic web 20 HW XIP Application Inventor Application Modules WG 23 System Services PLUG WG 23 System Services SOCKET GRID CLIENT SERVICES DICOM SERVICES (DCMTK) OTHER SERVICES VTKITKAIMTKother OS NCIA XIP IDE RadLex AIM NCI Protégé EVS XIP MIDDLEWARE DICOM DICOM Services IVI Middleware caGrid CaBIG caDSR, EVS, RadLex, AIM ontology, etc Client access Service access Grid Data Service Grid Analytical Service AIM Data Service XIP App Service Host WG23 DICOM Image Sources 20

21. Alternative architectural form… 21 SW Stack J2SE (J2EE compliant) MySQL caGrid Globus Application: JBoss caCore 21 With pros and cons “opposite” that of the XIP based architecture

22. Functionality view annotated with architecture 22 HW XIP Application Inventor Application Modules WG 23 System Services PLUG WG 23 System Services SOCKET GRID CLIENT SERVICES DICOM SERVICES (DCMTK) OTHER SERVICES VTKITKAIMTKother OS NCIA XIP IDE RadLex AIM NCI Protégé EVS XIP MIDDLEWARE DICOM DICOM Services IVI Middleware caGrid CaBIG caDSR, EVS, RadLex, AIM ontology, etc Client access Service access Grid Data Servi ce Grid Analytica l Service AI M Dat a Ser vic e XIP Ap p Ser vic e Ho st W G2 3 DIC OM Imag e Sour ces 22 MIDAS Core ApacheFile SystemPostGreSQL Publication DB MIDAS Data Server MIDAS e- journal MIDAS Compute Server MIDAS Visualization MIDAS Client MIDAS C++ API MIDAS Web API When annotation and markup has already been done Reference data sets Annotation and markup AIM-enabled (e.g., ClearCanvas) workstation RIS worklist items DICOM Q/R

23. First step to rationalizing architecture: mash them together and see what falls out 23 NCIA RadLex AIM NCI XIP MIDDLEWARE DICOM DICOM Services IVI Middleware SADI framework (e.g., wrapped caGrid) CaBIG caDSR, EVS, RadLex, AIM ontology, etc Client access Service access Grid Data Service Grid Analytical Service AIM Data Service HW XIP Application Inventor Application Modules WG 23 System Services PLUG WG 23 System Services SOCKET GRID CLIENT SERVICES DICOM SERVICES (DCMTK) OTHER SERVICES VTKITK AIMTK other OS XIP IDE Protégé EVS XIP App Service Host WG23 DICOM Image Sources This is an ongoing discussion. More to come!

24. 24

25. Value proposition of QI-Bench Efficiently collect and exploit evidence establishing standards for optimized quantitative imaging: – Users want confidence in the read-outs – Pharma wants to use them as endpoints – Device/SW companies want to market products that produce them without huge costs – Public wants to trust the decisions that they contribute to By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders 25

26. Summary: QI-Bench Contributions We make it practical to increase the magnitude of data for increased statistical significance. We provide practical means to grapple with massive data sets. We address the problem of efficient use of resources to assess limits of generalizability. We make formal specification accessible to diverse groups of experts that are not skilled or interested in knowledge engineering. We map both medical as well as technical domain expertise into representations well suited to emerging capabilities of the semantic web. We enable a mechanism to assess compliance with standards or requirements within specific contexts for use. We take a “toolbox” approach to statistical analysis. We provide the capability in a manner which is accessible to varying levels of collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access. 26

27. QI-Bench Structure / Acknowledgements Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette) Co-Investigators – Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer) – Stanford (David Paik, Tiffany Ting Liu) Financial support as well as technical content: NIST (Mary Brady, Alden Dima, Guillaume Radde) Collaborators / Colleagues / Idea Contributors – FDA (Nick Petrick, Marios Gavrielides) – UCLA (Grace Kim) – UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad) – VUmc (Otto Hoekstra) – Northwestern (Pat Mongkolwat) – Georgetown (Baris Suzek) Industry – Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner) – Device/Software: Definiens (Maria Athelogou), Claron Technologies (Ingmar Bitter) Coordinating Programs – RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao) – Under consideration: CTMM TraIT (Andre Dekker, Jeroen Belien) 27