1. Monthly Program Update March 8, 2012 Andrew J. Buckler, MS Principal Investigator WITH FUNDING SUPPORT PROVIDED BY NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

2. Agenda Summary and close-out of the „Winter 2012“ development iteration – Covering what’s been accomplished from multiple points of view Preview of „Spring 2012“ development iteration – With focus on directions in StudyDescription and „ISA“ storage model, evaluation of workflow engine. 22

3. 3 Winter 2012 (n=47) Autumn 2011 (n=19) Spring 2012 (n=32) Unstaged (n=19) 333 Ramp-up of formal development environment, (including issue tracking) Initial Specify (including QIBO and Knowledgebase) Ramp-up of formal development environment, (including issue tracking) Initial Specify (including QIBO and Knowledgebase) Major update to Execute: Metadata extraction Better Batchmake GUI Initial Formulate Specify now creates instances in knowledgebase Major update to Execute: Metadata extraction Better Batchmake GUI Initial Formulate Specify now creates instances in knowledgebase Change studies Scripted reader studies Export to Analyze Import from Formulate Evaluate workflow application “Iterate” Change studies Scripted reader studies Export to Analyze Import from Formulate Evaluate workflow application “Iterate” Clojure DSL for executable specifications Major update to Analyze RDF-compliance in Specify Formulate using SPARQL Service APIs for Iterate Clojure DSL for executable specifications Major update to Analyze RDF-compliance in Specify Formulate using SPARQL Service APIs for Iterate 3A Pilot 3A Pivotal ISA Storage Model Analyze project Specify/ Formulate project

4. User: Lab Protocol Develop and run queries based on data requirements – Use of Formulate Use of Formulate Load Reference Data into the Reference Data Set Manager – Example Pilot3A Data Processing Steps Example Pilot3A Data Processing Steps Server-Side Processing using the Batch Analysis Service – Package Algorithm or Method using Batch Analysis Service API Package Algorithm or Method using Batch Analysis Service API – Prepare Data Set Prepare Data Set Create Ground Truth or other Reference Annotation and Markup Create Ground Truth or other Reference Annotation and Markup Importing location points and other data for use – Writing Scripts Writing Scripts – Initiate a Batch Analysis Run Initiate a Batch Analysis Run Perform statistical analysis – Analyze Use Instructions Analyze Use Instructions Developer: Design Documents User Needs and Requirements Analysis Architecture Application-specific Design – Specify "Specify" Scope Description (ASD) "Specify" Architecture Specification (AAS) "Quantitative Imaging Biomarker Ontology (QIBO)" Software Design Document (SDD) "Quantitative Imaging Biomarker Ontology (QIBO)" Software Design Document (SDD) "Biomarker DB" (a.k.a., the triple store) Software Design Document (SDD) "Biomarker DB" (a.k.a., the triple store) Software Design Document (SDD) AIM Template Builder Design Documentation: – Formulate "Formulate" Scope Description (ASD) "Formulate" Architecture Specification (AAS) "NBIA Connector" Software Design Document (SDD) – Execute "Execute" Scope Description (ASD) "Execute" Architecture Specification (AAS) Reference Data Set Manager (RDSM) Software Design Document (SDD) Reference Data Set Manager (RDSM) Software Design Document (SDD) Batch Analysis Service Software Design Document (SDD) – Analyze "Analyze" Scope Description (ASD) "Analyze" Architecture Specification (AAS) – Package "Package" Scope Description (ASD) "Package" Architecture Specification (AAS) 4444

5. (Form of) Early Analysis Results 3A Challenge Series 1.Median Technologies 2.Vital Images, Inc. 3.Fraunhofer Mevis 4.Siemens 5.Moffitt Cancer Center 6.Toshiba 5555 Pilot Pivotal Investigation 1 Tr ain Te st Pilot Pivotal Investigation Tr ain Te st Pilot Pivotal Investigation Tr ain Te st Pilot Pivotal Investigation n Tr ain Te st PrimaryPrimary SecondarySecondary Defined set of data Defined challenge Defined test set policy First Participants 7.GE Healthcare 8.Icon Medical Imaging 9.Columbia University 10.INTIO, Inc. 11.Vital Images, Inc.

6. Standardized Representation of Quantitative Imaging Statistical Validation Services for Quantitative Imaging 66666

7. OK. Now into the details for Spring 2012 Iteration: Starting with what we said in January… 7777 Formulate Reference Data Sets QIBO Specify RDF Triple Store CT Volumetry CT obtained_by Tumor growth measure_of Therapeutic Efficacy Therapeutic Efficacy used_for Analyze Y=β 0..n +β 1 (QIB)+β 2 T+ e ij Execute Feedbac k

8. …and where we left off in February… // Business Requirements FNIH, QIBA, and C-Path participants don’t have a way to provide precise specification for context for use and applicable assay methods (to allow semantic labeling): BiomarkerDB = Specify (biomarker domain expertise, ontology for labeling); Researchers and consortia don’t have an ability to exploit existing data resources with high precision and recall: ReferenceDataSet+ = Formulate (BiomarkerDB, {DataService} ); Technology developers and contract research organizations don’t have a way to do large- scale quantitative runs: ReferenceDataSet.CollectedValue+ = Execute (ReferenceDataSet.RawData); The community lacks way to apply definitive statistical analyses of annotation and image markup over specified context for use: BiomarkerDB.SummaryStatistic+ = Analyze ( { ReferenceDataSet.CollectedValue } ); Industry lacks standardized ways to report and submit data electronically: efiling transactions+ = Package (BiomarkerDB, {ReferenceDataSet} ); 888888

9. Rotate it to align with the horizontal rather than vertical presentation of our splash screen… // Business Requirements FNIH, QIBA, and C-Path participants don’t have a way to provide precise specification for context for use and applicable assay methods (to allow semantic labeling): BiomarkerDB = Specify (biomarker domain expertise, ontology for labeling); Researchers and consortia don’t have an ability to exploit existing data resources with high precision and recall: ReferenceDataSet+ = Formulate (BiomarkerDB, {DataService} ); Technology developers and contract research organizations don’t have a way to do large-scale quantitative runs: ReferenceDataSet.CollectedValue+ = Execute (ReferenceDataSet.RawData); The community lacks way to apply definitive statistical analyses of annotation and image markup over specified context for use: BiomarkerDB.SummaryStatistic+ = Analyze ( { ReferenceDataSet.CollectedValue } ); Industry lacks standardized ways to report and submit data electronically: efiling transactions+ = Package (BiomarkerDB, {ReferenceDataSet} ); 999999

10. …to arrive at a new more complete view (interpreting the braces as a separate application) // Business Requirements FNIH, QIBA, and C-Path participants don’t have a way to provide precise specification for context for use and applicable assay methods (to allow semantic labeling): BiomarkerDB = Specify (biomarker domain expertise, ontology for labeling); Researchers and consortia don’t have an ability to exploit existing data resources with high precision and recall: ReferenceDataSet+ = Formulate (BiomarkerDB, {DataService} ); Technology developers and contract research organizations don’t have a way to do large-scale quantitative runs: ReferenceDataSet.CollectedValue+ = Execute (ReferenceDataSet.RawData); The community lacks way to apply definitive statistical analyses of annotation and image markup over specified context for use: BiomarkerDB.SummaryStatistic+ = Analyze ( { ReferenceDataSet.CollectedValue } ); Industry lacks standardized ways to report and submit data electronically: efiling transactions+ = Package (BiomarkerDB, {ReferenceDataSet} ); 10

11. Worked Example (starting from claim analysis we discussed in February 2011) Measurements of tumor volume are more precise (reproducible) than uni- dimensional tumor measurements of tumor diameter. Longitudinal changes in whole tumor volume during therapy predict clinical outcomes (i.e., OS or PFS) earlier than corresponding uni-dimensional measurements. Therefore, tumor response or progression as determined by tumor volume will be able to serve as the primary endpoint in well-controlled Phase II and III efficacy studies of cytotoxic and selected targeted therapies (e.g., antiangiogenic agents, tyrosine kinase inhibitors, etc.) in several solid, measurable tumors (including both primary and metastatic cancers of, e.g., lung, liver, colorectal, gastric, head and neck cancer,) and lymphoma. Changes in tumor volume can serve as the endpoint for regulatory drug approval in registration trials. Biomarker claim statements are information-rich and may be used to set up the needed analyses. 11

12. The user enters information from claim into the knowledgebase using Specify Measurements of tumor volume are more precise (reproducible) than uni-dimensional tumor measurements of tumor diameter. Longitudinal changes in whole tumor volume during therapy predict clinical outcomes (i.e., OS or PFS) earlier than corresponding uni-dimensional measurements. Therefore, tumor response or progression as determined by tumor volume will be able to serve as the primary endpoint in well- controlled Phase II and III efficacy studies of cytotoxic and selected targeted therapies (e.g., antiangiogenic agents, tyrosine kinase inhibitors, etc.) in several solid, measurable tumors (including both primary and metastatic cancers of, e.g., lung, liver, colorectal, gastric, head and neck cancer,) and lymphoma. Changes in tumor volume can serve as the endpoint for regulatory drug approval in registration trials. SubjectPredicateObject CTimagesTumor VolumetryanalyzesCT Longitudinal Volumetry estimatesTumorSize Change TumorSize Change predictsTreatment Response Categ oric Contin uous 12

13. …pulling various pieces of information, Measurements of tumor volume are more precise (reproducible) than uni-dimensional tumor measurements of tumor diameter. Longitudinal changes in whole tumor volume during therapy predict clinical outcomes (i.e., OS or PFS) earlier than corresponding uni-dimensional measurements. Therefore, tumor response or progression as determined by tumor volume will be able to serve as the primary endpoint in well- controlled Phase II and III efficacy studies of cytotoxic and selected targeted therapies (e.g., antiangiogenic agents, tyrosine kinase inhibitors, etc.) in several solid, measurable tumors (including both primary and metastatic cancers of, e.g., lung, liver, colorectal, gastric, head and neck cancer,) and lymphoma. Changes in tumor volume can serve as the endpoint for regulatory drug approval in registration trials. SubjectPredicateObject CTimagesTumor VolumetryanalyzesCT Longit udinalVolumetry estimatesTumorSizeChange predictsCytotoxicTreatment Response TyrosineKinase Inhibitor isCytotoxicTreatment well-controlled Phase II and III efficacy studies usesCytotoxicTreatment Response Cytotoxic Treatment influencesNonSmallCellLung Cancer CTimagesThorax containsNonSmallCellLung Cancer Interv ention Target Indicat ion 13

14. …to form the specification. Measurements of tumor volume are more precise (reproducible) than uni-dimensional tumor measurements of tumor diameter. Longitudinal changes in whole tumor volume during therapy predict clinical outcomes (i.e., OS or PFS) earlier than corresponding uni-dimensional measurements. Therefore, tumor response or progression as determined by tumor volume will be able to serve as the primary endpoint in well- controlled Phase II and III efficacy studies of cytotoxic and selected targeted therapies (e.g., antiangiogenic agents, tyrosine kinase inhibitors, etc.) in several solid, measurable tumors (including both primary and metastatic cancers of, e.g., lung, liver, colorectal, gastric, head and neck cancer,) and lymphoma. Changes in tumor volume can serve as the endpoint for regulatory drug approval in registration trials. To produce data for registration To substantiate quality of evidence development SubjectPredicateObject CTimagesTumor VolumetryanalyzesCT Longitudinal Volumetry estimatesTumorSizeChange predictsCytotoxicTreatmentResponse TyrosineKinaseInhibitorisCytotoxicTreatment well-controlled Phase II and III efficacy studies usesCytotoxicTreatmentResponse CytotoxicTreatmentinfluencesNonSmallCellLungCancer CTimagesThorax containsNonSmallCellLungCancer regulatory drug approvaldependsOnPrimaryEndpoint well-controlled Phase II and III efficacy studies assessPrimaryEndpoint CT Volumetryis SurrogateEndpoint 14

15. Formulate interprets the specification as testable hypotheses, Measurements of tumor volume are more precise (reproducible) than uni-dimensional tumor measurements of tumor diameter. Longitudinal changes in whole tumor volume during therapy predict clinical outcomes (i.e., OS or PFS) earlier than corresponding uni-dimensional measurements. Therefore, tumor response or progression as determined by tumor volume will be able to serve as the primary endpoint in well- controlled Phase II and III efficacy studies of cytotoxic and selected targeted therapies (e.g., antiangiogenic agents, tyrosine kinase inhibitors, etc.) in several solid, measurable tumors (including both primary and metastatic cancers of, e.g., lung, liver, colorectal, gastric, head and neck cancer,) and lymphoma. Changes in tumor volume can serve as the endpoint for regulatory drug approval in registration trials. Type of biomarker, in this case predictive (could have been something else, e.g., prognostic), to establish the mathematical formalism Technical characteri stic SubjectPredicateObject CTimagesTumor VolumetryanalyzesCT Longitudinal Volumetry estimatesTumorSizeChange predictsCytotoxicTreatmentResponse TyrosineKinaseInhibitorisCytotoxicTreatment well-controlled Phase II and III efficacy studies usesCytotoxicTreatmentResponse CytotoxicTreatmentinfluencesNonSmallCellLungCancer CTimagesThorax containsNonSmallCellLungCancer regulatory drug approvaldependsOnPrimaryEndpoint well-controlled Phase II and III efficacy studies assessPrimaryEndpoint CT Volumetryis SurrogateEndpoint 1 3 2 15

16. …setting up an investigation (I), study (S), assay (A) hierarchy… SubjectPredicateObject CTimagesTumor VolumetryanalyzesCT Longitudinal Volumetry estimatesTumorSizeChange predictsCytotoxicTreatmentResponse TyrosineKinaseInhibitorisCytotoxicTreatment well-controlled Phase II and III efficacy studies usesCytotoxicTreatmentResponse CytotoxicTreatmentinfluencesNonSmallCellLungCancer CTimagesThorax containsNonSmallCellLungCancer regulatory drug approvaldependsOnPrimaryEndpoint well-controlled Phase II and III efficacy studies assessPrimaryEndpoint CT Volumetryis SurrogateEndpoint 1 3 2 16 Investigations to Prove the Hypotheses: 1.Technical Performance = Biological Target + Assay Method 2.Clinical Validity = Indicated Biology + Technical Performance 3.Clinical Utility = Biomarker Use + Clinical Validity Investigation-Study-Assay Hierarchy: Investigation = {Summary Statistic} + {Study} Study = {Descriptive Statistic} + Protocol + {Assay} Assay = RawData + {AnnotationData} AnnotationData = [AIM file|mesh|…]

17. …ADDING TRIPLES TO CAPTURE URIs: SubjectPredicateObject ClinicalUtilityisInvestigation URI ClinicalValidityisInvestigation URI TechnicalPerformanceisInvestigation URI InvestigationhasSummaryStatisticType InvestigationhasStudy URI StudyhasDescriptiveStatisticType StudyhasProtocol URI StudyhasAssay URI AssayhasRawData URI …and loading data into Execute (at least raw data, possibly annotations if they already exist) SubjectPredicateObject AIsPatient AisDiagnosedWithDiseaseA IsNonSmallLCellLunCancer PazopanibIsTyrosoineKinaseInhibitor AhasBaselineCT AhasTP1CT AhasTP2CT BisDiagnosedWithDiseaseA BhasBaselineCT BhasTP1CT AhasOutcomeDeath BhasOutcomeSurvival 17 DISCOVERED DATA:…LOADING DATA INTO THE RDSM:

18. If no annotations, Execute creates them (in either case leaving Analyze with its data set up for it) SubjectPredicateObject ClinicalUtilityisInvestigation URI ClinicalValidityisInvestigation URI TechnicalPerformanceisInvestigation URI InvestigationhasSummaryStatisticType InvestigationhasStudy URI StudyhasDescriptiveStatisticType StudyhasProtocol URI StudyhasAssay URI AssayhasRawData URI AssayhasAnnotationData URI AIM fileisAnnotationData URI MeshisAnnotationData URI 18 Either in batch or via Scripted reader studies (using “Share” and “Duplicate” functions of RDSM to leverage cases across investigations) (self-generating knowledgebase from RDSM hierarchy and ISA-TAB description files)

19. Analyze performs the statistical analyses… SubjectPredicateObject AIsPatient AisDiagnosedWithDiseaseA IsNonSmallLCellLunCancer AhasClinicalObserva tion B BIsTumorShrinkage CIsPatient ChasClinicalObserva tion B D B PazopanibIsTyrosoineKinaseInhibitor AisTreatedWithPazopanib AhasOutcomeDeath ChasOutcomeSurvival SubjectPredicateObject CTimagesTumor VolumetryanalyzesCT Longitudinal Volumetry estimatesTumorSizeChange predictsCytotoxicTreatmentResponse TyrosoineKinaseInhibitorisCytotoxicTreatment well-controlled Phase II and III efficacy studies usesCytotoxicTreatmentResponse CytotoxicTreatmentinfluencesNonSmallCellLungCancer CTimagesThorax containsNonSmallCellLungCancer regulatory drug approvaldependsOnPrimaryEndpoint well-controlled Phase II and III efficacy studies assessPrimaryEndpoint CT VolumetryisSurrogateEndpoint for CytotoxicTreatment 1 3 2 19

20. …and adds the results to the knowledgebase (using W3C “best practices” for “relation strength”). SubjectPredicateObject 45324biasMethod 45324bias 45324variabilityMethod 45324variability 9956 Method 9956correlation 9956 Method 9956ROC 98234Effect of treatment on true endpoint 98234Effect of treatment on surrogate endpoint 98234Effect of surrogate on true endpoint 98234Effect of treatment on true endpoint relative to that on surrogate endpoint SubjectPredicateObject CTimagesTumor VolumetryanalyzesCT Longitudinal Volumetry estimatesTumorSizeChange predictsCytotoxicTreatmentResponse TyrosoineKinaseInhibitorisCytotoxicTreatment well-controlled Phase II and III efficacy studies usesCytotoxicTreatmentResponse CytotoxicTreatmentinfluencesNonSmallCellLungCancer CTimagesThorax containsNonSmallCellLungCancer regulatory drug approvaldependsOnPrimaryEndpoint well-controlled Phase II and III efficacy studies assessPrimaryEndpoint CT VolumetryisSurrogateEndpoint for CytotoxicTreatment 1 3 2 URI=45324 URI=9956 URI=98234 20

21. Package Structure submissions according to eCTD, HL7 RCRIM, and SDTM Section 2Summaries 2.1.Biomarker Qualification Overview 2.1.1.Introduction 2.1.2.Context of Use 2.1.3.Summary of Methodology and Results 2.1.4.Conclusion 2.2.Nonclinical Technical Methods Data 2.2.1.Summary of Technical Validation Studies and Analytical Methods 2.2.2.Synopses of individual studies 2.3.Clinical Biomarker Data 2.3.1.Summary of Biomarker Efficacy Studies and Analytical Methods 2.3.2.Summary of Clinical Efficacy [one for each clinical context] 2.3.3.Synopses of individual studies Section 3Quality Section 4Nonclinical Reports 4.1.Study reports 4.1.1.Technical Methods Development Reports 4.1.2.Technical Methods Validation Reports 4.1.3.Nonclinical Study Reports (in vivo) 4.2.Literature references Section 5Clinical Reports 5.1.Tabular listing of all clinical studies 5.2.Clinical study reports and related information 5.2.1.Technical Methods Development reports 5.2.2.Technical Methods Validation reports 5.2.3.Clinical Efficacy Study Reports [context for use] 5.3.Literature references 21 SubjectPredicateObject 45324biasMethod 45324bias 45324variabilityMethod 45324variability 9956 Method 9956correlation 9956 Method 9956ROC 98234Effect of treatment on true endpoint 98234Effect of treatment on surrogate endpoint 98234Effect of surrogate on true endpoint 98234Effect of treatment on true endpoint relative to that on surrogate endpoint

22. Iterate: Reproducible Workflows with Documented Provenance (with illustration expansion of databases) 22 Knowledge base Triples NN+500N+1000 N+2000-> eCTD Reference Data Sets MMM+1000M+10,000 -> eCTD

23. 23

24. Value proposition of QI-Bench Efficiently collect and exploit evidence establishing standards for optimized quantitative imaging: – Users want confidence in the read-outs – Pharma wants to use them as endpoints – Device/SW companies want to market products that produce them without huge costs – Public wants to trust the decisions that they contribute to By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders 24

25. Summary: QI-Bench Contributions We make it practical to increase the magnitude of data for increased statistical significance. We provide practical means to grapple with massive data sets. We address the problem of efficient use of resources to assess limits of generalizability. We make formal specification accessible to diverse groups of experts that are not skilled or interested in knowledge engineering. We map both medical as well as technical domain expertise into representations well suited to emerging capabilities of the semantic web. We enable a mechanism to assess compliance with standards or requirements within specific contexts for use. We take a “toolbox” approach to statistical analysis. We provide the capability in a manner which is accessible to varying levels of collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access. 25

26. QI-Bench Structure / Acknowledgements Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette) Co-Investigators – Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer) – Stanford (David Paik) Financial support as well as technical content: NIST (Mary Brady, Alden Dima, John Lu) Collaborators / Colleagues / Idea Contributors – Georgetown (Baris Suzek) – FDA (Nick Petrick, Marios Gavrielides) – UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad, Yelena Yesha) – Northwestern (Pat Mongkolwat) – UCLA (Grace Kim) – VUmc (Otto Hoekstra) Industry – Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner) – Device/Software: Definiens, Median, Intio, GE, Siemens, Mevis, Claron Technologies, … Coordinating Programs – RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao) – Under consideration: CTMM TraIT (Andre Dekker, Jeroen Belien) 26