1. Siemens1 AVT Phase II Use Case Model Bob Schwanke Version of 2008-Dec-11

2. Siemens2 How to read this document A Use Case is a category of behavior –Use Cases are organized as a hierarchy, partially ordered by Includes Includes, optionally [“is extended by” in UML] Has special case [“has subclass” in UML] –Use Case hierarchy has a single root –The “includes” list only suggests an ordering, but does not specify the order or the number of repetitions –A composite use case usually has 3-9 children A Scenario is an example of a Use Case –Specifies a sequencing of parts of the use case { braces } contain implementation priorities –{1} will certainly be built for SIIM ’09, to support the first user study –{2} will probably be built for SIIM ’09, to support additional user studies specified in this model –{later} will probably not be built for SIIM ’09 Also note Vocabulary and Terms to AvoidVocabulary Terms to Avoid

3. Siemens3 Vocabulary Scan: one logical image, whether 2D, 3D, or 4D; includes PET-CT fused data scan. Scan-series: a temporal sequence of 2 or more scans of the same subject A scan-series is a constructed object that associates two or more scans of the same subject in a temporal sequence. Scan-series are used for AVT studies that investigate change in tumors from one scan to the next, such as “coffee break” and “before and after treatment”. Landmark: a naturally-occurring anatomical feature used for systematic, uniform orientation and measurement. Fiducial: an artificial mark, such as a tattoo, placed in or on a subject to facilitate systematic, uniform orientation and measurement. Nodule: lesion, tumor, lump, etc. See also Terms to AvoidTerms to Avoid Note: this terminology is temporary, pending reconciliation with the RadLex team.

4. Siemens4 Terms to Avoid The following terms are ambiguous and should be avoided. Image: always modify it, e.g. DICOM Image or logical image or image processing, or use scan. Study: always modify it, e.g. DICOM Study, AVT Study, or reader study. Series: always modify it, e.g. DICOM Series or scan-series.

5. Siemens5 AVTAVT context Interfaces: DICOM Database –Exchange DICOM objects with other DICOM databases (which are (a) local or (b) caGRID DICOM Data Services) {1} AIM Database –Exchange AIM objects with other AIM databases (which are (a) local or (b) caGRID AIM Data Services). (Data model will be an extension of the AIM object model.) {1} AVT Study Database –Exchange AVT study data with other study databases, (a) directly, and (b) eventually using a suitable caGRID Data Service. {later} Manual annotation interface {1} Algorithm execution interface –Batch execution of algorithm on many images {2} Measurement Variability Analysis Interface {1} Installation and configuration interface {1} Development and customization interface {1} Data management interface –Collect and organize images, annotations, and study data {1}

6. Siemens6 AVT {1} Includes install AVT {1} design AVT study {1}AVT study (optionally) install image processing algorithm {2}install image processing algorithm (optionally) customize annotation tool {2}customize annotation tool (optionally) customize measurement variability tool {2}customize measurement variability tool (optionally) exchange data with other sites {1} conduct AVT study {1}conduct AVT study Context: AVT contextAVT context

7. Siemens7 AVT study (data type) {1} Includes Relationship to nominal ground truth: [ unknown | known {1} | calculating] General Goal: [ exploration {1} | validation {1} | algorithm improvement] annotation task to be analyzed {1}annotation task Source of scan data {1} independent variables {1}independent variables comparisons {1}comparisons statistical methods {1}statistical methods study representation {1}study representation Has special case Example AVT Studies {1}Example AVT Studies Part of AVT  conduct AVT study  algorithmically annotate scansAVTconduct AVT studyalgorithmically annotate scans

8. Siemens8 annotation task {1} “Annotation” includes both attaching graphics (markup) to images and attaching structured data (annotations) to scans and scan-series. Most annotation tasks could be carried out manually, semi-automatically, or automatically. Except where indicated, the model describes all three possibilities with the same use-cases. Includes, optionally (manually) read annotation instructions {2} scan annotation task {1}scan annotation task scan-series annotation task {later}scan-series annotation task evaluate task output: confidence, accuracy {2} add audit trail information {2}audit trail information add pedigree information {2?}pedigree information Part of AVT  AVT studyAVTAVT study Part of AVT  conduct AVT study  algorithmically annotate scansAVTconduct AVT studyalgorithmically annotate scans

9. Siemens9 scan annotation task {1} Includes, optionally, locate task-object {2?}locate task-object measure diameter(s): RECIST {1}, WHO {2?}, Wolfe segment task-object {1}segment task-object describe task-object {2?}describe task-object create atlas of scan {later} rate scan quality {later} has special case annotate scout scan {later}annotate scout scan annotate tumor {1}annotate tumor Part of AVT  AVT study  annotation taskAVTAVT studyannotation task

10. Siemens10 locate task-object {2?}task-object Includes locating one or more of the following Seed point Maximal slice Center of mass Landmark point Region of interest (ROI) surrounding task object Part of AVT  AVT study  annotation task  scan annotation taskAVTAVT studyannotation taskscan annotation task Note: finding an ROI and then segmenting the object in it can also be called unsupervised segmentation

11. Siemens11 task object {1} Has special case Nodule {1} Lymph node {later} Organ {later} Landmark {later} Some of these task object types are not relevant for certain tasks. Includes (attributes) ??? Part of AVT  AVT study  annotation task  scan annotation task  locate task object  segment task object  describe task objectAVTAVT studyannotation taskscan annotation tasklocate task objectsegment task objectdescribe task object Part of AVT  AVT study  annotation task  scan-series annotation task  track task-object across scan-seriesAVTAVT studyannotation taskscan-series annotation task track task-object across scan-series

12. Siemens12 segment task object {1}task object Includes, optionally mark a seed point contained within the task object {1} mark a “stroke” contained within the task object mark an ROI containing the task-object {later} algorithmically estimate boundary {1} interactively delineate or improve boundaries of entity {1} algorithmically calculate diameter(s), area(s) and/or volume {1} Has special case 2D segmentation {later} 2½D segmentation {2?} 3D segmentation {1?} –Done semi-automatically, with nudging to improve on algorithmic estimate –create 3D model out of tumor boundary on each slice or three orthogonal slices automatic or interactive (nudging) –Region growing, watershed methods, level sets, etc. Part of AVT  AVT study  annotation task  scan annotation task  annotate tumorAVTAVT studyannotation taskscan annotation taskannotate tumor

13. Siemens13 describe task object {2?}task object Includes, optionally Classify Characterize –Density –Margin distinctness –Shape regularity –etc. Actionability Attach clinical information to task object –biopsy info –outcome –etc. Part of AVT  AVT study  annotation task  scan annotation taskAVTAVT studyannotation taskscan annotation task RECIST demo of this capability could perhaps be ready for SIIM 09

14. Siemens14 annotate scout scan (localizer) {later} Purpose: define range and orientation of a diagnostic scan Includes (optional) select best slices Attach landmark points to landmarks [according to instructions] Special case of AVT  AVT study  annotation task  scan annotation taskAVTAVT studyannotation taskscan annotation task

15. Siemens15 annotate tumor {1} Includes, optionally find tumor (identification, seed point?) {later?} find center of mass of tumor (algorithmic only) {later?} find maximal slice (axial slice in which tumor has maximum extension) {later?} find most aggressive part {later} measure longest diameter within slice (RECIST) {1} measure longest orthogonal diameters (2D for WHO {2?}, 3D for Wolfe Criteria {later} ) imprecise localization (e.g. ellipse, ROI) {later} segment task object {1}segment task object Describe task object (tumor) {later?}Describe task object –e.g. density, margin distinctness, attach biopsy information to tumor (biopsy info may have been collected separately under curation) Note: annotation methods are different depending on the type of tumor, the organ where it is located, and its size, among other things. Special case of AVT  AVT study  annotation task  scan annotation task AVTAVT studyannotation taskscan annotation task

16. Siemens16 scan-series scan-series annotation task {later} Includes, optionally register scan-series track task-object across scan-seriestrack task-object across scan-series Measure change in task-object –classify change –characterize change –attach additional treatment history (e.g. radiation therapy) Has scenario Oncocare registration scenario Part of AVT  AVT study  annotation taskAVTAVT studyannotation task

17. Siemens17 register scan-series {later}scan-series Includes, optionally Select ROI to register Use registration algorithms to estimate registrationregistration algorithms Manually adjust registration 4D registration to account for tumor movement while breathing measure quality of registration (e.g. normalized mutual information) visualize quality of registration (e.g. subtraction image) Has special case Register images of same type (e.g. helical CT) Register images of different types (e.g. PET/CT, MR/CT, Helical/Cone CT) Part of AVT  AVT study  annotation task  scan-series annotation taskAVTAVT studyannotation taskscan-series annotation task

18. Siemens18 registration algorithms {later} These are automatic registration methods, which the reader may invoke, but which may cause bias if used to speed up defining registration nominal ground truth. Includes, optionally Rigid registration Affine registration Deformable registration Part of AVT  AVT study  annotation task  scan-series annotation task  register scan-seriesAVTAVT studyannotation taskscan-series annotation taskregister scan-series

19. Siemens19 track task-object across scan-series {later}task-object scan-series Linking objects (e.g. tumors) in two or more scans to indicate that they are believed to be the same object, or one birthed from the other, etc. Includes, optionally Linking objects by seed points Linking objects by centers of mass Linking objects by contours Linking objects by maximal slices Tracking 3D object in 4D scan Part of AVT  AVT study  annotation task  scan-series annotation taskAVTAVT studyannotation taskscan-series annotation task RECIST Adjudicator could illustrate this use-case{2?}

20. Siemens20 OncoCare registration scenario {later} With two scans, 1.Perform rigid registration of whole scans 2.Select small ROI around tumor in one scan 3.Calculate deformation field to other scan 4.Map center of mass of tumor from first to second scan Example of AVT  AVT study  annotation task  scan-series annotation task AVTAVT studyannotation taskscan-series annotation task Can be semi-automatic (for manual annotation) or automatic (for auto- annotate), depending on whether ROI can be found automatically.

21. Siemens21 audit trail information {2} The term pedigree, as it appears in the Gap Analysis Document, combines two concepts: audit trail information and expert commentary, the latter explaining why an annotation is what it is. In this model, we use “pedigree” to refer only to expert commentary. AVT will take care of recording audit trail information automatically, so it is not explicitly listed as a part of any particular use case. audit trail information includes, optionally start/source data changes/measurements made tools used, with versions user date/time of annotation Part of AVT  AVT study  annotation taskAVTAVT studyannotation task Audit trail information might be attached to almost any annotation.

22. Siemens22 pedigree information {2?} includes, optionally methods used to establish nominal ground truth subjective uncertainty of observations Part of AVT  AVT study  annotation taskAVTAVT studyannotation task Part of AVT  conduct AVT study  curate collectionAVTconduct AVT studycurate collection The term pedigree in the Gap Analysis document combines two concepts: audit trail information and expert commentary, the latter explaining why an annotation is what it is. In this model, we use “pedigree” to refer only to expert commentary. Pedigree might be attached to almost any annotation. Need better word.

23. Siemens23 independent variables {1} Scanner variables –ModalityModality –Hardware type (field strength, detector number, etc.) –Slice thickness, spacing, exposure Subject variables (Age, Sex; phantom; species; vitality; country of scan; organ scanned; time between scans in scan-series)organ scanned scan-series Disease variables (e.g. Referral criteria; outcomes) Reader variables (incl. inter-reader variation) Algorithm variables repetition method (incl. intra-reader variation)repetition method Part of AVT  AVT studyAVTAVT study

24. Siemens24 scan (data type) {1} Has special case Modality-type scan, e.g. –CT scan –MR scan –PET scan –PET-CT (fused data) scan –Nuclear scan –Ultrasound (3D, 4D) scan –Xray (e.g. mammogram) –Etc. ? Organ-type scan, e.g. –Liver scan –Lung scan –Brain scan –Breast scan –Etc. Subject-type scan, e.g. –Patient scan –Healthy subject scan –Animal scan –Phantom scan –Post-mortem scan? Derived scan, e.g. –Translated/rotated scan –Injected-noise scan –Simulated scan (derived from statistical characteristics of real scans? derived from examples?) –Manipulated scan Includes hybrids, such as simulated tumors added to images of healthy patients. Part of AVT  AVT study  independent variablesAVTAVT studyindependent variables Part of AVT  conduct AVT study  curate collectionAVTconduct AVT studycurate collection Scan: one logical image, whether 2D, 3D, or 4D. The special cases below are only intended to note some of the DICOM attributes that are relevant to AVT scans. Each special case requires certain parametric differences in how annotation tasks are performed.

25. Siemens25 repetition method {1} Has special case Rescan before disease progresses (e.g. minutes, days) Re-reading same image after “forgetting” Reading derived images Part of AVT  AVT study  independent variablesAVTAVT studyindependent variables

26. Siemens26 comparisons {1} Includes attributes to compare data organization for inputs and outputs of comparisons Includes, optionally comparisons to nominal ground truth annotations {1} diameter comparisons {1}diameter comparisons volume comparisons {1}volume comparisons registration comparisons {later} alignment comparisons {later} description comparisons {later} More, TBD Part of AVT  AVT studyAVTAVT study Part of AVT  conduct AVT study  compare annotationsAVTconduct AVT studycompare annotations ? Reqt: Job batching

27. Siemens27 diameter comparisons {1} Includes, optionally RECIST difference (1D) {1} WHO difference (2D) {2?} Wolfe difference (3D) {later} normalized RECIST difference {1} normalized WHO difference {2?} Normalized Wolfe difference {later} Part of AVT  AVT study  comparisonsAVTAVT studycomparisons Part of AVT  conduct AVT study  compare annotations  comparisonsAVTconduct AVT studycompare annotationscomparisons

28. Siemens28 volume comparisons {1} Includes, optionally volume difference {1} abs. of volume difference {1} normalized volume difference {1} abs. of normalized volume difference {1} volume overlap ratio {later} over-segmented ratio {later} (what is this???) under-segmentation ratio {later} surface distance (RMS, 75 th percentile, maximum) {later} smoothness difference {later} shape difference {later} connectedness difference {later} Part of AVT  AVT study  comparisonsAVTAVT studycomparisons Part of AVT  conduct AVT study  compare annotations  comparisonsAVTconduct AVT studycompare annotationscomparisons

29. Siemens29 statistical methods {1} Includes, optionally mean {1} Bias {1} SD (Standard Deviation) {1} CV (Coefficient of Variation) {later} linear regression {later} correlation {later} Kruskal-Wallis analysis multiple comparison {later} Wilcoxon signed rank test {later} sensitivity, specificity {later} PPV (positive predictive value), NPV (negative predictive value {later} STAPLE (boundary or classification agreement metric, pixel by pixel???) {later} ROC (Receiver Operating Characteristic) curve (efficiency of classification) {later} FROC(efficiency for detecting targets) {later} Bland-Altmann charts {1} Box-and-Whisker charts {1} ANOVA methods {1} outlier criteria {1}outlier criteria More, TBD Part of AVT  AVT studyAVTAVT study Part of AVT  conduct AVT study  analyze annotation variabilityAVTconduct AVT study analyze annotation variability

30. Siemens30 outlier criteria {1} Includes, optionally Outlying data points {1} –Scaling by interquartile range (IQR) {1} –Scaling by standard deviations (SD) {1} –Cutoffs (e.g. 25 th percentile, ±2 SD) {1} Outlying sub-distributions (means and SDs) {2} –Scaling by SD of means {2} –Scaling w.r.t. Levene test? {later?} –Scaling w.r.t. Bartlett test? {later?} –Scaling w.r.t. ANOVA test? {2?} –Scaling w.r.t. Kruskal-Wallis test? {later?} Part of AVT  AVT study  statistical methodsAVTAVT studystatistical methods A sub-distribution is a distribution of values of a dependent variable drawn from a subset of the data, (or, equivalently, a sub-group of the cases), typically selected by a query that restricts the ranges of the independent variables, e.g. “just the women over age 65”. If a sub-distribution has significantly different aggregate statistics from the remainder of the collection, it may indicate a source of variation.

31. Siemens31 study representation {1} These are AVT-specific details Includes Annotation labeling plan {1} –Need labels on annotations indicating their roles in the study, so that AVT can find and process them. Measurement and statistics representation (“data cube” schema) {1} –Specifies how the independent and dependent variables, comparisons, and statistical calculations are organized. Data views (e.g. image views, table views, chart views) {1} –How the data appears in the user interface of AVT’s MVT. Part of AVT  AVT studyAVTAVT study

32. Siemens32 example AVT studies {1} Has special case RIDER Lung Nodules Study(?) {?} CDRH Phantom Tumor Reader Study {1?}CDRH Phantom Tumor Reader Study RadPharm Study {2?}RadPharm Study AVW Platform Liver Tumor Study {?}AVW Platform Liver Tumor Study U. Maryland Injected Noise Study {?} More, TBD Template for Example AVT Study Is special case of AVT  AVT studyAVTAVT study

33. Siemens33 RadPharm Study {2?} (draft) Context: measuring tumor change in drug clinical trials Relationship to nominal ground truth: calculate, then known Goal: validate a tumor segmentation algorithm Annotation task: annotate tumor, esp.annotate tumor measure diameters: RECIST, WHO segment tumor, esp.segment tumor –3D segmentation, volume Source of scan data MICCAI Data? FDA Phantom Tumor Data? Independent variables: Scanner variables: TBD Subject variables: TBD Disease variables: TBD Reader variables –N human readers –nominal ground truth Annotations performed in AVT –Inter-reader variation Algorithm variables –Semi-automatic segmentation performed off-line and transcoded from DICOM SR to AIM repetition method: intra-reader variation, more TBDrepetition method Fallback plan: If semi-automatic algorithm not available in time, just study inter-reader variability ComparisonsComparisons, esp. comparisons to nominal ground truth annotations volume comparisons, esp.volume comparisons –volume difference –abs. of volume difference –normalized volume difference –abs. of normalized volume difference –Not interested in volume overlap measurements diameter comparisons, espdiameter comparisons –RECIST difference (1D) –normalized RECIST difference Statistical methodsStatistical methods, esp. Mean bias SD (Standard Deviation) Bland-Altmann Plot ANOVA methods study representationstudy representation: TBD Special case of AVT  AVT study  example AVT studiesAVT AVT study example AVT studies

34. Siemens34 CDRH Phantom Tumor Reader Study {1?} (very early draft) Context: analyzing sources of variation in tumor imaging nominal ground truth: known General Goal: exploration of reader variation annotation task: annotate tumor, esp.annotate tumor manual RECIST and WHO annotations semi-automatic volume segmentation Source of scan data: FDA Phantom Thorax nodule data independent variables CT Acquisition parameters: Exposure, Pitch, Collimation, Slice Thickness, Reconstruction Kernel Scanner variables: CT: Exposure, Pitch, Collimation, Slice Thickness, Reconstruction Kernel Subject variables: Phantom; thorax Disease variables: (Nodule characteristics: Attached, unattached, spherical, elliptical, lobulated, spiculated, random) Reader variables: Human –Using AVT annotation tool –Inter-reader variation –Etc, TBD Algorithm variables: TBD repetition method: repositioning, intra-reader variationrepetition method Comparisons comparisons to nominal ground truth annotations –RECIST and WHO diameters and volume known by construction of phantom. diameter comparisons, esp. RECIST, WHOdiameter comparisons volume comparisons, esp.volume comparisons –Tumor 3D segmentation and its volume –Interested in volume overlap measurements? –How do(es) algorithm(s) compare to human readers? statistical methods Bias Mean error variance Prioritize outliers by measuring in multiples of Standard Deviation or Inter-quartile range Box-and-whiskers plots Bland-Altmann Plot? See Chenyang’s team’s papers? study representationstudy representation: TBD Special case of AVT  AVT study  example AVT studiesAVT AVT study example AVT studies

35. Siemens35 AVW Platform Liver Tumor Study TBD Special case of AVT  AVT study  example AVT studiesAVT AVT study example AVT studies

36. Siemens36 Template for Example AVT Study Context: nominal ground truth: [ unknown | known | calculating] General Goal: [ exploration | validation | algorithm improvement] annotation task annotation task to be analyzed scan annotation task scan-series annotation task Source of scan data independent variables Scanner variables Subject variables (Age, Sex; phantom; species; vitality; country of scan; organ scanned)organ scanned Disease variables (e.g. Referral criteria; outcomes) Reader variables, incl. inter-reader variation Algorithm variables repetition method: [Repositioning | Re-reading | derived images], incl. intra-reader variationrepetition method Comparisons attributes to compare comparisons to nominal ground truth annotations volume comparisons diameter comparisons registration comparisons alignment comparisons description comparisons statistical methods Mean bias SD (Standard Deviation) CV (Coefficient of Variation) linear regression correlation Kruskal-Wallis analysis multiple comparison Wilcoxon signed rank test sensitivity, specificity PPV (positive predictive value), NPV (negative predictive value STAPLE (boundary or classification agreement metric, pixel by pixel???) ROC (Receiver Operating Characteristic) curve (efficiency of classification) FROC(efficiency for detecting targets) Bland-Altmann charts Box-and-Whisker charts ANOVA methods outlier criteria study representationstudy representation: TBD Special case of AVT  AVT study  example AVT studiesAVT AVT study example AVT studies Copy and modify this template to describe your study. Follow links for relevant parts of business use-case model.

37. Siemens37 install image processing algorithm {2} Includes, optionally install c++ image processing algorithm in a scene graph libraryinstall c++ image processing algorithm Install algorithm written in another programming language in a scene graph library Configure AVT to load custom scene graph library Configure a scene graph pipeline to implement a (new) image processing algorithm Part of AVTAVT

38. Siemens38 install C++ image processing algorithm {2} Includes Embed image processing algorithm in a subclass of a suitable XIP scene graph component class Compile and link scene graph component into a binary library (e.g. a DLL in Windows) Part of AVT  install image processing algorithmAVTinstall image processing algorithm AVT provides an example to copy and modify.

39. Siemens39 customize annotation tool {2} Tailor the annotation tool to fit the intended AVT study, thereby eliminating extraneous variability caused by using a too-general tool. Includes, optionally develop IA user interface components to fit AVT study develop IA scene graph to fit AVT study Part of AVTAVT

40. Siemens40 customize measurement variability tool {2} Customize MVT to support the particular kinds of calculation and exploration needed for the intended statistical scan. For data exploration, typically means adding new modules to the growing bag of tricks, and perhaps configuring the menus to conveniently access frequently-used ones. For the actual validation, may involve writing new components that pull together chosen analysis methods to present the results. Includes, optionally Develop R package to fit MVT/R interface Load R package into MVT Configure MVT menus Configure MVT tabcards or write new ones Configure MVT report generators or write new ones Configure MVT table views or write new ones Configure MVT image views or write new ones Configure MVT image scene graph Part of AVTAVT AVT supplies an example to copy and modify

41. Siemens41 conduct AVT study {1} Includes, optionally curate collection {1}curate collection manually do annotation task specified in AVT study {1}annotation task algorithmically annotate scans {2}algorithmically annotate scans compare annotations {1}compare annotations analyze annotation variability {1}analyze annotation variability Part of AVTAVT

42. Siemens42 curate collection {1} Includes Define criteria for collecting scans, {1} e.g.scans –Age, Sex, Referral criteria, Hardware type (field strength, detector number, etc.) –Slice thickness, exposure –Country of scan, scan subject typesubject type Define criteria and instructions for annotating scans {1} –Need automated query support for selecting based on these criteria Add scans to collection {1} (optional) create scan-series and add to collection {later}scan-series add vetted patient demographics (maybe from DICOM header) {1} add vetted clinical data (such as outcomes) {1} add pedigree information to whole collection {2}add pedigree information certify nominal ground truth {1}certify nominal ground truth Part of AVT  conduct AVT studyAVTconduct AVT study

43. Siemens43 certify nominal ground truth {1} nominal ground truth is established on scans and scan-series, typically by statistically analyzing multiple attempts to estimate that annotation, manually and/or automatically, and recording the estimate as a new annotation on the relevant case. nominal ground truth can also be associated with histology, diagnoses, pathology, and follow-up includes, optionally Add an annotation to an annotation indicating that it is the most recent best estimate of the nominal ground truth for that annotation type, for the object being annotated. Add a pedigree note to a collection indicating that a certain tag on annotations in that collection marks it as nominal ground truth for a specified purpose. The note also documents how the nominal ground truth was determined. Part of AVT  conduct AVT study  curate collectionAVTconduct AVT studycurate collection Need version tracking information to handle cases where nominal ground truth evolves.

44. Siemens44 algorithmically annotate scans {2} Includes Specify algorithms to run to perform annotation task specified in AVT study {2}annotation task configure labeling scheme for generated annotations, so they can be found later for analysis {2} Specify sampling scheme for choosing which scans in collection will be annotated {later} –Sampling from space of Execute specified algorithm on chosen scans {2} Review results of algorithm execution (did it do what I meant?) {2} Part of AVT  conduct AVT studyAVTconduct AVT study

45. Siemens45 compare annotations {1} Includes Execute comparisons specified in AVT studycomparisons Review comparisons (did it do what I meant?) Part of AVT  conduct AVT studyAVTconduct AVT study

46. Siemens46 analyze annotation variability {1} Includes Calculate statistics specified in AVT study {1}statistics review statistics results {1}review statistics results Generate variability reports specified in study representation of AVT study {1} study representation –E.g. statistics reports with some typical images Part of AVT  conduct AVT studyAVTconduct AVT study

47. Siemens47 review statistics results {1} Review data, interactively, as specified in study representation of AVT study.study representation Includes, optionally Prioritized lists of outliers {1} Prioritized lists of outlying sub-distributions {2} Graphs {1} Tables {1} Drill-down to details of outliers –view scans {1} –sub-distributions {2} More, TBD Part of AVT  conduct AVT study  analyze annotation variabilityAVTconduct AVT studyanalyze annotation variability

48. Siemens48 Backup slides No! Don’t Back Up!

49. Siemens49 Concepts not yet covered adequately in model From UCLA team Algorithms involving functional measures (Dynamic Contrast Enhanced MRI, for example) Other Oncology examples (talk about later) Morphological change Suggestions solicited for how to incorporate these in model.