N-Dimensional Presentation States Progress and Paths from WG 11 Michael Aratow, M.D., FACEP Nicholas F. Polys Ph.D. Web3D Consortium

Convergence in Medical IT New opportunities in improving quality and reducing cost of care: Standards enable new analytics – Reproducible Interactive presentations and reports – Leveraging the web multimedia ecology (W3C) – Semantics (FMA, SNOMED CT) Standards enable new levels of care – Consistent communication between providers and organizations

Critical Path: Medical Imaging & Informatics Interoperability challenges for the presentation of medical images are well-recognized – Accessibility outside the radiology suite – Procedural and Cognitive Education – Informed Consent – Custom prostheses – Surgical Planning – Radiation Therapy and Planning Why risk a new wheel? – Proven and unencumbered scenegraph abstractions yield a quantum jump: the ‘Hopscotch Strategy’

A decade of experience with 3D and the web in an ISO environment – Aligning with proven web delivery and presentation technologies Leads to the obvious reason these standards bodies are working together: – Existing DICOM requirements overlap existing X3D functionality – ISO standardization – Open technology The Hopscotch Strategy

The Web3D Consortium A not-for-profit organization that develops and maintains ISO standards for the description and transmission of 4D data over telecommunication networks. The X3D, H-Anim and VRML standards specify both a file format and a runtime specification that are open and royalty-free

Web3D & X3D Proven Innovation (see web3d.org): Real time interactive graphics – Lights, cameras, text, meshes, textures and materials – Animations, sensors, volumes & style Interoperability with web – distributed resources referred via URL/URI – E.G. images, sounds, videos, models, worlds Durability – ISO Specifications: Abstract Spec (scenegraph) Encodings Bindings H-Anim

Highlights Abstract scenegraph with multiple encodings and API bindings Well-defined runtime semantics ISO (SC-24), Royalty-Free Multiple open-source and commercial implementations (approx 24) demonstrated capability for medical applications covers significant majority of DICOM requirements remainder being specified through WG11 work item

US ARMY TATRC Contract Successfully completed in 2007 MedX3D Draft Specification available to members Implementations: – H3D.org – Open Source Haptics incl. X3D, MedX3D and VRML – Fraunhofer IGD’s Instant Reality Semantic Scenegraph inline or referenced Volume Rendering Demos

MIP Map Opacity MapISO Surface Composed Volume Segmented Volume MedX3D Volume rendering styles with default fields (TATRC deliverable, H3D)

Web3D + DICOM Recent History Started in WG-17: Web3DC contributed to 3D Surface data structures (compatible with ISO (X3D)) Now primarily in WG11: Scoped Work Item on n-Dimensional Presentation States Web3D Consortium Medical WG presenting specs and royalty-free technology meeting (and exceeding) requirements

Timeline Recap

DICOM: Critical Mass Continue to gather momentum, requirements, alignments with DICOM stakeholders (you!) Web3D Consortium brings a time-proven solution: a standardized scenegraph to meet the requirements of the healthcare enterprise Recognizing the value of consistent renderings across caregivers and stakeholders, the US Army TATRC provided critical support (spec draft and open source implementations) We are pleased to report: Significant progress on this work item!

X3D Interactive Presentations use DNA Basic Interactive Scenes: 2D & 3D Geometry, Appearance, Text, Layers Lighting Sensors Event Utilities Interpolators, Sequencers No PROTOs, No Scripts

Sample Presentation Simple X3D Interactive demos Data from Volume to Surfaces by OSIRIX

6-DOF Camera Animation includes coordination of transformation graph and behavior graph Endoscopy path example: – Timesensor – PositionInterpolator, OrientationInterpolator – Routes [event model]

Specification Plan: Med X3D Profile 1.Specify additional nodes and components for X3D abstract spec 2.Create MedX3D Profile for X3D abstract spec 3.Encoding » Create DICOM encoding of MedX3D Profile » Create Binary encoding of MedX3D Profile

X3D Encodings and Bindings Equivalent scenegraphs can be: encoded in and manipulated by multiple languages

X3D Components and Profiles Components – Families of related nodes Levels – Of support (fields and functionality) Profiles – Well-defined nodesets for vertical application functionality See: AbstractSpecification/index.html

X3D Volume Rendering Component Specified and developed w/ US Army TATRC; two implementations (one open-source) General transfer function Shading styles – Blinn-Phong(including advanced features such as shadows and light scattering) – Cartoon shading – Gooch shading(tone mapped) Enhancement styles (enables enhancement of important features in an easy way) – Boundary enhancement – Edge enhancement – Silhouette enhancement Maximum, Minimum and Average intensity projection Iso-surface rendering Segmented data rendering (render different parts of volume data using different styles)

WG 11 Direction for n-D PS Functional Profile (nodeset): Med X3D – X3D Interactive+ – Volumes & Rendering styles – Cutting/ clipping (i.e. for Multiplanar Reconstruction (MPR)) Scenegraph could be encoded in X3D Binary Encoding scenegraph profile in DICOM

Specification: Include Base X3D Concepts Include some set of components for meeting requirements: – Grouping: Transformation, Switch, LOD – Viewpoints (Perspective and Ortho) – Lights (Directional, Point, Spot) – 3D, 2D surfaces, lines and point geometries including Shape Appearance and Materials Per-vertex data such as RGBA and ImageTexture maps – Annotation (Text, Layers) – Interactivity Sensors Event model, animation Scripting?

Specification: MedX3D Nodes / Components 3D Texture Component Volume Component Annotation Component – Image plane projection & linking – YES – 3D equivalents of Extended (2D) Presentation State objects – YES probably

Specification Progress: Volumetric Presentation Volume Rendering – MedX3D styles have necessary and sufficient coverage Masks, subvolumes & segments – MedX3D voxel data structure and access per WG2 and WG12 requirements Addressing voxels within volume Volume time series UI (Toshiba input) – e.g. Clipping & MPR Lines generate plane Planes 3D volume

DICOM WG 11 + Web3D Medical WG Specification Plan: 1.Med X3D Functional Specification (abstract) » Specify additional nodes and components for X3D abstract spec » Create MedX3D Profile for X3D abstract spec 1.X3D (XML, Binary) encodings exist; DICOM encoding of MedX3D Profile?

Going Forward ISO scenegraph technology yields a quantum leap in health care interoperability Immediate value in cross-platform transmission and rendering of medical image presentations Continue development of DICOM+MedX3D Profile: specification (X3D 3.3) including annotations, implementations, conformance examples

Ways to get Engaged WG 11 IEEE VR 2010 in Waltham, MA this month Web3D Symposium in Los Angeles in July 2010 SIGGRAPH in Los Angeles in July 2010