1. Feature Aligned Volume Manipulation for Illustration and Visualization Carlos D. Correa, Deborah Silver Rutgers, The State University of New Jersey Min Chen University of Wales, Swansea, UK

2. Motivation Nucleus IncAntonio Scrantoni and Paolo Mascagni, 1833. U.S. National Library of Medicine Hand-drawn illustrations often include manipulating parts of an object: They often contain cuts They allow feature sensitive operations They often represent virtual operations (do not necessarily conform to reality)

3. Motivation (cont.) We refer to such manipulation as lllustrative Deformation Priority to interactivity, operatability and quality As opposed to physically-based deformation, this can be thought of as a top-down approach This type of deformations provides an intuitive depiction of internal structure. It serves as an abstraction of different stages of a procedure, e.g. a surgical operation. It is useful in surgery illustration/planning, education, and as a visualization tool in general.

4. Feature Alignment Traditional volume deformations are continuous and treat volumes as an homogeneous collection of points [Westermann et al. 2001, Rezk- Salama et al. 2001] McGuffin [2003] introduced 3D widgets with pre-computed segmented data to allow feature sensitive manipulation of volumes. Can this approach be extended to direct volume rendering? Recent approaches allow the definition of cuts [Correa,2006]. However, cuts appear flat as no semantics are introduced axis alignment Cuts in general are difficult to model in computer graphics. Require costly re-tessellations. This is further complicated when cuts have to be aligned with certain features.

5. Axis Alignment Treating volumes as homogeneous collections of voxels leads to axis alignment of cuts. Difficult to see features of interest

6. Goal To render deformations while preserving features of interest, by aligning cuts to a given: Distance from surface surface alignment Feature based on segmentation segment alignment CT Dataset Illustrative Deformation Illustration

7. Rendering Pipeline (axis aligned cuts) Select operator Sample and Deform OPERATORS TRANSFORMATION Adapted from Correa et al. 2006 via inverse space warping

8. Feature-Aligned Rendering Pipeline Select operator Apply mask Sample and Deform OPERATORS MASKS TRANSFORMATION Adjust opacity/lighting according to alignment Definition of features using a volumetric mask

9. Operators Inspired by surgical tools and procedures Generic: they can be applied to any dataset Defined as a 3D texture. Iconic representations are obtained when applied to a volumetric cube (or cylinder) OPERATORS

10. Volume Transformation Sample Apply transformationEstimate normalCompute lighting Subject to Alignment Mask Sample and Deform TRANSFORMATION

11. Modeling Deformation and Cuts Forward transformation is simple but limited for volumes undersampling unless space between points is interpolated (for cuts, it requires re-tessellation) Inverse transformation. Solves sampling problem, but discontinuous deformations are not a 1:1 mapping. Forward Transformation Backward Transformation. Note introduction of special value to model discontinuity

12. Modeling Feature Alignment Define a smooth mask M Binary masks may cause aliasing M(p) >= 0.5 p is non operatable M(p) < 0.5 p is operatable Three cases for inverse transformation Not affected by mask: apply inverse mapping Point inside mask: it is not transformed Point outside mask but maps back into mask: empty space left by feature

13. Definition of Features Distance-based vs. Segmentation based (1) Surface Alignment: Features are defined with the shape. Distance field of surface of object defines a series of shells, which define features. Useful when no segmentation is available. MASKS M = 0.0 M = 0.5 M = 1.0 Outer shell ( ) Features outer surface (DT = ) Features interior

14. Mask Definition (2) Segment Alignment: Mask is already defined as a segment Usually segmentations are discrete (binary for 1 segment), For proper rendering without aliasing, a smooth definition is required: Using a smoothing operator Using Distance Field of segmented part M = 1 M = 0.5 M = 0.0

15. Rendering and Lighting Cuts are now along a certain feature. For surface alignment, a new surface appears. Pre-computed normals are not necessarily perpendicular to that surface For segment alignment, gradient already defines almost correctly one surface. However, it cannot define properly the underside of the cut

16. Normal Adjustment: Surface Alignment Normals are oriented depending on density, not necessarily aligned with surface shell. This can be fixed by blending of normals T * T DT

17. Normal Adjustment: Segment Alignment Normals on the underside of a cut point in the opposite direction T * T - T

18. Results (1) Peeling of Skin AXIS SURFACE SEGMENT Original Dataset

19. Results (2) Frog Dissection AXIS SURFACE SEGMENT Original Dataset

20. Results (3) Hand Surgery AXISSURFACESEGMENT Original Dataset

21. Results (4) Forefoot Retractor AXIS SURFACE SEGMENT Original Dataset

22. Implementation Details Based on discontinuous displacement mapping [Correa et al. 2006], using texture based volume rendering Operators are stored as 3D textures (size is much smaller than size of dataset). Feature Mask is also stored as a 3D texture Interactive results (Pentium XEON 2.8GHz Quadro FX 4400 (512 MB): d =1

23. Video

24. Applications Medical and Biological Illustration. Operators are metaphors of the tools used in dissection Surgical Planning Manipulation of operators allows the generation of deformations and cuts in various stages of a procedure Improved Visualization Cutaway views with arbitrary cut geometry Focus+Context, Distortion Lens

25. Conclusions Volume deformation techniques often treat volumes as homogeneous collection of voxels. When modeling cuts and breaks, they appear to be axis aligned, which results in decreased realism and limited use. It is possible to extend volume deformation to align cuts with certain features of interest. These can be defined as shells of the surface of the object using the distance transform, or as the product of segmentation Feature alignment can be implemented efficiently on commodity hardware. Proper handling of cut information to reduce aliasing, and adjustment of normals near the surfaces of cuts are necessary to produce high quality rendering of cuts.

26. Future Work Merge illustrative deformation with illustrative rendering NPR techniques can be used to emphasize new surfaces due to cuts or to exaggerate deformation (e.g. rendering of stress lines) Inclusion of rigid constraints for enhanced deformations, collision avoidance. More intuitive user interface and manipulation widgets to create and place operators.

27. Thanks Acknowledgements Volumetric datasets are courtesy of Lawrence Berkeley Laboratory, UNC Chapel Hill, University of Iowa, U.S. National Library of Medicine, Viatronix Inc. and Vienna University of Technology. The illustrations are courtesy of U.S. National Library of Medicine and Nucleus Medical Art, Inc. We want to thank Dr. Stanley Trooskin, Dr. Sid Roychowdhury and Dr. Marsha Jessup for valuable input on surgical and medical illustration. Further Information http://www.caip.rutgers.edu/~cdcorrea

28. Thanks