1 Multimodal Registration Clinic “All Things Registered” I.Theory & Tool Overview II.Live Demo of Registration in 3D Slicer III.Open Discussion: what’s on your wishlist? Dominik S. Meier, Ph.D.

2 Multimodal Registration Clinic “All Things Registered” I.Background: Registration Theory II.Image Registration Tools in 3DSlicer (v.3.5) III.User-support, Training & Documentation

3 Part I : Registration Theory Background

4 Registration Concept Image registration seeks to bring two or more images into anatomical alignment. In mathematical terms: Image registration transforms multiple images into one coordinate system. This is necessary to compare or integrate/fuse the data obtained from different measurements. Purpose –change detection (small regional change, subtraction imaging) –atlas building (normalize for individual anatomical idiosyncrasies) –distortion or motion correction (different protocols or sensors) –protocol matching (e.g. sagittal into axial) –group analysis (anatomical reference space) –Navigation, patient to reference, surgical planning –atlas-based morphometry (estimates from atlases)

5 Quality Assessment speed robustness precision Cross-sectional: group analysis Large numbers of images processed in a fully automated fashion Limited review and editing Processing speed less relevant than reliable performance across all images in the study speed robustness precision speed robustness precision IGT: Supervised, so robustness can be supplemented with user interaction Speed and precision are critical Precision/error estimates also critical Longitudinal: Change Assessment Smaller study size compared to cross-sectional Some review and editing Precision determines detectable change, is the key criterion

6 Registration Concept Image Registration has 4 main components –spatial transform: a model equation that describes how the two images should be aligned. –similarity metric: a criterion that defines how well the images are aligned, i.e. what constitutes a “good match” (cost function). –optimizer: an iterative exploring of the realm of possible solutions, looking to find the best one (search algorithm). –interpolator: an algorithm to apply the transform and build the newly aligned images (resampling). It is important to know what these 4 are and what they do to achieve the best possible result. We will briefly discuss each in turn. metric optimizer interpolator transform

7 Transform f(x,y,z) analytical model deformation/vector field invertable if linear and isomorphic e.g. translation + scale : x’ = x + 10 y’ = y - 3 z’ = 1.05 z + 5 z x y z x y metric optimizer interpolator transform

8 translation rotation scaling shearing affine transform 12 DOF 3 x similarity transform 9 DOF rigid transform 6 DOF Linear Transform: DOF shift transform 3 DOF metric optimizer interpolator transform

9 Non-linear Transform: DOF rigid affine ≈ 50 Million distances and angles change proportionally Motion guided by single equation. many image control points move independently, i.e. no single equation type/nameDOFshapedescriptioncaveat intact globally distorted careful with volumetry non-rigid locally distorted 3-pt Bspline grid: 5-pt Bspline grid: full image: rigid body motion, distances and angles preserved. Motion guided by single equation. distances and angles change dis- proportionally will not match global scale distortions DOF matching required: careful not to use excessive DOF and thus normalize/remove the differennces you want to measure metric optimizer interpolator transform

10 Coordinate Systems: the hidden Xform Each image has a basic (linear) transform that connects the digital image grid to the physical world. This is usually part of the image header. RAS : right - anterior - superior RL L S I AP vox2ras z x y Image to Image Transform z x y vox2ras RL L S I AP metric optimizer interpolator transform

11 Registration Glossary DOF fixed & moving image linear vs. nonlinear rigid vs. non-rigid forward vs. backward/inverse mapping multi-modality registration parameters affine, similarity, B-spline,warping pivot point, image origin, coordinate system pixel space vs. raster space voxel size & anisotropy intensity vs. feature-based registration

12 Metric defines how well the two images align qualitative: you; visual assessment quantitative: sum of intensity similarity sum of topographical features –Intensity Difference –Intensity Ratio –Cross-correlation –Mutual Information metric transform optimizer interpolator speed robustness precision –same subject, same contrast –same subject, different contrast –same subject, different modalities –different subject, same contrast –different subject, different contrast

13 Similarity Metric metric optimizer interpolator transform speed robustness precision same subject, same contrast same subject, different contrast same subject, different modalities different subject, same contrast different subject, different contrast Difference Ratio Correlation Mutual-Info

14 Interpolation Applies the transform to the image and generates a new volume. back from physical space into the image grid: the newly calculated position of an image voxel will not fall exactly onto a grid-point. Therefore its value is determined by the intensities of neighboring pixels. This process is known as interpolation. T metric transform optimizer interpolator

15 Interpolation nearest neighbor: picks value of the voxel nearest the point coordinates + fast - coarse + MUST use for label-maps linear: picks a weighted mean of neighboring voxel intensities + stable, default - introduces blurring cubic, sinc: fits a non-linear model to estimate the intensity + sharper, less blurring - slower - may introduce spurious outliers near edges (e.g. negative intensities) original nearest neighbor linear cubic Example of a T1-weighted brain MRI, rotated by 6 degrees. Showing magnified sagittal view of cerebellum and midbrain. nearest neighbor: note the false contouring around the pons linear: note the blurring cubic: less blurring than linear metric optimizer transform interpolator

16 Optimizer manual: you automated: iterates between metric and transform exhaustive search gradient-based search annealing/stochastic schemes metric transform optimizer interpolator

17 Optimization rotation x rotation y similarity optimum local maxima (suboptimal solutions) The algorithm moves/wiggles one of the images around trying to find the best match, according to the similarity metric. It does so in increments from its current position, evaluating if the new position is better than the old one. A “local maximum” is a position around which all nearby changes appear worse, but farther away there is a better solution available. Optimization algorithms often get “stuck” in such positions. Depending on the difference in contrast between the two images, the similarity metric employed, and the amount of initial misalignment, this is more or less likely to happen to an automated registration. metric optimizer transform interpolator

18 Part I : further topics the main components: transform, similarity metric, optimization, interpolation coordinate systems: physical vs. image space, RAS vs. LPI etc. relevant image meta-data: coordinate system, axis orientation, image/CS origin, voxel size overview of scenarios and their different challenges: image pairings, DOF, multi- modal, intra/inter-subject etc. how to evaluate a match: tools & concepts common mistakes to avoid: inappropriate DOF, overly flat similarity metric, CS inconsistencies, FOV discrepancies, wrong interpolation, insufficient search (sample points, multi-scale, DOF scale-space) Troubleshooting guide: insufficient match - what next? Parameter modification, DOF change, initial alignment assist, fiducial help, ROI masking (e.g. skull stripping)

19 Part II : 3DSlicer Registration Tools

20 Overview of Registration Tools in 3D Slicer Registration Main Modules Registration Auxilary Modules Driver: manual intensity surfaces fiducials segmentation DOF: ~ 10 3 Support for: fiducials ROI definition mask building & editing resampling visualization/evaluation

21 Registration Modules: Manual - Interactive ideal for initial alignment immediate feedback in 3D fail-safe if automated registration fails or is too slow

22 speed & precision mask starting point robustness contrast & content DOF presets Registration Modules: Intensity Affine

23 Registration Modules NEW: Multi-resolution Affine Method of choice for robustness Supports masking

24 presets DOF image contrast constraints speed & precision Registration Modules: Non-rigid BSpline

build 2 fiducial lists with 3 or more points each 2.click: Apply 3.supports translation to similarity (3-9 DOF) very fast (< 1 sec) Example: inter-subject knee registration Registration Modules: Fiducial Registration

26 Input: 2 surface models click: Apply supports rigid to affine (6-12 DOF) very fast (~ 1 sec) Registration Modules: Surface Registration

27 Aligns brain image along midsagittal plane and places anterior-posterior commissures on a horizontal line Input: 2 fiducial pairs defining anterior & posterior commissure midsagittal plane Registration Modules: AC-PC alignment

28 Non-rigid Cortical surface alignment based on WM/GM segmentation warps based on attributes derived at gyrus crown, sulcal root and ventricle corners. Registration Modules: HAMMER Cortical Surface Matching

29 Non-rigid registration based on optical flow principle considered very robust Registration Modules: Demons - Warping

30 define and edit in 3D use for masking registration Auxilary Tools: ROI Module

31 define and edit in 2D or 3D use for masking registration where masking is not (yet) explicitly supported increase speed and robustness Auxilary Tools: Extract Subvolume

32 Auxilary Tools: Visualization Checkerboard Filter to evaluate registration quality (particularly for areas with high contrast/edges) Subtraction Images to evaluate overall alignment Subtraction Subtraction Images to evaluate regional changes and alignment

33 Editor create and manipulate binary label maps from grayscale images fix labelmaps returned by other modules (skull stripping, Otsu’s etc.) Auxilary Tools: Editor

34 Auxilary Tools: Resampling Apply a transform to a scalar or vector (e.g. DTI) volume select tailored interpolation scheme (nearest neighbor, linear, sinc, b-spline) correctly reorients vector data

35 define and edit in 3D organize in fiducial lists use ordered lists of fiducial pairs for registration Auxilary Tools: Fiducials

36 Part III : Registration User Support Training Documentation

37 Registration Case Library Brain Other A growing collection of example registration problems, complete with image data, tutorial, solution, discussion and a parameter preset file that can be loaded into 3DSlicer.

38 Registration Case Library

39 Registration Case Library:Tutorials Guided/narrated Video Tutorials Step-by-step Powerpoint/PDF Tutorials

40 Call For Datasets "if you have a registration problem that is not yet covered in our library, send us your case: we will post it along with our best registration solution/strategy. If you agree to the posting of the anonymized image data, you get a free registration, the user community gets a new example case. Everybody wins.” What We Will Do seek the best possible registration obtainable with the most recent version of 3DSlicer post the anonymized image as a new case in our Slicer Registration Case LibrarySlicer Registration Case Library post the exact workflow used to obtain the shown solution registration will be posted alongside the data as a guided step-by-step tutorial the parameters for successful registration will also be posted as a loadable custom "Registration Preset" file that you can load directly into Slicer and apply on your data if you can provide us with fiducial pairs or other criteria that define a good registration, we will use them in optimization efforts. the registration objective & background, main challenges and strategy recommendations will be posted an acknowledgment of your lab as the data source is posted, if desired with a link to your institution and/or related research papers

41 Registration Case Library:Tutorials Slicer Training Compendium: Tutorials for all skill levels

42 Acknowledgments National Alliance for Medical Image Computing NIH U54EB Neuroimage Analysis Center NIH P41RR Surgical Planning Laboratory, Brigham and Women’s Hospital