J OURNAL C LUB : Yang and Ni, Xidian University, China “Multimodality medical image fusion based on multiscale geometric analysis of contourlet transform.” Jul 21, 2014 Jason Su

Motivation Visualization of multiple image modalities or contrasts is difficult – Side by side comparisons are often not precise – Flipping back and forth helps to pronounce changes but some modalities may have no structural landmarks Beginning to collect time-resolved “MP-nRAGE” with view-sharing methods – What is the best way to visualize such data, esp. for thalamic nuclei? – Can we do something else other than fitting a T1 map?

Goal: Image Fusion The combination of multiple images into one while preserving the important information from each Common examples: – fMRI overlays on structural images – Segmentation overlays – Nuclear medicine overlays – HDR photography Compared to quantitative imaging, the goal is to achieve a pleasing effect to the eye instead of fitting to a model – Thus there are many possible algorithms and no necessarily “correct” way to do things

Background: Types of data fusion Signal level, pixel level – Image fusion, e.g. averaging, SOS, MIP – Region-based: consider neighborhood around current pixel Feature level – Label fusion segmentation: combine multiple candidate labels to identify features Decision level – Image biomarkers

Gaussian and Laplacian Pyramid GP: Successively blurred and downsampled versions of the image – Gives scale of features in the image LP: take differences between Gaussian levels – Gives information about edges of varying widths Pyramids are multiresolution decompositions of images Each level is subsampled by a factor of 2, i.e. each level is an octave

ROLP/Contrast Pyramid Ratio of low-pass pyramid: take ratios between Gaussian levels Contrast = (L-L b )/L b R = L/L b = C + 1

Background: The -lets Discontinuities destroy the sparsity of a Fourier series, the Gibbs phenomenon Wavelets – are localized and multi-scale – Perform well in 1D, but poor sense of orientation for 2D – Only horizontal, vertical or diagonal How to better represent a 2D image? – Want multiresolution, localization, critical sampling, directionality, anisotropy Curvelets – Candés et al. – Developed in continuous domain then adapted to discrete – Optimally sparse representation for smooth 2D functions except for a discontinuity along a curve – Models wave propagation Contourlets – Do and Vetterli – Developed in discrete domain Pointillism-like

Background: Curvelet Decomposition Directional filter bank. (a) Frequency partitioning where l=3 and there are 2 3 = 8 real wedge-shaped frequency bands.

Methods: Algorithm Take 2 input images, how to combine them? Tale contourlet transform of each – Each level apears to gain a factor of 2 in angular resolution Yang’s decomposition – How does this effect the quality?

Algorithm: Lowpass Subband Treat the lowest level of pyramid differently – This is a tiny thumbnail of the original information – Higher-level detail is added to this to reconstruct the whole image 2 modes of operation: selection or averaging Choose based on a threshold criterion: salience – If the correlation between the input windows in a 3x3 patch in curvelet space is above a threshold -> weighted averaging – Else choose the one with more energy (sum sq. over window) Averaging is only done at a fixed alpha blend amount – Not variable dependent on data A bit ad hoc in that there are many unspecified preset tunable parameters: thresholds, blend factors – They can be optimized for nuclei

Highpass Subband Algo Contrast = (L-L b )/L b = L h /L b – Ratio of a high level curvelet coefficient to lowest level 1.Compute contrast as above, L b comes from the pixels in the lowest level that contribute to the highest level 2.Blur this to get weighted neighborhood contrast 3.Select coefficients from the image that has the higher value on this metric, i.e. the one that has more local contrast “Using contourlet contrast, more dominant features can be preserved precisely at all the resolution levels”

Reconstruction Take the inverse curvelet transform of the blended pyramid

Methods Test cases – CT-MR – Gd and T2w – PD and T1w Compared against existing methods: average, PCA, wavelet maximum Metrics – Standard deviation – image variability – Entropy – how much information is in the image – Overall cross entropy – how close are the distributions, is information preserved in the fused result? – Spatial frequency – amount of energy energy in high frequencies Only looking at horizontal and vertical freqs. – Correlation – how similar is the fusion to the inputs

Results: Metrics Proposed algorithm generally shows to have more variability and capture more information from the inputs

Notes PCA table values seem off? There is a Matlab implementation of curvelets by the creators How to handle multiple image fusion?