1. Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University of California at San Francisco, AND Center for Imaging of Neurodegenerative Diseases (CIND) San Francisco VA Medical Center email: ashish.raj@ucsf.edu Webpage: http://www.cs.cornell.edu/~rdz/SENSE.htm http://www.vacind.org/faculty

2. CIND, UCSF Radiology 2 San Francisco, CA

3. CIND, UCSF Radiology 3 Overview  We propose a new image structure called local influence neighbourhoods (LINs)  LINs are basically locally adaptive neighbourhoods around every voxel in image  Like “superpixels”  Idea of LIN not new, but first principled cost minimization approach  Thus LINs allow us to probe the intermediate structure of local features at various scales  LINs were developed initially to address image processing tasks like denoising and interpolation  But as local image features they have wide applications

4. CIND, UCSF Radiology 4 Local neighbourhoods as intermediate image structures Pixel-level Neighbourhood-level 1 2 3 Region-level Low levelHigh level Too cumbersome Computationally expensive Not suited for pattern recognition Prone to error propagation Great for graph theoretic and pattern recognition Good intermediaries between low and high levels?

5. CIND, UCSF Radiology 5 Outline  Intro to Local Influence Neighbourhoods  How to compute LINs? –Use GRAPH CUT energy minimzation  Some examples of LINs in image filtering and denoising  Other Applications: –Segmentation –Using LINs for Fractal Dimension estimation –Use as features for tracking, registration

6. CIND, UCSF Radiology 6 Local Influence Neighbourhoods  A local neighbourhood around a voxel (x 0, y 0 ) is the set of voxels “close” to it –closeness in geometric space –closeness in intensity  First attempt: use a “space-intensity box”  Definition of ,  arbitrary  Produces disjoint, non-contiguous, “holey”, noisy neighbourhoods!  Need to introduce prior expectations about contiguity  We develop a principled probabilistic approach, using likelihood and prior distributions

7. CIND, UCSF Radiology 7 Example: Binary image denoising  Suppose we receive a noisy fax: –Some black pixels in the original image were flipped to white pixels, and some white pixels were flipped to black  We want to recover the original input image output image

8. CIND, UCSF Radiology 8 Problem Constraints  Our Constraints: 1.If a pixel is black (white) in the original image, it is more likely to get the black (white) label 2.Black labeled pixels tend to group together, and white labeled pixels tend to group together original image good labeling bad labeling (constraint 1) bad labeling (constraint 2) likelihood prior

9. CIND, UCSF Radiology 9 Example of box vs. smoothness

10. CIND, UCSF Radiology 10 Example of box vs. smoothness

11. CIND, UCSF Radiology 11 A Better neighbourhood criterion 1.Incorporate closeness, contiguity and smoothness assumptions 2.Set up as a minimization problem 3.Solve using everyone’s favourite minimization algorithm –Simulated Annealing –(just kidding) - Graph Cuts!  A) Closeness: lets assume neighbourhoods follow Gaussian shapes around a voxel

12. CIND, UCSF Radiology 12 A) Closeness criterion in action

13. CIND, UCSF Radiology 13 B) Contiguity and smoothness  This is encoded via penalty terms between all neighbouring voxel pairs p q G(x) =  p,q V(x p, x q ) V(x p, x q ) = distance metric A)Closeness B) Contiguity/smoothness Define a binary field F p around voxel p s.t. 0 means not in LIN, 1 means in LIN Bayesian interpretation: this is the log-prior for LINs

14. CIND, UCSF Radiology 14 MAP can be written as energy minimization  E.g. consider linear system y = Hx + n  Pr(y|x) (likelihood function) = exp(- ||y-Hx|| 2 )  Pr(x) (prior PDF) = exp(-G(x))  MAP can be converted to energy minimization by taking logarithm x est = arg min ||y-Hx|| 2 + G(x)

15. CIND, UCSF Radiology 15 Markov Random Field Priors  Imposes spatial coherence (neighbouring pixels are similar) G(x) =  p,q V(x p, x q )  V(x p, x q ) = distance metric p q  Potential function is discontinuous, non-convex  Potts metric is GOOD but very hard to minimize

16. CIND, UCSF Radiology 16 Bottomline  Maximizing LIN prior corresponds to the minimization of E(x) = E closeness (x) + E smoothness (x)  MRF priors encode general spatial coherence properties of images  E(x) can be minimized using ANY available minimization algorithm  Graph Cuts can speedily solve cost functions involving MRF’s, sometimes with guaranteed global optimum.

17. Graph Cut based Energy Minimization

18. CIND, UCSF Radiology 18 How to minimize E?  Graph cuts have proven to be a very powerful tool for minimizing energy functions like this one  First developed for stereo matching –Most of the top-performing algorithms for stereo rely on graph cuts  Builds a graph whose nodes are image pixels, and whose edges have weights obtained from the energy terms in E(x)  Minimization of E(x) is reduced to finding the minimum cut of this graph

19. CIND, UCSF Radiology 19 Minimum cut problem  Mincut/maxflow problem: –Find the cheapest way to cut the edges so that the “source” is separated from the “sink” –Cut edges going from source side to sink side –Edge weights now represent cutting “costs” a cut C “source” A graph with two terminals S T “sink”

20. CIND, UCSF Radiology 20 Graph construction  Links correspond to terms in energy function  Single-pixel terms are called t-links  Pixel-pair terms are called n-links  A Mincut is equivalent to a binary segmentation  I.e. mincut minimizes a binary energy function

21. CIND, UCSF Radiology 21 Table1: Edge costs of induced graph n-links s t t-link

22. CIND, UCSF Radiology 22 Graph Algorithm  Repeat graph mincut for each voxel p

23. CIND, UCSF Radiology 23 Examples of Detected LINs

24. CIND, UCSF Radiology 24 Results: Most Popular LINs

25. CIND, UCSF Radiology 25 Filtering with LINs  Use LINs to restrict effect of filter –Convolutional filters: – Rank order filter: =

26. CIND, UCSF Radiology 26 Maximum filter using LINs

27. CIND, UCSF Radiology 27 Median filter using LINs

28. CIND, UCSF Radiology 28 EM-style Denoising algorithm Likelihood for i.i.d. Gaussian noise: Image prior: Maximize the posterior: Noise model: O = I + n

29. CIND, UCSF Radiology 29 Bayesian (Maximum a Posteriori) Estimate Bayes Theorem: Pr(x|y) = Pr(y|x). Pr(x) Pr(y) likelihood prior posterior  Here x is LIN, y is observed image  Bayesian methods maximize the posterior probability: Pr(x|y)  Pr(y|x). Pr(x)

30. CIND, UCSF Radiology 30 EM-style image denoising Joint maximization is challenging We propose EM-style approach: Start with Iterate: We show that

31. CIND, UCSF Radiology 31 Results: LIN-based Image Denoising

32. CIND, UCSF Radiology 32 Results: Bike image

33. CIND, UCSF Radiology 33 Table1: Denoising Results

34. Other Applications of LINs  LINs can be used to probe scale-space of image data –By varying scale parameters  x and  n  Measuring fractal dimensions of brain images  Hierarchical segmentation – “superpixel” concept  Use LINs as feature vectors for –image registration –Object recognition –Tracking

35. CIND, UCSF Radiology 35 Hierarchical segmentation  Begin with LINs at fine scale  Hierarchically fuse finer LINs to obtain coarser LINS  segmentation

36. How to measure Fractal Dimension using LINs?  How LINs vary with changing  x and  n depends on local image complexity  Fractal dimension is a stable measure of complexity of multidimensional structures  Thus LINs can be used to probe the multi-scale structure of image data

37. CIND, UCSF Radiology 37 FD using LINs  For each voxel p, for each value of  x,  n :  count the number N of voxels included in B p. CP 1 CP 2 ln  x ln N extend to (  x,  n ) plane phase transition  Slope of each segment = local fractal dimension

38. CIND, UCSF Radiology 38 Possible advantages of LIN over current techniques  LINs provide FD for each voxel  Captures the FD of local regions as well as global  Ideal for directional structures and oriented features at various scales  Far less susceptible to noise –(due to explicit intensity scale  n which can be tuned to the noise level)  Enables the probing of phase transitions

39. CIND, UCSF Radiology 39 Possible Discriminators of Neurodegeneration  Fractal measures may provide better discriminators of neurodegeneration (Alzheimer’s Disease, Frontotemporal Dementia, Mild Cognitive Disorder, Normal Aging, etc)  Possibilities: –Mean (overall) FD -- D(0) –Critical points, phase transitions in ( x,  n ) plane –More general Renyi dimensions D(q) for q ¸ 1 –Summary image feature f()  D(q) –Phase transitions in f()  Fractal structures can be characterized by dimensions D(q), summary f() and various associated critical points  These quantities may be efficiently probed by the Graph Cut –based local influence neighbourhoods  These fractal quantities may provide greater discriminability between normal, AD, FTD, etc.

40. CIND, UCSF Radiology 40 Summary  We proposed a general method of estimating local influence neighbourhoods  Based on an “optimal” energy minimization approach  LINs are intermediaries between purely pixel- based and region-based methods  Applications include segmentation, denoising, filtering, recognition, fractal dimension estimation, …  … in other words, Best Thing Since Sliced Bread

41. Ashish Raj CIND, UCSF email: ashish.raj@ucsf.edu Webpage: http://www.cs.cornell.edu/~rdz/SENSE.htm http://www.vacind.org/faculty Deducing Local Influence Neighbourhoods in Images Using Graph Cuts