1. Motion Analysis (contd.) Slides are from RPI Registration Class.

2. 2 Transformations Geometric transformations Geometric transformations Intensity transformations Intensity transformations

3. 3 Forward Geometric Transformation “Moving” image, I m “Fixed” image, I f Transformed moving image, I m ’. It is mapped into the coordinate system of I f

4. 4 Mapping Pixel Values - Going Backwards “Moving” image, I m “Fixed” image, I f Need to go backward to get pixel value. This will generally not “land” on a single pixel location. Instead you need to do interpolation.

5. 5 Backward Transformation Because of the above, some techniques estimate the parameters of the “backwards” transformation, T -1 : Because of the above, some techniques estimate the parameters of the “backwards” transformation, T -1 : –From the fixed image to the moving image This can sometimes become a problem if the mapping function is non-invertible This can sometimes become a problem if the mapping function is non-invertible Intensity-based algorithms generally estimate the backwards transformation Intensity-based algorithms generally estimate the backwards transformation

6. 6 Intensity Transformations When the intensities must be transformed as well, the equations get more complicated: When the intensities must be transformed as well, the equations get more complicated: Intensity mapping function Intensity mapping parameters

7. 7 MRI image registration, similarity transformation (rotated by 10 degrees, with a translation of 17mm and 13mm)

8. 8 Intensity-Based Registration Use the intensities more or less directly Use the intensities more or less directly Compare intensities between Compare intensities between –Mapped (transformed) version of the moving image I m (based on an estimated transformation) and –Fixed image I f Need: Need: –Pixel-by-pixel error measure –Mapping technique –Minimization technique

9. 9 Example Error Measure: SSD Region of intersection between images Pixel location within region

10. 10 SSD Example: Initial Alignment

11. 11 SSD: Sum of Squared Errors Advantages: Advantages: –Simple to compute –Differentiable –Optimal for Gaussian error distributions Disadvantages: Disadvantages: –Doesn’t allow varying “gain” between the images, which may be caused by different illuminations or different camera settings –Biased by large errors in intensity E.g. caused by contrast agent injection E.g. caused by contrast agent injection

12. 12 Working in the Parameters Remember: Remember: This means that to evaluate the effect of a transformation estimate what we really want to evaluate is This means that to evaluate the effect of a transformation estimate what we really want to evaluate is

13. 13 The Role of the Region Observe: the region over which the transformation is evaluated depends on the parameters: Observe: the region over which the transformation is evaluated depends on the parameters: This can cause problems in practice: This can cause problems in practice: –A transformation resulting in no overlap leads to 0 error!

14. 14 Evaluating the Objective Function Pixel-by-pixel evaluation within the region Pixel-by-pixel evaluation within the region Apply the inverse mapping at each pixel Apply the inverse mapping at each pixel Problem: inverse mapping of pixel does not “land” on a discrete pixel location! Problem: inverse mapping of pixel does not “land” on a discrete pixel location!

15. 15 Many Interpolation Options Nearest neighbor Nearest neighbor Bilinear (or trilinear in 3d) Bilinear (or trilinear in 3d) Spline Spline

16. 16 Bilinear Interpolation in Moving Image Weighted average of 4 surrounding pixels Weighted average of 4 surrounding pixels –8 surrounding pixels in 3d Weight proportional to distance in x and in y Weight proportional to distance in x and in y

17. 17 Bilinear: Resulting Intensity

18. 18 Two Options In Practice Create intensity, pixel-by-pixel, but don’t create an explicit image I m ’ Create intensity, pixel-by-pixel, but don’t create an explicit image I m ’ Create actual image I m ’ Create actual image I m ’

19. 19 Resetting the Stage We have: We have: –Formulated the SSD objective function –Discussed how to evaluate it Next step is how to minimize it with respect to the transformation parameters Next step is how to minimize it with respect to the transformation parameters

20. 20 Before Proceeding We will estimate the parameters of the backward transformation We will estimate the parameters of the backward transformation Abusing notation, we will minimize the equation Abusing notation, we will minimize the equation It should be understood (implicitly) that this is the inverse transformation and the parameter values will be different It should be understood (implicitly) that this is the inverse transformation and the parameter values will be different

21. 21 Function Minimization Function to minimize: Function to minimize: Possibilities Possibilities –Amoeba (simplex) methods - non- differential –Gradient / steepest descent –Linearization (leading to least- squares) –Newton’s method –Many more …

22. 22 Gradient / Steepest Descent Compute gradient of objective function (with respect to transformation parameters), evaluated at current parameter estimate Compute gradient of objective function (with respect to transformation parameters), evaluated at current parameter estimate Make tentative small change in parameters in the negative gradient direction Make tentative small change in parameters in the negative gradient direction –  is called the “learning rate” Re-evaluate objective function and accept change if it is reduced (otherwise reduce the learning rate) Re-evaluate objective function and accept change if it is reduced (otherwise reduce the learning rate) Continue until no further changes are possible Continue until no further changes are possible

23. 23 Computing the Derivative Issue: Issue: –Images are discrete –Parameters are continuous Two methods Two methods –Numerical –Continuous (eventually numerical as well) Abstract definition of parameter vector: Abstract definition of parameter vector:

24. 24 Numerical Derivatives Form each partial derivative by taking a small step in each parameter, i = 1,..,k: Form each partial derivative by taking a small step in each parameter, i = 1,..,k: Choice of step size can be difficult Choice of step size can be difficult Requires k+1 function evaluations to compute the derivative Requires k+1 function evaluations to compute the derivative Sometimes this is the only thing you can do! Sometimes this is the only thing you can do!

25. 25 Continuous Computation of Derivative Apply chain rule: Intensity gradient in moving image Change in transformation wrt change in parameters Current error at pixel location

26. 26 Computing Image Derivatives Many ways. Many ways. –Simplest is pixel differences. –More sophisticated methods account for image noise Computed at each pixel Computed at each pixel

27. 27 Derivative In Moving Image Equation Equation In detail In detail –Pre-compute derivatives in moving image I m –During minimization, map pixels back into moving image coordinate system and interpolate

28. 28 Image Derivative Example

29. 29 dT/d  Similarity transform: Similarity transform: –Where So derivative is 2x4 matrix (Jacobian): So derivative is 2x4 matrix (Jacobian):

30. 30 Putting It All Together At each pixel in overlap region: At each pixel in overlap region: –Calculate intensity difference (scalar) –Multiply by 1x2 intensity gradient vector computed by mapping pixel location back to moving image –Multiply by 2x4 Jacobian of the transformation, evaluated at pixel location –Result is 1x4 gradient vector at each pixel Sum each component of vector over all pixels Sum each component of vector over all pixels

31. 31 Algorithm Outline Initialize transformation Initialize transformation Repeat Repeat –Compute gradient –Make step in gradient direction –Update mapping equation –Remap image Until convergence Until convergence

32. 32 Initialization Since this is a minimization technique, an initial estimate is required, Since this is a minimization technique, an initial estimate is required, There are many ways to generate this estimate: There are many ways to generate this estimate: –Identity transformation, e.g. –Prior information –Different technique Steepest descent only finds a local minimum of the objective function Steepest descent only finds a local minimum of the objective function

33. 33 Convergence Ideal is that gradient is 0. Ideal is that gradient is 0. In practice, algorithm is stopped when: In practice, algorithm is stopped when: –Step size becomes too small –Objective function change is sufficiently small –Maximum number of iterations is reached

34. 34 Example Initial errors Iteration 100Iteration 200 Iteration 300Final: 498 iterations

35. 35 Discussion Steepest descent is simple, but has limitations: Steepest descent is simple, but has limitations: –Local minima –Slow (linear) convergence

36. 36 Summary Intensity-based registration is driven by direct functions of image intensity Intensity-based registration is driven by direct functions of image intensity SSD is a common, though simple example SSD is a common, though simple example Evaluating the SSD objective function (and most other intensity-based functions) requires image interpolation Evaluating the SSD objective function (and most other intensity-based functions) requires image interpolation Gradient descent is a simple, commonly-used minimization technique Gradient descent is a simple, commonly-used minimization technique Derivatives may be calculated using either numerical approximations or differentiation of the objective function. Derivatives may be calculated using either numerical approximations or differentiation of the objective function.