1. Active Appearance Models master thesis presentation Mikkel B. Stegmann IMM – June 20th 2000

2. Presentation outline  Aim  Method  Metacarpals – a case study  Discussion  Conclusion

3. Aim  To locate non-rigid objects in digital images  The vision utopia Fully automated General Specific Robust Accurate Holistic Non-parametric Fast

4. Active Appearance Models  A model-based approach towards segmentation  A priori knowledge is not programmed into the model, but learned through observation  Relies on statistical analysis of shape and texture variation in a training set  Derives a compact object class description which can be used to rapidly search images for new object instances

5. Model building 1) Data capture Shape: point annotation Texture: pixel sampling 3) Statistical analysis Principal component analysis on shape and texture 3) Combining shape and appearance Shape and texture PCA is combined into a 3rd PCA 4) Model truncation Parameters are truncated to satisfy a variance constraint 2) Normalisation Shape: pose alignment using the Procrustes shape metric Texture: photometric normalisation

6. Shape analysis  Shape is represented by a linear spline of landmarks: X = ( x 1, …, x n, y 1, …, y n ) T Assumes point correlation Requires point correspondence Alignment w.r.t. position, scale, orientation Principal component analysis Compact shape representation 102030405060708090100 10 20 30 40 50 60 70 80 90 100

7. Texture analysis  Texture – the intensities across the object – is sampled inside the shape using a suitable warp function  Warp function: A piece-wise affine warp using the Delaunay triangulation g = ( x 1, …, x n ) T Principal component analysis Compact texture representation

8. Combined Model  Shape and texture is combined into a compact model representation  This representation is capable of derforming in a similar manner to what is observed in the training set  Thus making the model specific to the class of objects it represents  Generative (self-contained)

9. Model Optimisation  Deforms the AAM to fit the image being searched  Assumes a linear relationship between model parameters and the observed fit: C = RX  Solved using multivariate linear regression on a large set of experiments Actual dy (pixels) Predicted dy (pixels)

10. Implementation  Open source C++ API based on the Windows platform [and partly on VisionSDK, LAPACK, Intel MKL, ImageMagick a.o.]  Well documented [cross-referenced HTML and PDF]  Fast [using Intel BLAS for matrix handling and widely use of dynamic programming]  Suitable for education & research [lots of visual and numerical documentation: *.m *.avi *.bmp]  Example usage included [in the form of a console interface]

11. Metacarpals – a case study  20 x-ray images of the human hand supplied by Pronosco  Metacarpal 2, 3, 4 annotated using 50 points on each  Difficult segmentation problem due to large shape variability and the ambiguous nature of radiographs

12. Building the model Annotation of set of training images Capture of shape & texture Statistical analysis on shape & texture

13. Modes of variation 0510152025 0 5 10 15 20 25 0510152025 0 5 10 15 20 25 30 35 40 45 0510152025 0 2 4 6 8 10 12 14 16 18 ShapeTextureCombined

14. Metacarpal AAM  Image modality: radiographs (x-rays)  20 images/shapes in training set  300 points in shape model  ~10.000 pixels in texture model  95% variation explained using 16 model parameters

15. Search

16. Metacarpal results  Using automatic initialisation Good mean location accuracy 0.98 pixel (point to border) Acceptable mean texture fit 6.57 gray levels (byte range)  Difficult to locate the exact bone extents at the proximal and distal end mean pt. errors proximal distal

17. Discussion  “Hidden” benefits Automatic registration Variance analysis (group/longitudinal studies) Discrimination/interpretation using the model parameters  Weaknesses Requires landmarks (point correspondence) Can only deform texture by moving edge points Not robust to large-scale texture noise

18. Discussion - cont’d  Image modalities on which AAMs has been evaluated successfully: Radiographs - x-rays of human hands Normal gray scale images - hands, pork carcasses MRI - human hearts  Initialisation has been added, thus making AAM a fully automated segmentation method  The AAM approach extends to 3D and multivariate imaging

19. Conclusion  AAM has been implemented and extended as a fully automated and data-driven approach towards image segmentation  AAM performs well on very different segmentation problems and different image modalities  Properties General Specific Captures domain knowledge without the need for technical knowledge Robust Non-parametric Self-contained Fast

20. fin http://www.imm.dtu.dk/~aam