1. Automated Segmentation of Computed Tomography Images
Justin Senseney
Paul Hemler, PhD
Matthew J. McAuliffe, PhD
Hello -----
My name is Justin Senseney. I work at the National Institutes of Health (NIH) on a software program called MIPAV. MIPAV stands for Medical Image Processing, Analysis, and Visualization software program. It is funded by and geared towards scientists performing biomedical research within centers of the NIH in Bethesda, Maryland. The system I’m presenting is a plug-in for MIPAV that is designed to meet the requirements of a longitudinal study to identify risk factors of chronic osteoarthritis. This study is led by the National Institute on Aging, and is an outgrowth of the Osteoarthritis Initiative that Dr. Jaffe mentioned in yesterday’s opening talk. The system uses both automatic and semiautomatic tools on computed tomography images to help identify these risk factors.

2. Introduction Chronic osteoarthritis risk factors, over time
Obesity (NIH: Body Mass Index > 30)
BMI discussion
Obesity well measured by relative loss of muscle
Fat around muscle tissue independent of BMI
Need for automated systems to quantitatively measure muscle loss
Osteoarthritis is fundamentally a set of diseases that results in the degradation of joints. The National Institute of Arthritis, Musculoskeletal and Skin Diseases estimates that 29 million people in just the United States suffer from chronic osteoarthritis. In older populations, the potential to develop chronic osteoarthritis largely stems from the risk factor of obesity. Obesity has been shown to be an important topic in the discussion of osteoarthritis due to increased load and strain placed on joints throughout the body. However, many beyond those accepted as obese by the medical community develop osteoarthritis. So the need exists to better define these risk factors, and there is currently a push to use more quantitative methods, and as a result a more refined definition of obesity, to both identify and determine risk factors of osteoarthritis.
Now, obesity is a term that is well defined by the National Institutes of Health. It’s defined as a person whose Body Mass Index (BMI) exceeds 30. Body mass index is just person’s height divided by weight squared. Although BMI has served as a useful tool in medicine for over a century, its predictive power for diseases such as chronic osteoarthritis has been called into question, as has its role as the common definition of obesity. Instead, researchers are looking at the relative loss of muscle mass in tissue using computed tomography scans on a longitudinal basis. Current research is suggesting that obesity, and perhaps the risk of chronic osteoarthritis may be well defined by measuring relative loss of muscle mass in subjects over time.
However, the extent to which this muscle loss occurs is especially important in populations which are commonly associated with chronic osteoarthritis. It has already been shown that fat inside of muscles can increase with age without a significant change in BMI. So a patient who is thought to have little risk of developing chronic osteoarthritis because of a low BMI may still in fact be at risk for developing osteoarthritis, assuming their relative loss of muscle mass qualifies for a new definition of obese.
This definition of obese using the relative loss of muscle mass is still very much in flux. In part, quantitative methods that researchers in partnership with radiologists can actually use may help to refine this definition. The system is designed to help researchers at the National Institutes on Aging more precisely evaluate these physiological changes to determine how the above risk factors play a role in chronic osteoarthritis. This system is then a hopefully user-friendly tool that will allow researchers to accomplish this task.
K. F. Adams, A. Schatzkin, T. B. Harris, V. Kipnis, T. Mouw, R. Ballard-Barbash, A. Hollenbeck, and M. F. Leitzmann. Overweight, obesity, and mortality in a large prospective cohort of persons 50 to 71 years old. New England Journal of Medicine, 355(8):763–778, 2006.

3. Methods Automatic segmentation Semiautomatic segmentation
Before I continue, I want you to know that computed tomography images are particularly useful for this kind of quantification project. Pixel intensities are computed as radiographic densities referred to as Hounsfield units that exactly describe the type of tissue at a particular location. Pixels that are fat or muscle at any given location have defined possible Hounsfield units that I will mention later.
So, in evaluating what was needed for this tool to be successfully used, some central themes emerged. Time needed to be saved measuring regions of interest for researchers. As a long term study, we needed flexible tools that could be used both for their intended purpose and for new purposes as they are created. The system also needed to be reliable, with results that could be depended on as valid statistics for drawn regions of interest.
The resulting system consists of two available interfaces that have available both automatic and semiautomatic tools. I have developed automatic segmentation tools for the thigh, bone, marrow, abdomen, and subcutaneous fat, while the semiautomatic tools available in the Medical Image Processing, Analysis, and Visualization software package have also been made available.
I will first give an overview of the automatic methods, then the semiautomatic methods, user customization, and other customization options. Finally I will conclude with a discussion of the results when using the automatic methods of this system.
M. McAuliffe, F. Lalonde, D. McGarry, W. Gandler, K. Csaky, and B. Trus. Medical image processing, analysis and visualization in clinical research. In Computer- Based Medical Systems, CBMS Proceedings. 14th IEEE Symposium on, pages 381–386, 2001.

4. Automatic Segmentation – Thigh
Thigh segmentation
Threshold
Connected thigh segmentation
Identify separation
The topic of single thigh segmentation of computed tomography images was well covered from work in the original osteoarthritis initiative, and is readily available in current research. This system provides a customization of that research where based on the orientation of the image, the image sections can be separated and identified as left and right thigh areas. With the thigh image regions properly identified, a center of mass is calculated for each thigh, attempting to disregard other objects that may be present in the image. This point is perturbed to be guaranteed outside of the bone, and is used for an initial region growing process to capture all tissue within (-300, 900) Hounsfield units of the initial seed point.
Commonly, the thighs for a given image are not separated by air, in which case the thresholding process will identify an initial thigh contour. Approximate derivatives are used to find the most likely point of thigh separation, and if successful, these thighs can then be identified based on the orientation of the image. These are identified as volumes of interest in MIPAV and saved to the file system for later use.
In the case of a 3-dimensional image, slices are corroborated to identify accurate segmentations.
S. Ohshima, S. Yamamoto, T. Yamaji, M. Suzuki, M. Mutoh, M. Iwasaki, S. Sasazuki, K. Kotera, S. Tsugane, Y. Muramatsu, and N. Moriyama. Development of an automated 3d segmentation program for volume quantification of body fat distribution using ct. Japanese Journal of Radiological Technology, 64(9):1177–1181, 2008.

5. Automatic Segmentation – Thigh (2)
Bone segmentation
Region growing
Bone scattering
Marrow segmentation
Fascia, muscle segmentation
Bone scattering is sometimes seen in computed tomography images with some bone densities. A smoothing process ensures any bone scattering present in the original image is also captured, and factored into the overall volume calculations of the thigh. A final thresholding is often enough to yield the relevant bone and marrow components. The resulting left and right images contain the bone and marrow components of both the left and right thigh.
The bone and marrow are then saved and later presented to the user.
Ultimately fascia and muscle segmentations are desired components for this project. While algorithms certainly exist, I haven’t found usable methods that will meet the general needs of this particular project. However I’ve written the code in such a way that all components are processed independently, and relevant algorithms could be included at a later point, possibly for customization by the user. The fascia border is the border you can see in the four images on the right, these are various methods that clearly miss this extremely thin muscle/fat boundary. Algorithms for this process are certainly an area of future research.

6. Automatic Segmentation - Abdomen
Abdomen segmentation
Subcutaneous fat segmentation
Method from Zhao, et al.
The initial operation for identifying the abdomen is to threshold the image, again removing all pixels with Hounsfield units not likely to be part of the abdomen. A seed point of subcutaneous fat just within the skin boundary is identified to begin a region growing operation to provide a conservative estimate for the abdomen. The resulting image as shown to the right will often have air pockets within the abdomen that need to be included within the solution set. A morphological close is performed using dilations and erosions of the likely abdomen set with the complement of the previously identified background image.
Once this process has completed, a method for generating the subcutaneous fat of an abdomen is generated by first from the center of mass of a particular abdomen, radial bands are traversed in one degree increments to determine the location of the subcutaneous area at each point.
A particular band first encounters multiple air pockets (type C) and tissue boundaries (type B) as it searches for the previously segmented abdomen.
Once a band encounters the skin boundary at point A, the subcutaneous fat boundary is determined by traversing over possibly fat area until a muscle has been encountered.
Each band is completed independently before a snaking and smoothing operations are performed.
B. Zhao, J. Colville, J. Kalaigian, S. Curren, J. Li, P. Kijewski, and L. Schwartz. Automated quantification of body fat distribution on volumetric computed tomography. Journal of Computer Assisted Tomography, 30(5), 2006.

7. Semi-Automatic Segmentation
2D options:
Livewire
Level set
2D/3D options:
Region growing
B-spline approximations during slice propagation
By developing this plugin within the MIPAV framework, semiautomatic methods are available for segmenting the previously discussed troublesome muscles, fascia border and other regions of interest. Livewire, level set methods are of course widely used tools for semi-automatic segmentation. MIPAV also has the ability to run all algorithms available through the Insight ToolKit. These tools are all directly available to the researcher, made possible by the MIPAB user interface and some customizations.
These tools have also been modified for specific use within the plugin. For example, during the semiautomatic segmentation of muscle regions of interest, a drawn region can be propagated to connecting slices through customized buttons that also deform the VOI by an approximate derivative, in this case a b-spline approximation.

8. Tissue Classification
Partial voluming concerns:
-190 <= fat pixel <= -30
0 <= muscle pixel <= 100
-30 < partial volume pixel < 0
Custom look-up table: fat-red, muscle-blue, partial volume-white.
As I previously mentioned, threshold classification of CT images allows identification of muscle and fat tissue within the thigh and abdomen. However, one needs to be aware of partial voluming effects. By adopting a schema found in recent literature pixels with indeterminate Hounsfield units do not provide unusual additions to the fat or abdomen calculations. This schema identifies fat pixels as being between and -30 Hounsfield units. Those with Hounsfield units between 0 and 100 are muscle pixels, and finally those pixels in the exclusive range of -30 to 0 are identified as partial volume pixels. This classification system can be assigned to a lookup table (LUT). Applying the given LUT to a CT abdomen image yields the result you see on the right.
The partial voluming effects are denoted by white pixels in this figure. Such pixels are counted as part of a particular muscle’s total tissue area/volume when they fall under a given VOI. However, such pixels are not counted as part of a given VOIs total muscle or intramuscular fat area/volume.
S. Ohshima, S. Yamamoto, T. Yamaji, M. Suzuki, M. Mutoh, M. Iwasaki, S. Sasazuki, K. Kotera, S. Tsugane, Y. Muramatsu, and N. Moriyama. Development of an automated 3d segmentation program for volume quantification of body fat distribution using ct. Japanese Journal of Radiological Technology, 64(9):1177–1181, 2008.

9. Muscle and Fat Quantification
Reports
Text
PDF, using iText
Standard output
MIPAV Statistics generator
This segmentation tool calculates the total fat, lean, and aggregate area in any particular VOI and also computes the mean Hounsfield unit for each of these sets. In cases where multiple curves exist (such as objects like liver cysts) the total area is the compilation of all created curves.
To express these calculations in a standardized way, a few different methods exist. The PDF report on the far right is automatically created using the open source and freely available program iText. The MIPAV statistics generator has also been customized for this tool to produce all the relevant calculations I just mentioned for each VOI (Volume of Interest) in addition to the statistics already available. For a given VOI, these might include the center of mass, eccentricity, and major or minor axis length. The output generated from these tools can be imported into any other analysis program for further evaluation.
B. Lowagie. iText in Action: Creating and Manipulating PDF. Manning, New York, 2006.

10. Customization Interface for customized CT projects, options:
New regions of interest
Calculation dependencies
Display options
Calculation options
Start Pane: Abdomen
Start Voi: Abdomen
Color: 255,200,0
Do_Calc: true
End Voi
Start Voi: Subcut.
Color: 255,0,0
Start Voi: Phantom
Color: 0,255,0
End Pane
Start Pane: Tissue
Start Voi: Visceral
End Voi
Start Voi: Liver
Color: 255,0,0
Start Voi: Liver cysts
Num_Curves: 7
Color: 0,255,0
Do_Calc: true
Do_Fill: true
Start Voi: Bone sample
Color: 0,255,255
Start Voi: Water sample
Color: 255,0,255
End Pane
Derivations of the presented classification and quantification tasks are possible through an interface I designed that allows this system to be used for other research projects that involve computed tomography images. The abdomen and thigh tools I mentioned generate local preference files in the format you can see on the left. For a given VOI, attributes such as color, possible symmetry, number of curves, and whether to perform calculations are all options that can be performed. Advanced users can also modify this interface file to define calculation dependencies such as the current convention of not including bone and marrow area within the total thigh area. This allows researchers to specify muscles or other tissues of interest to a particular project.
Most interface modifications can be made either manually or within the figure shown, generating a new set of saved preferences that can be selected later. These modifications are presented to the researcher as custom created objects that are present in all parts of the research framework. Items like intestinal obstructions, unusual calcifications, or specific muscle bundles are examples of objects that can be added through this interface.
These objects can be included in relevant calculations and displayed appropriately. Items of interest that are added in this way are shown in the automatically generated PDF report such as the one shown.

11. Customization Options
Orientation invariant
Volume/Area Options
Units Specification
I just want to mention some other features this plugin has available.
So, what I’ve been talking about uses the terms right, left, top, and bottom from the perspective of the observer to describe the work without regard to a particular orientation, when running the plugin by default the images are rotated to an axial orientation if that’s not already the case.
Next, when a 3-dimensional CT image with a small slice thickness is being used, a researcher will often be interested in obtaining the volume of muscle on a per-slice basis. The results section of this plugin allows for the user to specify this output, as you can see in the dialog on this slide.
Finally, the default output is by convention in centimeters. This can be changed to any unit of length that MIPAV has support for and is reflected in all the available outputs that I showed in a previous slide.
Do you see the red text on the abdomen and subcutaneous fat? That indicates that the particular VOI was automatically segmented, and the user has gone to the VOI review panel to confirm it’s an accurate segmentation. Even though as I’ll present, the automatic methods are fairly reliable, this is a nice way to keep the user informed and engaged.

12. Thigh Results
% Difference
Standard Deviation
Left Thigh
0.18
0.38
Right Thigh
0.24
0.55
Compared to 13 freehand segmented images from the University of California, San Diego (UCSD)
Useful for manually demanding segmentations
To generate some preliminary results from this tool, I compared a set of automatically segmented dataset were compared to manually segmented regions of this dataset from the University of California, San Diego (UCSD). The UCSD data were composed of 13 thigh and 13 abdomen CT scans. The freehand segmentations had been generated and processed using a program developed at UCSD. This program computed the areas of regions of interest, subject to the same partial voluming errors encountered within the MIPAV based tool. The results were printed in a PDF report.
I compared the automatically segmented area of the thigh directly compared to the UCSD’s manually segmented data values after subtracting area values for the bone and marrow.
The small average percent difference between the UCSD and MIPAV produced areas suggests to me that this tool’s automatic algorithms could be a useful innovation when encountering a project where the manual demands of segmenting an image would be too laborious and subjective. Of course comparison’s to other automatic methods and other radiologist’s evaluations would need to be made before I could make that statement conclusively.

13. Abdomen Results Larger variability Needs manual attention % Difference
Standard Deviation
Abdomen
1.12
2.92
Abdomen Fat
2.34
1.92
Subcutaneous Fat
3.06
3.58
Subcutaneous Fat HU
1.57
1.50
Larger variability
Needs manual attention
The automatic segmentation of abdomen areas were a little less encouraging. On average, this tool is overextending the definition of abdomen tissue to phantoms that may exist in the image.
Automatic segmentation of the subcutaneous area encountered similar problems to those from attempting to segment the fascia border in the thigh case, and the results reflect this. This suggests to me that semiautomatic segmentation tools are useful in optimizing the results of the subcutaneous segmentation.
The average percent difference of the subcutaneous fat Hounsfield unit is only about half as large as the average percent difference for the subcutaneous fat. These percentages present either an area of further refinement for our tool or the need for manual inspection of the automatically generated VOIs to ensure the integrity of results.

14. Conclusion Useful automatic and semi-automatic methods
Ability to later refine these
Benefits from manual overview
Larger analysis set needed
Comparison to automatic methods
Comparison to other qualified people for manual segmentation
This software package presents a useful framework for testing new image processing algorithms and functionality. The ones presented here show promising results, but more analysis is definitely necessary. I’m very interested in comparing the segmentation efforts of this program to accessible clinical measurements to determine the general reliability and accuracy of the program I’ve developed. It’s clear that no matter what the system, manual expert review is still necessary.
Testing really is critical to ensure the reliability of the automatically generated results. The testing presented here indicates that our system does provide invariant, accurate results across a spectrum of computed tomography images. The segmentation process using this system is in fact faster than existing manual segmentation systems. This system provides segmentation and calculation abilities that with further work will make large scale studies such as the world-wide osteoarthritis cohort study feasible.

15. Download http://mipav.cit.nih.gov SenseneyJ@mail.nih.gov
Look in the plugins folder for the MuscleSegmentation plugin
Open Source? No….
This plugin can be downloaded along with the rest of MIPAV from the website. The release version reflects a tested, well functioning system for image processing tasks.
The nightly version is guaranteed to have compiled. New version coming out on Friday.
Not open source, yet, currently I’m working on an interface for a JHU project to combine MIPAV and Slicer algorithms into a GUI for image processing, this is the first step on what I wish would be a short road to becoming open source, but we’ll see…

16. Acknowledgments
This work was supported by the Intramural Research Program of the National Institutes of Health and the Center for Information Technology at the National Institutes of Health.

17. Questions?

18. Watershed? Requires pre-processing steps Limited viability
MIPAV’s implementation the watershed segmentation requires the user or algorithm to identify background pixels and likely object pixels. For disjointed regions this works quite well, but as you can see in this particular connected thigh case, the algorithm could not produce separate objects.
In general I found keeping these algorithms fairly simple resulted in an overall gain of successful segmentations.

19. Future work? Algorithms Data Usability Extensibility
There are a number of different ways this work could go from here. New automatic algorithms, more data from resources similar to caBig, a comprehensive usability study to further design needs, attempts to make plugin even more extended.