1. Applications in Medical Imaging
By:
Prof G R Sinha

2. My dissertation work Research areas: Image Enhancement & Applications
Dissertation topic: Design & Implementation of Image Enhancement Techniques in Frequency Domain
Research hypothesis:
“A picture is worth thousands of words…”
“There is enough information in the image content to perform image retrieval whose similarity results correspond to the human perceived similarity”.

3. Medical informatics research
What is medical informatics?
Medical informatics is the application of computers, communications and information technology and systems to all fields of medicine
- medical care
- medical education
- medical research.

4. What is medical informatics?
Medical informatics is the branch of science concerned with the use of computers and communication technology to acquire, store, analyze, communicate, and display medical information and knowledge to facilitate understanding and improve the accuracy, timeliness, and reliability of decision-making.
Warner, Sorenson and Bouhaddou, Knowledge Engineering in Health Informatics, 1997

5. Clinical decision making
Making sound clinical decisions requires:
– right information, right time, right format
Clinicians face a surplus of information
– ambiguous, incomplete, or poorly organized

6. Clinical decision making: What is the problem?
Man is an imperfect data processor
– We are sensitive to the quantity and organization of information
Army officers and pilots commit ‘fatal errors’ when given too many, too few, or poorly organized data
The same is true for clinicians who ‘watch’ for events
Clinicians are particularly susceptible to errors of omission

7. Subdomains of medical informatics (by Wikipedia)
imaging informatics
clinical informatics
nursing informatics
consumer health informatics
public health informatics
dental informatics
clinical research informatics
bioinformatics
pharmacy informatics

8. What is medical imaging (MI)?
The study of medical imaging is concerned with the
interaction of all forms of radiation with tissue
and
the development of appropriate technology to extract clinically useful information (usually displayed in an image format) from observation of this technology.

9. Examples of medical images

10. The imaging “chain” Raw data Reconstruction Filtering “Raw data”
                         
Signal
acquisition
Processing
Analysis
123……………
2346…………..
65789…………
6578…………..
Quantitative output

11. Image analysis: Turning an image into data
User extracted qualitative features
User extracted quantitative features
Exam Level: Feature 1
Feature 2
Feature 3 .
Finding: Feature 1

12. Major advances in medical imaging
Image Segmentation
Image Classification
Computer-Aided Diagnosis Systems
Computer-Aided Diagnostic Characterization
Content-based Image Retrieval
Image Annotation
These major advances can play a major role in early detection, diagnosis, and computerized treatment planning in cancer radiation therapy.

13. Computer-Aided Diagnosis
Computed Aided Diagnosis (CAD) is diagnosis made by a radiologist when the output of computerized image analysis methods has been incorporated into his or her medical decision-making process.
CAD may be interpreted broadly to incorporate both
the detection of the abnormality task and
the classification task: likelihood that the abnormality represents a malignancy
Classification, comparison, or analysis of images is performed almost always in terms of a set of features extracted from the images. Usual this is necessary for one of the following reasons:
Reduction of dimensionality: an 8-bit per pixel image of size 256x256 pixels has 25665,536 =10157,826 possible realisations. Clearly, it is worth –while to express structure within and similarities between images in ways that depends on fewer, higher-level representations of their pixels and relationship. It will important to show that the reduction nevertheless preserves information important to the task.
Incorporation of cues from human perception. Much is known about the effects of basic stimuli on the visual system. In many situations, we have considerable insight into how humans analyse images (essential in the training of radiologist and photo interpreters). Use of the right kinds of features would allow for the incorporation of that experience into automated analysis.
Transcend the limit of human perception. Though we can very easily understand many kinds of images, there are properties (e.g. some textures) of images that we cannot perceive visually, but which could be useful in characterising them. Features can be constructed from various manipulations of the images that make those properties evident.
Need for invariance. The meaning and the utility of an image are often unchanged when the image is perturbed in various way. Changes in one or more of scale, location, brightness and orientation for example and the presence of noise, artefacts and intrinsic variation are image alteration to which well-designed featured are wholly or partially invariant.

14. Motivation for CAD systems
The amount of image data acquired during a CT scan is becoming overwhelming for human vision and the overload of image data for interpretation may result in oversight errors.
Computed Aided Diagnosis for:
Breast Cancer
Lung Cancer
A thoracic CT scan generates about 240 section images for radiologists to interpret.
Colon Cancer
CT colonography (virtual colonoscopy) is being examined as a potential screening device ( images)
Classification, comparison, or analysis of images is performed almost always in terms of a set of features extracted from the images. Usual this is necessary for one of the following reasons:
Reduction of dimensionality: an 8-bit per pixel image of size 256x256 pixels has 25665,536 =10157,826 possible realisations. Clearly, it is worth –while to express structure within and similarities between images in ways that depends on fewer, higher-level representations of their pixels and relationship. It will important to show that the reduction nevertheless preserves information important to the task.
Incorporation of cues from human perception. Much is known about the effects of basic stimuli on the visual system. In many situations, we have considerable insight into how humans analyse images (essential in the training of radiologist and photo interpreters). Use of the right kinds of features would allow for the incorporation of that experience into automated analysis.
Transcend the limit of human perception. Though we can very easily understand many kinds of images, there are properties (e.g. some textures) of images that we cannot perceive visually, but which could be useful in characterising them. Features can be constructed from various manipulations of the images that make those properties evident.
Need for invariance. The meaning and the utility of an image are often unchanged when the image is perturbed in various way. Changes in one or more of scale, location, brightness and orientation for example and the presence of noise, artefacts and intrinsic variation are image alteration to which well-designed featured are wholly or partially invariant.

15. CAD for Breast Cancer
A mammogram is an X-ray of breast tissue used as a screening tool searching for cancer when there are no symptoms of anything being wrong. A mammogram detects lumps, changes in breast tissue or calcifications when they're too small to be found in a physical exam.
Abnormal tissue shows up a dense white on mammograms.
The left scan shows a normal breast while the right one shows malignant calcifications.

16. CAD for Lung Cancer
Identification of lung nodules in thoracic CT scan; the identification is complicated by the blood vessels
Once a nodule has been detected, it may be quantitatively analyzed as follows:
The classification of the nodule as benign or malignant
The evaluation of the temporal size in the nodule size.

17. Role of Image Analysis & Machine Learning for CAD
An overall scheme for computed aided diagnosis systems

18. SoC Medical imaging research projects
1. Computer-aided characterization for lung nodules
Goal: establish the link between computer-based image features of lung nodules in CT scans and visual descriptors defined by human experts (semantic concepts) for automatic interpretation of lung nodules
Example: This lung nodule has a “solid” texture and has a “sharp” margin

19. Why computer-aided characterization?
Reader 1
Reader 2
Reader 3
Reader 4
Lobulation=4
Malignancy=5 “highly suspicious”
Sphericity=2
Lobulation=1 “marked”
Malignancy=5 “highly suspicious”
Sphericity=4
Lobulation=2
Malignancy=5 “highly suspicious”
Sphericity=5 “round”
Lobulation=5 “none”
Malignancy=5 “highly suspicious”
Sphericity=3 “ovoid”
Show how outlines can also be different. Explain that for the same nodule, slices with biggest nodule can be different for different radiologists. Start talking about calculating image features of a nodule, go to the next slide.
Ratings and Boundaries across radiologists are different!!! 19

20. Computer-aided characterization
Research Hypothesis
“The working hypothesis is that certain radiologists’ assessments can be mapped to the most important low-level image features”.
Methodology
new semi-supervised probabilistic learning approaches that will deal with both the inter-observer variability and the small set of labeled data (annotated lung nodules).
Our proposed learning approach will be based on an ensemble of classifiers (instead of a single classifier as with most CAD systems) built to emulate the LIDC ensemble (panel) of radiologists.

21. Computer-aided characterization (cont.)
Expected outcome:
an optimal set of quantitative diagnostic features linked to the visual descriptors (semantic concepts).
Significance:
The derived mappings can serve to show
the computer interpretation of the corresponding radiologist rating in terms of a set of standard and objective image features,
automatically annotate new images,
and augment the lung nodule retrieval results with their probabilistic diagnostic interpretations. 21

22. Computer-aided characterization
Preliminary results
NIH Lung Image Database Consortium (LIDC):
149 distinct nodules from about 85 cases/patients;
four radiologists marked the nodules using 9 semantic characteristics on a scale from 1 to 5 except for calcification (1 to 6) and internal structure (1 to 4)

23. LIDC high level concepts & ratings
Computer-aided characterization
LIDC high level concepts & ratings
Characteristic
Possible Scores
Margin
1. Poorly Defined
5. Sharp
Sphericity
1. Linear
2. .
3. Ovoid
4. .
5. Round
Spiculation
1. Marked
5. None
Subtlety
1. Extremely Subtle
2. Moderately Subtle
3. Fairly Subtle
4. Moderately Obvious
5. Obvious
Texture
1. Non-Solid
3. Part Solid/(Mixed)
5. Solid
Characteristic
Possible Scores
Calcification
1. Popcorn
2. Laminated
3. Solid
4. Non-central
5. Central
6. Absent
Internal structure
1. Soft Tissue
2. Fluid
3. Fat
4. Air
Lobulation
1. Marked
5. None
Malignancy
1. Highly Unlikely
2. Moderately Unlikely
3. Indeterminate
4. Moderately Suspicious
5. Highly Suspicious
Talk more about interpretation (and interpretation variability) of separate semantic characteristics and move to the next two slides to show a specific example. 23

24. Computer-aided characterization
Low-level image features
Shape Features
Size Features
Intensity Features
Texture Features
Circularity
Area
MinIntensity
11 Haralick features calculated from co-occurrence matrices
Roughness
ConvexArea
Maxintensity
24 Gabor features
Elongation
Perimeter
SDIntensity
5 Markov Random Field features
Compactness
ConvexPerimeter
MinIntensityBG
Eccentricity
EquivDiameter
MaxIntensityBG
Solidity
MajorAxisLength
MeanIntensityBG
Extent
MinorAxisLength
SDIntensityBG
RadialDistanceSD
IntensityDifference
Describe 4 types of features used in a study. Explain how features are mapped to the semantic characteristics. Describe vector representation of a nodule after mapping is done {c1…c7, f1…f64} as input for automatic interpretation algorithm. 24

25. Computer-aided characterization
Accuracy results
Characteristics
Decision trees
Add instances predicted with high confidence (60%)
Add instances predicted with high confidence (60%) and instances with low margin (5%)
Lobulation
27.44%
81.00%
69.66%
Malignancy
42.22%
96.31%
Margin
35.36%
98.68%
96.83%
Sphericity
36.15%
91.03%
90.24%
Spiculation
63.06%
58.84%
Subtlety
38.79%
93.14%
92.88%
Texture
53.56%
97.10%
97.36%
Average
38.52%
88.62%
86.02%
Present the results. Show that both approaches improved the accuracy for all semantic characteristics in comparison with the decision trees. Mention that difference in accuracies between two approaches are not significant except for lobulation. Depending on how much time will be left talk about further work (what we are doing right now) either show and explain the next slide or list what we have tried to do. 25

26. Computer-aided characterization
Challenges
Small number of training samples and large number of features “curse of dimensionality” problem
Nodule size
Variation in the nodules’ boundaries
Different types of imaging acquisition parameters
Clinical evaluation: observer performance studies
require collaboration with medical schools or hospitals

27. SoC Medical imaging research projects-
2. Texture-based Pixel Classification
- tissue segmentation
- context-sensitive tools for radiology reporting
Pixel Level Texture Extraction
Pixel Level Classification
Organ Segmentation

28. Texture-based Pixel Classification
Texture Feature extraction: consider texture around the pixel of interest.
Capture texture characteristic based on
estimation of joint conditional probability
of pixel pair occurrences Pij(d,θ).
Pij denotes the normalized co-occurrence matrix of specify by displacement vector (d) and angle (θ).
Neighborhood
of a pixel

29. Haralick Texture Features

30. Haralick Texture Features

31. Examples of Texture Images
Texture images: original image, energy and cluster tendency, respectively.
M. Kalinin, D. S. Raicu, J. D. Furst, D. S. Channin,, " A Classification Approach for Anatomical Regions Segmentation", The IEEE
International Conference on Image Processing (ICIP), Genoa, Italy, September 11-14, 2005.

32. Texture Classification of Tissues in CT Chest/Abdomen
Example of Liver Segmentation: (J.D. Furst, R. Susomboon, and D.S. Raicu, "Single Organ Segmentation Filters for Multiple Organ Segmentation", IEEE 2006 International Conference of the Engineering in Medicine and Biology Society (EMBS'06))
Original Image
Initial Seed at 90%
Split & Merge at 85%
Split & Merge at 80%
Region growing at 70%
Region growing at 60%
Segmentation Result

33. Classification models: challenges
(a) Optimal selection of an adequate set of textural features is a challenge, especially with the limited data we often have to deal with in clinical problems. Consequently, the effectiveness of any classification system will always be conditional on two things:
(i) how well the selected features describe the tissues
(ii) how well the study group reflects the overall target patient population for the corresponding diagnosis

34. Classification models: challenges
(b) how other type of information can be incorporated into the classification models:
- metadata
- image features from other imaging modalities (need of image fusion)
(c) how stable and general the classification models are

35. Content-based medical image retrieval (CBMS) systems
Definition of Content-based Image Retrieval:
Content-based image retrieval is a technique for retrieving images on the basis of automatically derived image features such as texture and shape.
Applications of Content-based Image Retrieval:
Teaching
Research
Diagnosis
PACS and Electronic Patient Records

36. Diagram of a CBIR Query Results Image Features [D1, D2,…Dn]
Feature Extraction
Similarity Retrieval
Image Features
[D1, D2,…Dn]
Image Database
Query Image
Query Results
Feedback
Algorithm
User Evaluation
Diagram of a CBIR

37. CBIR systems: challenges
Type of features
image features:
- texture features: statistical, structural, model and filter-based
- shape features
textual features (such as physician annotations)
Similarity measures
-point-based and distribution based metrics
Retrieval performance:
precision and recall
clinical evaluation

38. uestions ?