1. Advances in the Use of Neurophysiologycally-based Fusion for Visualization and Pattern Recognition of Medical Imagery M. Aguilar, J. R. New and E. Hasanbelliu Knowledge Systems Laboratory MCIS Department Jacksonville State University Jacksonville, AL 36265

2. 2Knowledge Systems Laboratory Outline Introduce Med-LIFE. Introduce Med-LIFE. Revisit 3D image fusion architecture. Revisit 3D image fusion architecture. Compare 2D and 3D fusion results. Compare 2D and 3D fusion results. Fusion for segmentation and pattern recognition. Fusion for segmentation and pattern recognition. Contextual zoom tool. Contextual zoom tool. Segmentation results. Segmentation results.

3. 3Knowledge Systems Laboratory Med-LIFE: Learning, Image Fusion, and Exploration System

4. 4Knowledge Systems Laboratory 3D Shunt Equation Shunting Neural Network Equation: Where: A – decay rate B – maximum activation level (set to 1) D – minimum activation level (set to 1) I C – excitatory input I S – lateral inhibitory input C, G c and G s are as follows: 3D Shunt Operator Symbol Grossberg (1968), Elias & Grossberg (1972)

5. 5Knowledge Systems Laboratory 2-Band 3D Fusion Architecture

6. 6Knowledge Systems Laboratory 4-Band 3D Fusion Architecture

7. 7Knowledge Systems Laboratory 2D vs. 3D Fusion Results MRI-PD MRI-T1MRI-T2SPECT 2D Fusion 3D Fusion

8. 8Knowledge Systems Laboratory Color Fuse Result 4-Band Hybrid Fusion Architecture T1 Images T2 Images Q I Y Color Remap Noise cleaning & registration if needed Contrast Enhancement Between-band Fusion and Decorrelation SPECT Images PD Image _ +........

9. 9Knowledge Systems Laboratory Hybrid Fusion Results 2D Fusion 3D Fusion

10. 10Knowledge Systems Laboratory User-Driven Learning for Segmentation & Pattern Recognition

11. 11Knowledge Systems Laboratory Zoom in place supports: 1. focused attention 2. improved screen real- estate usage Contextual Zoom Visualization Zoom in place: 1. occludes information 2. reduces efficiency by forcing user to maintain context

12. 12Knowledge Systems Laboratory Contextual Zoom Visualization

13. 13Knowledge Systems Laboratory Contextual Zoom Visualization Developed based on COTS software developed by Idelix Supports visualization of fused imagery at multiple details levels Supports detailed analysis and selection for user-driven pattern learning…

14. 14Knowledge Systems Laboratory User-Driven Pattern Learning Supervised learning where training data is selected by user/expert (Waxman et al). Supervised learning where training data is selected by user/expert (Waxman et al). Results assessed and corrected by user. Results assessed and corrected by user. Fuzzy ARTMAP neural network for fast and stable learning. Fuzzy ARTMAP neural network for fast and stable learning. Address order sensitivity by introducing N voters trained with alternate ordering of the training data. Address order sensitivity by introducing N voters trained with alternate ordering of the training data.

15. 15Knowledge Systems Laboratory Pattern Recognition Results

16. 16Knowledge Systems Laboratory Heterogeneous Voting Train 3 Fuzzy ARTMAP systems with parameters as before (different data orderings) Train 3 Fuzzy ARTMAP systems with parameters as before (different data orderings) Train remaining 2 systems with all parameters as in the 3 rd system except for Vigilance (which is a threshold measure that controls the sensitivity of the system). Train remaining 2 systems with all parameters as in the 3 rd system except for Vigilance (which is a threshold measure that controls the sensitivity of the system).

17. 17Knowledge Systems Laboratory Homogeneous vs. Heterogeneous Voters 5 Homogeneous Voters5 Heterogeneous Voters

18. 18Knowledge Systems Laboratory 2D vs. 3D Fusion Segmentation Results 2D Fusion-based Segmentation 3D Fusion-based Segmentation

19. 19Knowledge Systems Laboratory Generalization Training Results Testing Results Slice 11 Slice 10

20. 20Knowledge Systems Laboratory Conclusions Modified fusion approach combines benefits of 2D and 3D fusion. Modified fusion approach combines benefits of 2D and 3D fusion. Preliminary learning segmentation results indicate robustness across slices and cases. Preliminary learning segmentation results indicate robustness across slices and cases. Demonstrated superior performance of 3D fusion for both visualization and pattern recognition. Demonstrated superior performance of 3D fusion for both visualization and pattern recognition. Heterogeneous voting scheme improves learning performance. Heterogeneous voting scheme improves learning performance.

21. 21Knowledge Systems Laboratory BACK-UPS

22. 22Knowledge Systems Laboratory 2D vs. 3D Generalization Testing Results Slice 10 2D Fusion3D Fusion

23. 23Knowledge Systems Laboratory Image Fusion