1. 1 ASU MAT 591: Opportunities in Industry! ASU MAT 591 Image Processing Science and Robotic Vision Rod Pickens Principal Research Engineer Lockheed Martin, Incorporated

2. 2 ASU MAT 591: Opportunities in Industry! Signals and Processing l Signals –Analog and discrete signals –Dimensionality of signals  1-D signals l Sounds (temporal), echocardiogram, seismic signal  2-D signals (this presentation) l Images (spatial)  3-D signals l Video sequences of images (spatial and temporal) l Signal processing –Synthesize and analyze signals –Filter signals using low-pass, band-pass, and high-pass filter –Modify signals such as warp, delay, stretch, rotate, shrink, … –Restore and enhance signals –Recognize patterns and detect signals

3. 3 ASU MAT 591: Opportunities in Industry! Signal Processing: Now Animal Touch Vision Hearing Smell Taste Robotic Touch Vision Hearing Smell Taste

4. 4 ASU MAT 591: Opportunities in Industry! The Processing Analogy

5. 5 ASU MAT 591: Opportunities in Industry! Analysis and Synthesis of Light White Light Out Fourier Analysis Fourier Synthesis White Light In Inverse Functions

6. 6 ASU MAT 591: Opportunities in Industry! Fourier Transforms are Inverse Functions

7. 7 ASU MAT 591: Opportunities in Industry! Inverse Functions Derivative Inv Fourier Trans Inv Radon Trans Warp Correction Integral Fourier Transform Radon Transform Warp Data

8. 8 ASU MAT 591: Opportunities in Industry! Filtering White Light In Filtering removes all but red colors Red Light Out

9. 9 ASU MAT 591: Opportunities in Industry! Television Channel 6 Filtering removes all but Channel 6 Television Stations 3, 5, 6, 13, 15, … Television

10. 10 ASU MAT 591: Opportunities in Industry! Television Television Stations 3, 5, 6, 13, 15, … Channel 15 Filtering removes all but Channel 15 Television

11. 11 ASU MAT 591: Opportunities in Industry! Radio Radio Stations Station 100.7 Filtering removes all but Station 100.7 Radio Stations 91.5, 96.9, 100.7 Radio

12. 12 ASU MAT 591: Opportunities in Industry! Radio Radio Stations Station 96.9 Filtering removes all but Station 96.9 Radio Stations 91.5, 96.9, 100.7 Radio

13. 13 ASU MAT 591: Opportunities in Industry! Vision Book Filtering removes all but a book Scene of a Room: walls, books, desks, chairs, windows,… Robot vision

14. 14 ASU MAT 591: Opportunities in Industry! Vision Scene of a Room Table Filtering removes all but a table Scene of a Room: walls, books, desks, chairs, windows,… Robot vision

15. 15 ASU MAT 591: Opportunities in Industry! Graphics to build a scene Synthesis Scene of a Room Descriptor of scene is D(w) All Room Contents

16. 16 ASU MAT 591: Opportunities in Industry! Data compression Signal Filter that eliminates less important data. Approximation of Signal

17. 17 ASU MAT 591: Opportunities in Industry! Data compression goal Signal Filter that eliminates less important data. Approximation of Signal

18. 18 ASU MAT 591: Opportunities in Industry! An Example of a Processing Architecture

19. 19 ASU MAT 591: Opportunities in Industry! Format Correct Errors PreprocessRestore Analyze Recognize The Example Architecture Format Descriptions Data Correct Errors Communications Preprocess Normalize Remove Noise Remove Distortions Restore Remove Sensor Effects Analyze Decompose Signals Recognize Label Signals Will Discuss in more detail!

20. 20 ASU MAT 591: Opportunities in Industry! Format Correct Errors PreprocessRestore Analyze Recognize Preprocess Descriptions Data Normalize Remove Noise Remove Distortions

21. 21 ASU MAT 591: Opportunities in Industry! Noisy Input Image Fourier Based Noise Filtering From Jason Plumb at http://noisybox.net/weblog/ Clearer Output Image Mostly Noise so is Zeroed Mostly Signal Fourier Synthesis Fourier Analysis Fourier Transform and Filter the Noise

22. 22 ASU MAT 591: Opportunities in Industry! Filtering and Enhancing Data From Mathworks homepage at http://www.mathworks.com/ Math to follow

23. 23 ASU MAT 591: Opportunities in Industry! Filtering: Analysis Image Analysis

24. 24 ASU MAT 591: Opportunities in Industry! Filtering: Removing Noise Filtering: removes noise Image

25. 25 ASU MAT 591: Opportunities in Industry! Filtering: Synthesis Enhanced Image Synthesis

26. 26 ASU MAT 591: Opportunities in Industry! Filtering Enhanced Filtering: removes noise Image Analysis Synthesis

27. 27 ASU MAT 591: Opportunities in Industry! Enhancing the Data: Linear map I=Intensity I1I1 p(I 1 ) Input Image Intensity Histogram I2I2 p(I 2 ) Output Image Intensity Histogram (more contrast) I1I1 I2I2 I 2 = m* I 1 Enhance (stretch) Using Linear Mapping

28. 28 ASU MAT 591: Opportunities in Industry! Warping data From Mathworks homepage at http://www.mathworks.com/ Suppose we have unwanted camera motion.

29. 29 ASU MAT 591: Opportunities in Industry! Warping data From Mathworks homepage at http://www.mathworks.com/ We can correct motion errors if we know motion model.

30. 30 ASU MAT 591: Opportunities in Industry! Warping data From Mathworks homepage at http://www.mathworks.com/

31. 31 ASU MAT 591: Opportunities in Industry! Warping Correction is an Inverse Function Warping Correction Warping

32. 32 ASU MAT 591: Opportunities in Industry! Linear Algebra to Flip x1x1 y1y1 x2x2 y2y2

33. 33 ASU MAT 591: Opportunities in Industry! Linear Algebra to Flip x1x1 y1y1 x1x1 x2x2 x 2 =- x 1 y1y1 y2y2 y 2 =y 1 x2x2 y2y2

34. 34 ASU MAT 591: Opportunities in Industry! Linear Algebra to Flip x1x1 y1y1 x1x1 x2x2 x 2 =- x 1 y1y1 y2y2 y 2 =y 1 I(x 1,y 1 ) x2x2 y2y2 x2x2 y2y2

35. 35 ASU MAT 591: Opportunities in Industry! Linear Algebra to Flip x1x1 y1y1 x1x1 x2x2 x 2 =- x 1 y1y1 y2y2 y 2 =y 1 I(x 1,y 1 ) I(x 2,y 2 ) x2x2 y2y2 x2x2 y2y2 x2x2 y2y2

36. 36 ASU MAT 591: Opportunities in Industry! Linear Algebra to Flip x1x1 y1y1 x1x1 x2x2 x 2 =- x 1 y1y1 y2y2 y 2 =y 1 I(x 1,y 1 ) I(f(x 1 ),g(y 1 )) x2x2 y2y2 x2x2 y2y2 x2x2 y2y2

37. 37 ASU MAT 591: Opportunities in Industry! Linear Algebra to Flip x1x1 y1y1 x1x1 x2x2 x 2 =- x 1 x2x2 y2y2 y1y1 y2y2 y 2 =y 1 I(x 1,y 1 ) I(x 2,y 2 )=I(f(x 1 ),g(y 1 )) x2x2 y2y2 x2x2 y2y2 x2x2 y2y2

38. 38 ASU MAT 591: Opportunities in Industry! Linear Algebra to Flip x1x1 y1y1 x2x2 x1x1 x 1 =- x 2 x2x2 y2y2 y2y2 y1y1 y 1 =y 2 I(x 2,y 2 )

39. 39 ASU MAT 591: Opportunities in Industry! Linear Algebra to Flip x1x1 y1y1 x2x2 x1x1 x 1 =- x 2 x2x2 y2y2 y2y2 y1y1 y 1 =y 2 I(f -1 (x 2 ), g -1 (y 2 )) I(x 2,y 2 )

40. 40 ASU MAT 591: Opportunities in Industry! Linear Algebra to Flip x1x1 y1y1 x1x1 x2x2 x 2 =- x 1 x2x2 y2y2 y1y1 y2y2 y 2 =y 1 I (x 1,y 1 )=I(f -1 (x 2 ), g -1 (y 2 )) I (x 2,y 2 )

41. 41 ASU MAT 591: Opportunities in Industry! Linear Algebra to Flip and Shrink x1x1 y1y1 x2x2 y2y2

42. 42 ASU MAT 591: Opportunities in Industry! Linear Algebra to Flip and Shrink x1x1 y1y1 x1x1 x2x2 x2x2 y2y2 y1y1 y2y2 y 2 = -0.5 * y 1 x 2 = 0.5 * x 1

43. 43 ASU MAT 591: Opportunities in Industry! Correcting warped data (camera motion) From Mathworks homepage at http://www.mathworks.com/ If we can determine f(), g(), f -1 (), and g -1 (), then we can correct camera motion!

44. 44 ASU MAT 591: Opportunities in Industry! Format Correct Errors PreprocessRestore Analyze Recognize Restoration Restore Descriptions Data Remove Sensor Effects

45. 45 ASU MAT 591: Opportunities in Industry! Restoring data for smear, optics,… From Mathworks homepage at http://www.mathworks.com/ Uses Linear Systems Theory Next Smear and optics can be viewed as filters that can degrade an image!

46. 46 ASU MAT 591: Opportunities in Industry! Restoring data for smear, optics,… From Mathworks homepage at http://www.mathworks.com/ Uses Linear Systems Theory Next Restoration

47. 47 ASU MAT 591: Opportunities in Industry! Restoration: Analysis Image Analysis

48. 48 ASU MAT 591: Opportunities in Industry! Filtering: Removing Smear Smr -1 (w x,w y ) is a filter that removes smear or restores the original object. Image

49. 49 ASU MAT 591: Opportunities in Industry! Filtering: Synthesis Object Image Synthesis

50. 50 ASU MAT 591: Opportunities in Industry! Filtering Image Restored to best look like original Object Image Object Smear inverted as a filter

51. 51 ASU MAT 591: Opportunities in Industry! Restoring data for smear, optics,… From Mathworks homepage at http://www.mathworks.com/ Uses Linear Systems Theory Image(w x,w y ) Next

52. 52 ASU MAT 591: Opportunities in Industry! Restoring data for smear, optics,… From Mathworks homepage at http://www.mathworks.com/ Uses Linear Systems Theory Image(w x,w y ) Smr(w x,w y )*Image(w x,w y ) Next

53. 53 ASU MAT 591: Opportunities in Industry! Restoring data for smear, optics,… From Mathworks homepage at http://www.mathworks.com/ Uses Linear Systems Theory Image(w x,w y ) Smr(w x,w y )*Image(w x,w y ) Image(w x,w y ) *Smr -1 (w x,w y )* Smr(w x,w y ) Image(w x,w y )= Image(w x,w y ) *1(w x,w y )

54. 54 ASU MAT 591: Opportunities in Industry! Format Correct Errors PreprocessRestore Analyze Recognize Synthesis and Analysis Descriptions Data Decompose / Compose Signals - Transforms: Fourier, SVD, Wavelets - Statistical Analysis: parametric and non-parametric Synthesize Analyze

55. 55 ASU MAT 591: Opportunities in Industry! Fourier Transform White Light Out Fourier Analysis Fourier Synthesis White Light In

56. 56 ASU MAT 591: Opportunities in Industry! Fourier Transform MagnitudePhase From Wolfram homepage at http://documents.wolfram.com

57. 57 ASU MAT 591: Opportunities in Industry! Radon Transform From Mathworks homepage at http://www.mathworks.com/

58. 58 ASU MAT 591: Opportunities in Industry! Wavelet Transform From Wolfram homepage at http://documents.wolfram.com

59. 59 ASU MAT 591: Opportunities in Industry! Common Transforms l Fourier l Discrete fourier l Cosine l Sine l Hough l Hadamard l Slant l Karhunen-Loeve l Fast KL l SVD l Sinusoidal

60. 60 ASU MAT 591: Opportunities in Industry! Statistics From Mathworks homepage at http://www.mathworks.com/

61. 61 ASU MAT 591: Opportunities in Industry! Format Correct Errors PreprocessRestore Analyze Recognize Recognition Recognize Descriptions Data Label Signals - Signal Detection - Pattern Recognition - Artificial Intelligence

62. 62 ASU MAT 591: Opportunities in Industry! Feature 1 Feature 2 * Feature 1 Feature 2 Class 1 (daisy) Class 2 (rose) Class 3 (sun flower) * Features are mathematical measurements Pattern Recognition Classification Bayesian Neural nets Nearest neighbors Linear Transforms: Fourier, Wavelet, … Statistics: mean, st. dev, … Shape: Fourier, Hough, Moments Texture: Cooccurrence, Eigen Filters, … Analysis ToolsFeatures Feature 1: Hough measure Feature 2: 3 rd Eigen Filter Analysis

63. 63 ASU MAT 591: Opportunities in Industry! Mathematical Decisions z o Class 1 is z Class 2 is o o o o o o o o o o o o o z z z z z z z z z z z z z f1f1 f2f2 z How do we separate the classes? o o o o o

64. 64 ASU MAT 591: Opportunities in Industry! Mathematical Decisions z o Class 1 is z Class 2 is o o o o o o o o o o o o o z z z z z z z z z z z z z f1f1 f2f2 z Linear decision o o o o o

65. 65 ASU MAT 591: Opportunities in Industry! Mathematical Decision z o Class 1 is z Class 2 is o o o o o o o o o o o o o z z z z z z z z z z z z z f1f1 f2f2 z Linear decision o o o o o

66. 66 ASU MAT 591: Opportunities in Industry! Mathematical Decision z o Class 1 is z Class 2 is o o o o o o o o o o o o o z z z z z z z z z z z z z f1f1 f2f2 z Quadratic decision o o o o o

67. 67 ASU MAT 591: Opportunities in Industry! Mathematical Decision z Class 1 is z z z z z z z z z z z z z z f1f1 f2f2 z

68. 68 ASU MAT 591: Opportunities in Industry! Mathematical Decision o Class 2 is o o o o o o o o o o o o f1f1 f2f2 o o o o o

69. 69 ASU MAT 591: Opportunities in Industry! Mathematical Decision z o Class 1 is z Class 2 is o o o o o o o o o o o o o z z z z z z z z z z z z z f1f1 f2f2 z 3 o o o o o

70. 70 ASU MAT 591: Opportunities in Industry! Isolate Object: Segmentation From Mathworks homepage at http://www.mathworks.com/ AnalysisSynthesis

71. 71 ASU MAT 591: Opportunities in Industry! Analyze Object: Features - Length - Width - Contour - Orientation - Edges - Skeleton - Texture Details - Intensity From Mathworks homepage at http://www.mathworks.com/

72. 72 ASU MAT 591: Opportunities in Industry! Matched Filtering (registration) From Mathworks homepage at http://www.mathworks.com/ Input Image or I in (x,y)

73. 73 ASU MAT 591: Opportunities in Industry! Matched Filtering (registration) From Mathworks homepage at http://www.mathworks.com/ Exemplar (reference) or I ref (x,y) Input Image or I in (x,y)

74. 74 ASU MAT 591: Opportunities in Industry! Matched Filtering (registration) From Mathworks homepage at http://www.mathworks.com/ Exemplar (reference) or I ref (x,y) Input Image or I in (x,y) x2x2 error x

75. 75 ASU MAT 591: Opportunities in Industry! Matched Filtering (registration) From Mathworks homepage at http://www.mathworks.com/ Exemplar (reference) or I ref (x,y) Input Image or I in (x,y) x2x2 error x Actually search form min of x,y simultaneously!

76. 76 ASU MAT 591: Opportunities in Industry! Format Correct Errors PreprocessRestore Analyze Recognize Image Processing: Summary Format Descriptions Data Correct Errors Communications Preprocess Normalize Remove Noise Remove Distortions Restore Remove Sensor Effects Analyze Decompose Signals Recognize Label Signals

77. 77 ASU MAT 591: Opportunities in Industry! References l Fundamentals of Image Processing by Jain l Digital Image Analysis by Gonzalez and Wintz l Pattern Recognition by Fukunaga l Pattern Recognition Principles Tou and Gonzalez l Detection, Estimation, and Modulation Theory by Van Trees l Pattern Classification by Duda and Hart l Robot by Hans Moravec (graphics from www.amazon.com)

78. 78 ASU MAT 591: Opportunities in Industry! Touch Vision Hearing Smell Taste Signal Processing: 50 years from now Touch Vision Hearing Smell Taste Robotic Evolved Hmmm. Vision

79. 79 ASU MAT 591: Opportunities in Industry! Touch Vision Hearing Smell Taste Signal Processing: 50 years from now Touch Vision Hearing Smell Taste Robotic Evolved Wow! Vision

80. 80 ASU MAT 591: Opportunities in Industry! Touch Vision Hearing Smell Taste Signal Processing: 50 years from now Touch Vision Hearing Smell Taste Robotic Evolved I see, therefore, am I? Hmmm. Vision