1. Image Analysis
Manipulate an image to extract information to help solve a problem.
Preprocessing - get rid of unnecessary information
Data reduction - transform the image to a useable form
Feature analysis - make inferences about the image.

2. Image Analysis Preprocessing Noise reduction Gray level quantization
Spatial quantization
Finding regions of interest
Reducing the number of bits

3. Image Analysis
Data reduction process, in which the image(s) are transformed into a more convenient form.
RGB  HSL
Image subtraction
Histogram
Feature extraction

4. Image Analysis Feature analysis - Specific results Blood cell counting
Tumor size and location
3D model

5. Image Analysis Geometric Noise Reduction Edge Detection
Resizing
Rotating
Noise Reduction
Image Smoothing
Median Filtering
Edge Detection
Searching for discontinuities
Histogram slicing
Blob detection

6. Geometric Transformation

7. Interpolation used to simplify data for further processing
Interpolation to determine image values at integer pixel locations of transformed image.
Numerous interpolation schemes exist
2 simple methods are
Nearest neighbor: assign value of nearest known point
Bilinear interpolation: example on next slide
Interpolation used to simplify data for further processing

8. Nearest Neighbor Interpolation
Interpolate this point
To this value

9. Linear Interpolation

10. Bi-linear Interpolation
I(x+1,y)
I(x+1,y+1)
I(x,y+1)
I(x,y)
I(x’,y’)
0  (a,b)  1
I(x’,y) = (1-a)I(x,y) + (a)I(x+1,y)
I(x’,y+1) = (1-a)I(x,y+1) + (a)I(x+1,y+1)
I(x’,y’) = (1-b)I(x’,y) + (b)I(x’,y+1)
I(x’,y’) = (1-b)(1-a)I(x,y) + (1-b)(a)I(x+1,y)
+ (b)(1-a)I(x,y+1) + (a)(b)I(x,y)

11. Bi-linear Interpolation
I(x’,y’)

12. Neighborhood Operations

13. Objectives Why are neighborhoods important?
What is linear convolution?
discrete
templates, masks or filters
algorithm mechanics
graphical interpretation
Describe non-linear operators
maximum
minimum
median
What is tiling?

14. Why are neighbourhoods important?
pixel

15. Because… Provide context for individual pixels.
Relationships between neighbors determine image features.
Neighborhood operations:
noise reduction
edge enhancement
zooming

16. Noise reduction
Edge Enhancement
Zooming

17. Neighbourhood Operations
Linear convolution (*)
A*B*C*D = B*C*D*A = ….
Non-linear operators
median, max, min, ...

18. Convolution versus Spectral
We learnt two methods of processing images:
Convolution
Spectral
We analyzed and demonstrated how to build a processor (systolic, pipelined, parallel, cellular automaton) for 1D convolution.
1D convolution is used in speech processing and in polynomial multiplication.
We will use visualized animations now to show in more detail how 2D convolution works for images.
This should convince you how important it is to do convolution quickly in modern Spectral Architectures, especially for 3D etc.

19. 2D Convolution
We will show more examples of convolution now, especially for 2D data
Consists of filtering an image A using a filter (mask, template) B.
Mask is a small image whose pixel values are called weights.
Weights modify relationships between pixels.

20.  = A C B Filter, mask or template Input image Convolved Image A1,1
2 2
3 3
4 4

21. A1,1
A1,2
A2,1
A2,2
B1,1
B1,2
B2,1
B2,2
A1,3
A1,4
A2,3
A2,4
A3,1
A3,2
A3,3
A3,4
A4,1
A4,2
A4,3
A4,4
A1,1B1,1
A1,2B1,2
A2,1B2,1
A2,2B2,2
C1,1=
A1,1B1,1 
A1,2B1,2 
A2,1B2,1 
A2,2B2,2

22. A1,1
A1,2
A2,1
A2,2
A1,3
A1,4
A2,3
A2,4
A3,1
A3,2
A3,3
A3,4
A4,1
A4,2
A4,3
A4,4
B1,1
B1,2
B2,1
B2,2
A1,2B1,1
A1,3B1,2
A2,2B2,1
A2,3B2,2
C1,2=
A1,2B1,1 
A1,3B1,2 
A2,2B2,1 
A2,3B2,2

23. A1,1
A1,2
A2,1
A2,2
A1,3
A1,4
A2,3
A2,4
A3,1
A3,2
A3,3
A3,4
A4,1
A4,2
A4,3
A4,4
B1,1
B1,2
B2,1
B2,2
A1,3B1,1
A1,4B1,2
A2,3B2,1
A2,4B2,2
C1,3=
A1,3B1,1 
A1,4B1,2 
A2,3B2,1 
A2,4B2,2

24. A1,3
A1,4
A2,3
A2,4
A3,1
A3,2
A3,3
A3,4
A4,1
A4,2
A4,3
A4,4
A1,1
A1,2
A2,1
A2,2
A2,1B1,1
A2,2B1,2
B1,1
B1,2
B2,1
B2,2
A3,1B2,1
A3,2B2,2
C2,1=
A2,1B1,1 
A2,2B1,2 
A3,1B2,1 
A3,2B2,2

25. A1,1
A1,2
A2,1
A2,2
A1,3
A1,4
A2,3
A2,4
A3,1
A3,2
A3,3
A3,4
A4,1
A4,2
A4,3
A4,4
B1,1
B1,2
B2,1
B2,2
B1,1
B1,2
B2,1
B2,2
B1,1
B1,2
B2,1
B2,2

26. Mathematical Notation
A1,1B1,1 
A1,2B1,2 
A2,1B2,1 
A2,2B2,2

27.  Convolution = A C B Filter, mask or template Input image Convolved4 7 9 6 23 21  -1 2 4 3 8 9 = 9 26 19 -1 2 3 5 9 6 10 16 27 17
2 2
3 3
4 4

28. Convolution size Image size = M1 N1 Mask size = M2 N2
M1- M2 +1 N1-N2+1 N1 N2
N1-N2+1
Typical Mask sizes
= 33, 5 5, 77,
9 9, 1111
What is the convolved
image size for a 128 
 128 image and
7  7 mask?

29. * =
We convolve with 9*9 averaging filter

30. Nonlinear Neighbourhood Operations
Maximum
Minimum
Median
We discussed already sorter architecture (three variants – pipelined, butterfly combinational and sequential controller). It can be used for all these operations, and also for other non-linear operators

31. Max and Min Operations C1,2= 61 62 57 60 59 65 63 56 55 58 49 53 45 1
63=max, 59=min

32. Median Operation 9 8 7 6 5 4 3 2 1 65 63 62 61 62 57 60 59 65 63 56 55 58 49 53 45 1 60 62 60 59 63 65 56 55 58 57
rank 59 58 57 56
C1,2= 59 55

33. 9x9 Median

34. Edge Detection
What do we mean by edge detection?
What is an edge?

35. What is Edge Detection? Redraws the image with only the edges showing
Detects large intensity transitions between pixels
Redraws the image with only the edges showing

36. What is an Edge?
Edge easy to find

37. What is an Edge?
Where is edge? Single pixel wide or multiple pixels?

38. What is an Edge? Noise is here
Noise: have to distinguish noise from actual edge

39. What is an Edge?
Is this one edge or two?

40. What is an Edge?
Texture discontinuity

41. Edge Detection – so what is an edge to be detected?
A large change in image brightness of a short spatial distance
Edge strength = (I(x,y)-I(x+dx,y))/dx
But this general definition still allows for many theories, software implementation and hardware architectures.

42. Now we will discuss and illustrate various kinds of filter operators

43. Edge Detection Filters
High - Pass Filtering Eliminates Uniform Regions (Low Frequencies)
edge “detection” or “enhancement”

44. Edge Detection Filters

45. Edge Detection Filters
Edge Detection Continued
Sum of Kernel Coefficients = 0
differences in signs emphasize differences in pixel values
reduces average image intensity
Negative pixel values in output?

46. Edge Direction
vertical
horizontal
diagonal

47. Directional High Pass Filters

48. Convolution Edge Detection using Sobel and similar operators

49. Example of Sobel Operator

50. Sobel Edge Detection

51. Convolution Application Examples
We apply Sobel Operator
--Edge Detection
Column Mask
Row Mask
as mask to a sub-field of a picture
p0, p1, p2
p3, p4, p5
p6, p7, p8 *
= (p6-p0)+2(p7-p1)+(p8-p2)
The final step of the convolution equation, dividing by the weight , must be ignored
We can learn from the result obviously
The result of the above calculation for column mask is horizontal difference
With Row Mask we will get vertical difference

52. Convolution Application Examples
--Edge Detection with Sobel Operator
The weight of a mask determines the grey level of the
image after convolution.
Like the weight of Sobel Mask W
W= (-1) + (-2) + (-1) = 0
The resulting image lost its “lightness” to be dark.

53. Sobel Operator

54. Sobel Operator -1 -2 -1 0 0 0 1 2 1 -1 0 1 -2 0 2 S2 = S1=
S2 =
S1=
Edge Magnitude =
Edge Direction =

55. Comparison of Edge Detection Algorithms
Sobel
Canny
Prewitt
Ticbetts

56. Edge Direction
Assymetric kernels detect edges from specific directions
NorthEast
East
North

57. Robinson Operator

58. Robinson Compass Masks
Arrows show edge directions

59. Roberts Operator

60. Roberts Operator or
1 0
0 -1
0 1 +
Does not return any information about the orientation of the edge

61. Prewitt Operator -1 -1 -1 0 0 0 1 1 1 -1 0 1 P2 = P1= Edge Magnitude =
P2 =
P1=
Edge Magnitude =
Edge Direction =

62. Edge Detection Filters
Prewitt Row

63.
Original and filtered cow

64. Edge Detection Filters: compare Prewitt and Sobel
Edge Detection (continued)
First Order (Gradient) Kernels
Prewitt Row
Sobel Row
1 0 -1
2 0 -2
Combine Row and Column Operators

65. first derivative
1D Laplacian Operator
second derivative

66. 2D Laplacian Operator Convolution masks approximating a Laplacian
This is just one example of Laplacian, we can use much larger window

67.
Input Mask Output

68. Image Processing Operations for Early Vision:
Edge Detection

69. Reminder: Effect of Filters
low
high

70. Edges … are the important part of images intensity color edges
simplest, least robust
intensity
color
edges
textures
contours
condensation...
most difficult, most robust
There are many letters B occluded by black shape here. How to find them?

71. Image Processing Operations
Edge Detection
Edges are curves in the image plane across which there is a “significant” change in image brightness.
The goal of edge detection is the construction of an idealized line drawing

72. Pixels on edges

73. Edges found

74. Edge effects: rarely ideal edges
Not all information is created equal...

75. Causes of edges Depth discontinuity Surface orientation discontinuity
One surface occludes another
Surface orientation discontinuity
the edge of a block
reflectance discontinuity
texture or color changes
illumination discontinuity
shadows

76. Edges: causes What are they? Why?
four physical events that cause image edges...

77. Edges: causes What are they? Why? discontinuities in
four physical events that cause image edges...
surface color/intensity
surface normal
discontinuities in
depth
lighting (specularities)

78. Edges: causes
Edges are image locations with a local maximum in image gradient in the direction of that gradient
(steepness)

79. Edge Detection
Finding simple descriptions of objects in complex images
find edges
interrelate edges

80. Examples of edges

81. Edges are not ideal...
One idea to detect edges is to differentiate the image and look for places where the brightness undergoes a sharp change
Consider a 1-D example. Below is an intensity profile for a 1-D image.

82. Edge Detection Below we have the derivative of the previous graph.
Here we have a peak at x=18, x=50 and x=75.
These errors are due to the presence of noise in the image.

83. Finding Edges Image Intensity along a line
First derivative of intensity
Smoothed via convolving with gaussian

84. Edge Detection
This problem is countered by convolving a smoothing function along with the differentiation operation.
The mathematical concept of convolution allows us to perform many useful image-processing operations.

85. Image Processing Operations
One standard form of smoothing is to use a Gaussian function.
Now using the idea of convolving with the Gaussian function
we can revisit the 1-D example.

86. Convolving to find edges
With the convolving applied we can more easily see the edge at x=50.
Using convolving we are able to discover where edges are located and this allows us to make an accurate line drawing.

87. Edge Detection by Convolution
Here is an example of using convolving in an 2-D picture of Mona Lisa

88. Zero Crossing
edge
derivative
Second derivative

89. Edge

90. Edge Parameters

91. Remainder: How the Point Detection Mask operates on one color image

92. Edge Detection uses Convolution
Goal: to find regions of an image with locally maximal gradient magnitude.
(1) Smooth the image to reduce the effects of noise
Replace each pixel by a weighted sum of its neighbors.
weight “mask”
old Image
new Image 34 37
137
148 35
210
210
210 1 2 1
I(x,y) = S wij I(x+i,y+j)
i,j = -1 1 29 46
141
140 x 43
210
210
210 2 4 2 = 34
130
149
131 60
210
210
210 1 2 1
new
old 41
142
152
144 72
210
210
210
weights
(scaled)

93. Edge Detection uses Convolution
Goal: to find regions of an image with locally maximal gradient magnitude.
(1) Smooth the image to reduce the effects of noise
Replace each pixel by a weighted sum of its neighbors.
weight “mask”
old Image
new Image 34 37
137
148 35 62
116
143 1 2 1
I(x,y) = S wij I(x+i,y+j)
i,j = -1 1 29 46
141
140 x 43 76
122
141 2 4 2 = 34
130
149
131 60
102
136
140 1 2 1
new
old 41
142
152
144 72
112
145
143
weights
(scaled)
It’s possible to do this one dimension at a time...
original image
smoothed vertically
smoothed horizontally

94. Edge Detection uses Convolution
Goal: to find regions of an image with locally maximal gradient magnitude.
(1) Smooth the image to reduce the effects of noise
I(x,y) = S wij I(x+i,y+j)
i,j = -1 1
Replace each pixel by a weighted sum of its neighbors.
new
old
weights
(2) Estimate the gradient at each pixel
same procedure (convolution)
Use another mask of weights -- this time to approximate taking derivatives.
with new weights 1
In the y direction -1 -1 1
In the x direction d dx I d dy I
= Ix
= Iy

95. Edge Detection uses Convolution
Goal: to find regions of an image with locally maximal gradient magnitude.
(1) Smooth the image to reduce the effects of noise
I(x,y) = S wij I(x+i,y+j)
i,j = -1 1
Replace each pixel by a weighted sum of its neighbors.
new
old
weights
(2) Estimate the gradient at each pixel
same procedure with new weights (convolution)
Use another mask of weights -- this time to approximate taking derivatives. 1 -1 1 y -1 x
(3) Find the gradient magnitude and thin the resulting lines.
 I(x,y) = sqrt(Ix + Iy ) 2
q = atan2(Ix, Iy) q
Seek out maxima in the gradient direction q.

96. Edge Detection uses Convolution
Goal: to find regions of an image with locally maximal gradient magnitude.
(1) Smooth the image to reduce the effects of noise
I(x,y) = S wij I(x+i,y+j)
i,j = -1 1
Replace each pixel by a weighted sum of its neighbors.
new
old
weights
(2) Estimate the gradient at each pixel
same procedure with new weights (convolution)
Use another mask of weights -- this time to approximate taking derivatives. 1 -1 1 y -1 x
(3) Find the gradient magnitude and thin the resulting information.
In the gradient direction, look for local maxima:
 I(x,y) = sqrt(Ix + Iy ) 2
q = atan2(Ix, Iy)
(4) Choose a threshold -- any gradients above it classify as edges.

97. Theory of Gradient Based Edge Detection

98. Formal Model of Edge

99. Formal Model of Edge

100. Formal Model of Edge (cont)

101. Formal Model of Edge (cont)

102. Formal Model of Edge (cont)

103. Formal Model of Edge: Roberts
Formal Model of Edge (cont)
Formal Model of Edge: Roberts

104. Formal Model of Edge: Laplacian and Marr-Hildreth
Formal Model of Edge (cont)
Formal Model of Edge: Laplacian and Marr-Hildreth

105. Formal Model of Edge (cont)

106. Formal Model of Edge (cont)

107. Thresholds
Thresholds are important, done before or during edge detection.
original image
very high threshold

108. Thresholds
Thresholds
original image
very high threshold

109. Thresholds
original image
very high threshold

110. Thresholds
original image
very high threshold
reasonable

111. Thresholds original image very high threshold reasonable too low !
this all takes time...