1. Exercise class 13 : Image Segmentation
Many slides from Steve Seitz

2. Image segmentation Segment – A region of the image which:
Belongs to a single object.
Looks uniform (gray levels, color…)
Have the same attributes (texture…)
Moves in a similar way (motion segmentation) …
Segmentation is a problem of assigning attributes (pixels) to objects
Compact representation of object to groups (also called grouping, perceptual organization, fitting)
Problem to segment – huge amount of data (pixels), we don’t know in which of the pixels should we use

3. Why do we need it ?
The first step towards higher level vision (object recognition etc.)
Treat different segments differently. (compression, histogram manipulations…)
Use it to find regions which are related to a single “model” (motion computation, texture analysis…)
Many more applications.
We want to treat object as a hole (high level vision – extracting the dog from the background) – very hard
In many algorithms we would like to treat differently different part and adjust the parameters to each part separately
Define regions that belong to single model (motion model)
Some examples: segmenting to shots in video, matching close area in images, locating faces / people in pictures, finding buildings in satellite images, searching in images database – compact representation of images

4. Image Segmentation
Looking at some more examples – there are many possible segmentation with different hierarchy between them
Another hard question – what is does the user want to see (level of details) – depends on the application

5. Image segmentation How do you pick the right segmentation?
Bottom up segmentation:
- Tokens belong together because
they are locally coherent.
Top down segmentation:
- Tokens grouped because
they lie on the same object.

6. “Correct” segmentation
There may not be a single correct answer.
Gestalt Laws seek to formalize what is an object
proximity, similarity, continuation, closure, common fate
Segmentation can be thought of a partition problem
Many approaches proposed
cues:
color, regions, contours, texture, motions
automatic vs. user-guided
no clear winner
Using low level image features to try and extract the required grouping
The human vision has usually very good ability to segment. It was shown that the context (top down approach) affects the segmentation
The gestalt principles offer explanation to grouping in different examples and situations

7. Main Approaches Histogram-based segmentation Region-based segmentation
Edge detection
Region growing
Region splitting and merging.
Clustering
K-means
Mean shift
Graph Based Clustering
Our visual system is proof that automatic methods are possible
classical image segmentation methods are automatic

8. Example: Splitting text and background
Thresholding: define a threshold T such that each pixel x,y is “text” iff I(x,y)<T.
How do we determine the threshold ?
Just choose 128 as a threshold (problematic for dark images)
Use the median/mean (both are not good, as most of the paper is white)

9. Histogram-based segmentation
Goal
Break the image into K regions (segments)
Solve this by reducing the number of gray levels / colors to K and mapping each pixel to the closest gray level / color

10. Histogram-based segmentation
Goal
Break the image into K regions (segments)
Solve this by reducing the number of gray levels / colors to K and mapping each pixel to the closest gray level / color
Possible threshold – minimum on histogram (between larger values)
Here’s what it looks like if we use two gray levels

11. Problems with that approach
Noise is considered as text.
Holes and discontinuities in the resulting text.
Changes in the illumination
Solutions:
1. Enhance the result using morphological operations
2. Adaptive threshold.
3. Use spatial information ! A

12. Adaptive Threshold Use a running window..
Assume a constant illumination in each window.
For each window: compute a threshold and use it to segment the pixel in the middle of the window.
Problem: This scheme will find text in blank areas!
Possible Solution: we can try to recognize that these regions are unimodal.

13. Segmentation as clustering
Consider the image as a set of points in the
N-dimensional space:
Color Histograms: p=(R(x,y),G(x,y),B(x,y)) in R3. (ignores the spatial information.)
Gray level images: p=(x,y,I(x,y)) in R3
Color images: p =(x,y,R(x,y),G(x,y),B(x,y)) in R5
Texture: p= (x,y,vector_of_fetures) …
From this stage, we forget the meaning of each coordinate. We deal with arbitrary set of points.
Important: We need to find the appropriate space in which the distance between points is meaningful !
Clustering problem is assigning tokens to groups. In the case of images as we said the assignment can be based on low level features such as color, texture, proximity
Need to define a distance matrix based on the features

14. Segmentation using simple clustering model
Two natural algorithms
Clustering by splitting the entire data set is one cluster and we split it to get a good clustering
Clustering by merging each point is a different cluster and we merge similar clusters
Usually the simple methods use greedy algorithms
Important issues :
Define a proper distance between data points
Define a proper distance between clusters
How many clusters are there?
Main difficulty –
Defining threshold : when should we stop the algorithm?
Very large number of pixels : hard to look for best split / merge.

15. Segmentation as Clustering
How to choose the representative colors?
This is a clustering problem! G G R R
Objective
Each point should be as close as possible to a cluster center
Minimize sum squared distance of each point to closest center
As opposed to simple greedy algorithms, we want to represent the problem as aiming for an objective function that represents good clustering and look for the best possible representation
Clustering problem is assigning tokens to groups. In the case of images as we said the assignment can be based on low level features such as color, texture, proximity
Need to define a distance matrix based on the features
In the following examples – distance in 2D but this can be thought also as a representation of distances in higher dimensional feature space
Instead of using Ad-hook solutions try to define a objective function and maximize/minimize it

16. Break it down into subproblems
Suppose I tell you the cluster centers ci
Q: how to determine which points to associate with each ci?
A: for each point p, choose closest ci
Suppose I tell you the points in each cluster
Q: how to determine the cluster centers?
We notice that we can split the problem to 2 easy problems : if we know the cluster centers we can easily allocate a pixel to cluster. If we know the allocation of points to cluster we can easily calculate the best cluster centers
A: choose ci to be the mean of all points in the cluster

17. K-means clustering K-means clustering algorithm
Randomly initialize the cluster centers, c1, ..., cK
Given cluster centers, determine points in each cluster
For each point p, find the closest ci. Put p into cluster i
Given points in each cluster, solve for ci
Set ci to be the mean of points in cluster i
If ci have changed, repeat Step 2
Java demo:
Properties
Will always converge to some solution
Can be a “local minimum”
does not always find the global minimum of objective function:

18. Example: Segmentation using Color + Texture

19. K-means limitations Converges to a local minima
Constant number of clusters, k
Cluster’s shape is typically spherical

20. Mean Shift
A non-parametric technique for analyzing complex multimodal feature spaces and estimating the stationary points (modes) of the underlying probability density function without explicitly estimating it.
Ukrainitz & Sarel

21. An Example Feature Space
Ukrainitz & Sarel

22. An Example Feature Space
Parametric Density Estimation?
Ukrainitz & Sarel

23. Distribution of identical billiard balls
Mean Shift Clustering
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Ukrainitz & Sarel

24. Distribution of identical billiard balls
Mean Shift Clustering
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Ukrainitz & Sarel

25. Distribution of identical billiard balls
Mean Shift Clustering
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Ukrainitz & Sarel

26. Distribution of identical billiard balls
Mean Shift Clustering
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Ukrainitz & Sarel

27. Distribution of identical billiard balls
Mean Shift Clustering
Region of
interest
Center of
mass
Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Ukrainitz & Sarel

28. Distribution of identical billiard balls
Mean Shift Clustering
Region of
interest
Center of
mass
Objective : Find the densest region
Distribution of identical billiard balls
Ukrainitz & Sarel

29. Synthetic Example
Ukrainitz & Sarel

30. Real Data Example
Ukrainitz & Sarel

31. Mean shift based image segmentation
Ukrainitz & Sarel

32. Mean shift based image segmentation
Ukrainitz & Sarel

34. Segmentation by min (s-t) cut [Boykov 2001]
Cut in a graph
set of edges whose removal makes a graph disconnected
cost of a cut:
Find minimum cut
gives you a segmentation of the graph

35. Segmentation by min (s-t) cut [Boykov 2001]
min cut s t
Graph construction:
node for each pixel, link between pixels
specify a few pixels as foreground and background
create an infinite cost link from each bg pixel to the “t” node
create an infinite cost link from each fg pixel to the “s” node
compute min cut that separates s from t
how to define link cost between neighboring pixels?
We can see segmentation as a problem of finding good cut in a graph. All the pixels in the same connected component will be considered as a single cluster
Finding - Min cut, we showed in previous lesson (on stitching) that this is a problem of flow in graph (min-cut max flow) and can be solved in polynomial time

36. Segmentation by min (s-t) cut [Boykov 2001]
how to define link cost between neighboring pixels?
Assign large weight on edges of pixels that you expect to be in the same group (body / background)
F(i)- gray level value in pixel I
- normalization factor

37. Examples for user assistance Min-cut segmentation
(example from Micusik B. and Hanbury A )

38. Automatic graph cut [Shi & Malik]j
w(i,j) w i
Fully-connected graph
node for every pixel
link between every pair of pixels, i,j
cost w(i,j) for each link
w(i,j) measures similarity
similarity is inversely proportional to difference in color and position

39. Automatic graph cut [Shi & Malik]
Construct a weighted graph from the image pixels
G = (V,E) by taking each pixel and connecting each pair of pixels in an edge
F(i) – gray level value in pixel i
X(i) – pixel i position

40. Segmentation by Graph Cutsw A B C
Break Graph into Segments
Delete links that cross between segments
Easiest to break links that have low cost (similarity)
similar pixels should be in the same segments
dissimilar pixels should be in different segments

41. Minimum cut Criterion for partition:
What is the problem with the min cut?
Problem!
Weight of cut is directly proportional to the number of edges in the cut.
Ideal Cut
Cuts with
lesser weight
than the
ideal cut
First proposed by Wu and Leahy

42. Normalized Cut Cut in a graph penalizes large segments
Fix by normalizing in size of segments
vol (A) = sum of costs of all edges that touch A
Finds better cut

43. Formal definitions Association of two sets: Volume of a set:
Normalized cut:
A fraction of the total edge connections to all the nodes in the graph

44. How to minimize Ncut? Transform Ncut equation to a matrix form
Solve an eigenvector problem for this matrix

45. Graph-based Image Segmentation
Image (I)
Intensity
Color
Edges
Texture
Graph Affinities
(W)
Slide from Timothee Cour (

46. Graph-based Image Segmentation
Image (I)
Intensity
Color
Edges
Texture
Graph Affinities
(W)
Slide from Timothee Cour (

47. Graph-based Image Segmentation
Image (I)
Eigenvector
X(W)
Intensity
Color
Edges
Texture
Graph Affinities
(W)
Slide from Timothee Cour (

48. Graph-based Image Segmentation
Image (I)
Eigenvector
X(W)
Discretization
Intensity
Color
Edges
Texture
Graph Affinities
(W)
Slide from Timothee Cour (

49. Graph-based Image Segmentation
G = {V,E}
V: graph nodes
E: edges connection nodes
Pixels
Pixel affinity
Slides from Jianbo Shi

50. Graph terminology
Affinity matrix:
Slides from Jianbo Shi

51. the size of W is quadratic
Affinity matrix
Similarity of image pixels to selected pixel
Brighter means more similar
N pixels
M pixels
Warning
the size of W is quadratic
with the number
of the image pixels!
Reshape
N*M pixels
N*M pixels

52. Graph terminology
Degree of node: … …
Slides from Jianbo Shi

53. Graph terminology
Volume of set:
Slides from Jianbo Shi

54. Graph terminology
Cuts in a graph:
Slides from Jianbo Shi

55. Normalized Cut A B A B Volume of set (or association):
Define normalized cut: “a fraction of the total edge connections to all the nodes in the graph”: A B A B

56. Optimization Problem
Given a set A of vertices , consider
so that 

57. y’s values are quantized
Optimization Problem
Replacing x by y:
Subject to:
1) (b is depended in W) 2)
NP-Hard!
y’s values are quantized

58. Optimization Problem - Relaxation
Relax the constranits on y:
Subject to
The solution is the (second) minimal eigenvector of D-W

59. Algorithm Define a similarity function between 2 nodes. i.e.:
Compute affinity matrix (W) and degree matrix (D).
Solve
Use the eigenvector with the second smallest eigenvalue to bipartition the graph.
Decide if re-partition current partitions.

60. Example (1)
Eigenvectors
Segments

61. Example (2) Original Segments
* Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003

62. Real segmentation example

63. Ideas ? Steps in solving specific problem
Make the assumptions appropriate for this case
Size of the warm is relatively constant
Warm does not move much between successive frames
Lighting is changing between frames and within the frame, but can be considered as constans on small window
Warm is not the only dark area in the picture
Use your assumption to “tailor” a specific solution

64. Real segmentation example

65. Real segmentation example

66. Examples Using the same thresh for 2 different frames
Using adaptive thresh based on gradient values

67. How to remove ‘wrong areas’ ?
Put additional condition on the selected area – to be next to large gradient pixel
Remove noise by using morphological operations

68. Possible Solution
Find the warm location in the first frame
Use gradient value to determine threshold
Remove from the selection pixels that are selected according to the threshold but do not have large gradient in their area
Use morphological operation to remove noise
Using the warm location in the previous frame, search in the current frame using a simple threshold on window around the warm
Update window location based on the new position of warm
And still – in practice can know if this works only after trying ….

69. Thanks
and
Good Luck!