We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byLeon Carson
Modified about 1 year ago
Image Segmentation Selim Aksoy Department of Computer Engineering Bilkent University
Examples of grouping in vision CS 484, Spring 2012©2012, Selim Aksoy2 [Figure by J. Shi] [http://poseidon.csd.auth.gr/LAB_RESEARCH/Latest/imgs/S peakDepVidIndex_img2.jpg] [Figure by Wang & Suter] [Figure by Grauman & Darrell] Determine image regions Object-level grouping Figure-ground separation Group video frames into shots
CS 484, Spring 2012©2012, Selim Aksoy3 Image segmentation
CS 484, Spring 2012©2012, Selim Aksoy4 Image segmentation
CS 484, Spring 2012©2012, Selim Aksoy5 From images to objects What defines an object? Subjective problem, but has been well-studied. Gestalt laws seek to formalize this. “What is interesting and what is not” depends on the application. Broad theory is absent at present.
CS 484, Spring 2012©2012, Selim Aksoy6 Gestalt laws A series of factors affect whether elements should be grouped together. Proximity: tokens that are nearby tend to be grouped. Similarity: similar tokens tend to be grouped together. Common fate: tokens that have coherent motion tend to be grouped together. Common region: tokens that lie inside the same closed region tend to be grouped together. Parallelism: parallel curves or tokens tend to be grouped together. Closure: tokens or curves that tend to lead to closed curves tend to be grouped together. Symmetry: curves that lead to symmetric groups are grouped together. Continuity: tokens that lead to “continuous” curves tend to be grouped. Familiar configuration: tokens that, when grouped, lead to a familiar object, tend to be grouped together.
CS 484, Spring 2012©2012, Selim Aksoy7 Gestalt laws
CS 484, Spring 2012©2012, Selim Aksoy8 Adapted from Kristen Grauman Similarity Symmetry
Gestalt laws CS 484, Spring 2012©2012, Selim Aksoy9 Adapted from Kristen Grauman Common fate Proximity
Gestalt laws CS 484, Spring 2012©2012, Selim Aksoy10 Continuity, explanation by occlusion
CS 484, Spring 2012©2012, Selim Aksoy11 Image segmentation Image segmentation is the operation of partitioning an image into a collection of connected sets of pixels. Segmentation criteria: a segmentation is a partition of an image I into a set of regions S satisfying: 1. S i = SPartition covers the whole image. 2. S i S j = , i jNo regions intersect. 3. S i, P(S i ) = trueHomogeneity predicate is satisfied by each region. 4. P(S i S j ) = false,Union of adjacent regions i j, S i adjacent S j does not satisfy it.
CS 484, Spring 2012©2012, Selim Aksoy12 Image segmentation So, all we have to do is to define and implement the similarity predicate. But, what do we want to be similar in each region? Is there any property that will cause the regions to be meaningful objects? Example approaches: Histogram-based Clustering-based Region growing Split-and-merge Morphological Graph-based
CS 484, Spring 2012©2012, Selim Aksoy13 Histogram-based segmentation How many “orange” pixels are in this image? This type of question can be answered by looking at the histogram.
CS 484, Spring 2012©2012, Selim Aksoy14 Histogram-based segmentation How many modes are there? Solve this by reducing the number of colors to K and mapping each pixel to the closest color. Here’s what it looks like if we use two colors.
CS 484, Spring 2012©2012, Selim Aksoy15 Clustering-based segmentation How to choose the representative colors? This is a clustering problem! K-means algorithm can be used for clustering.
CS 484, Spring 2012©2012, Selim Aksoy16 Clustering-based segmentation K-means clustering of color.
CS 484, Spring 2012©2012, Selim Aksoy17 Clustering-based segmentation K-means clustering of color.
CS 484, Spring 2012©2012, Selim Aksoy18 Clustering-based segmentation Clustering can also be used with other features (e.g., texture) in addition to color. Original ImagesColor Regions Texture Regions
Clustering-based segmentation Pros: Simple, fast to compute Converges to local minimum of within- cluster squared error Cons: Setting K? Sensitive to initial centers Sensitive to outliers Detects spherical clusters CS 484, Spring 2012©2012, Selim Aksoy19 Adapted from Kristen Grauman
CS 484, Spring 2012©2012, Selim Aksoy20 Clustering-based segmentation K-means variants: Different ways to initialize the means. Different stopping criteria. Dynamic methods for determining the right number of clusters (K) for a given image. Problem: histogram-based and clustering-based segmentation using color/texture/etc can produce messy/noisy regions. (Why?) How can these be fixed?
Mean shift algorithm The mean shift algorithm seeks modes or local maxima of density in the feature space. CS 484, Spring 2012©2012, Selim Aksoy21 Image Feature space (L*u*v* color values) Adapted from Kristen Grauman
Mean shift algorithm CS 484, Spring 2012©2012, Selim Aksoy22 Search window Center of mass Mean Shift vector
Mean shift algorithm CS 484, Spring 2012©2012, Selim Aksoy23 Search window Center of mass Mean Shift vector
Mean shift algorithm CS 484, Spring 2012©2012, Selim Aksoy24 Search window Center of mass Mean Shift vector
Mean shift algorithm CS 484, Spring 2012©2012, Selim Aksoy25 Search window Center of mass Mean Shift vector
Mean shift algorithm CS 484, Spring 2012©2012, Selim Aksoy26 Search window Center of mass Mean Shift vector
Mean shift algorithm CS 484, Spring 2012©2012, Selim Aksoy27 Search window Center of mass Mean Shift vector
Mean shift algorithm CS 484, Spring 2012©2012, Selim Aksoy28 Center of mass Search window
Mean shift clustering/segmentation Find features (color, gradients, texture, etc) Initialize windows at individual feature points Perform mean shift for each window until convergence Merge windows that end up near the same “peak” or mode CS 484, Spring 2012©2012, Selim Aksoy29
Mean shift segmentation CS 484, Spring 2012©2012, Selim Aksoy30
Mean shift segmentation CS 484, Spring 2012©2012, Selim Aksoy31
Mean shift segmentation Pros: Does not assume shape on clusters One parameter choice (window size) Generic technique Find multiple modes Cons: Selection of window size Does not scale well with dimension of feature space CS 484, Spring 2012©2012, Selim Aksoy32 Adapted from Kristen Grauman
CS 484, Spring 2012©2012, Selim Aksoy33 Region growing Region growing techniques start with one pixel of a potential region and try to grow it by adding adjacent pixels till the pixels being compared are too dissimilar. The first pixel selected can be just the first unlabeled pixel in the image or a set of seed pixels can be chosen (manually or automatically) from the image. We need to define a measure of similarity between a pixel and a set of pixels as well as a rule that makes a decision for growing based on this measure.
CS 484, Spring 2012©2012, Selim Aksoy34 Region growing Usually a statistical test is used to decide which pixels can be added to a region. Region is a population with similar statistics. Use statistical test to see if neighbor on border fits into the region population. Let R be the N pixel region so far and p be a neighboring pixel with gray tone y. Define the mean X and scatter S 2 (sample variance) by
CS 484, Spring 2012©2012, Selim Aksoy35 Region growing The T statistic is defined by It has a T N-1 distribution if all the pixels in R and the test pixel p are independent and identically distributed Gaussians (i.i.d. assumption).
CS 484, Spring 2012©2012, Selim Aksoy36 Region growing For the T distribution, statistical tables give us the probability Pr(T ≤ t) for a given degrees of freedom and a confidence level. From this, pick a suitable threshold t. If the computed T ≤ t for desired confidence level, add p to region R and update the mean and scatter using p. If T is too high, the value p is not likely to have arisen from the population of pixels in R. Start a new region. Many other statistical and distance-based methods have also been proposed for region growing.
CS 484, Spring 2012©2012, Selim Aksoy37 Region growing image segmentation
CS 484, Spring 2012©2012, Selim Aksoy38 Split-and-merge 1. Start with the whole image. 2. If the variance is too high, break into quadrants. 3. Merge any adjacent regions that are similar enough. 4. Repeat steps 2 and 3, iteratively until no more splitting or merging occur. Idea: good Results: blocky
CS 484, Spring 2012©2012, Selim Aksoy39 Split-and-merge
CS 484, Spring 2012©2012, Selim Aksoy40 Split-and-merge
CS 484, Spring 2012©2012, Selim Aksoy41 Watershed segmentation The image can be interpreted as a topographic surface, with both valleys and mountains. Three types of points can be considered: Points belonging to a regional minimum. Points at which a drop of water, if placed at the location of any of those points, would fall to a single minimum. catchment basins Points at which water would be equally likely to fall to more than one such minimum. watershed lines
CS 484, Spring 2012©2012, Selim Aksoy42 Watershed segmentation Assume that there is a hole in each minima and the surface is immersed into a lake. The water will enter through the holes at the minima and flood the surface. To avoid the water coming from two different minima to meet, a dam is build whenever there would be a merge of the water. Finally, the only thing visible of the surface would be the dams. These dam walls are called the watershed lines.
CS 484, Spring 2012©2012, Selim Aksoy43 Watershed segmentation
CS 484, Spring 2012©2012, Selim Aksoy44 Watershed segmentation
CS 484, Spring 2012©2012, Selim Aksoy45 A multi-scale segmentation can be obtained by iteratively smoothing the topographic surface.
CS 484, Spring 2012©2012, Selim Aksoy46 Watershed segmentation The key behind using the watershed transform for segmentation is this: change your image into another image whose catchment basins are the objects you want to identify. Examples: Distance transform can be used with binary images where the catchment basins correspond to the foreground components of interest. Gradient can be used with grayscale images where the catchment basins should theoretically correspond to the homogeneous grey level regions of the image.
CS 484, Spring 2012©2012, Selim Aksoy47 Watershed segmentation Binary image.Distance transform of the complement of the binary image. Watershed transform after complementing the distance transform, and forcing pixels that do not belong to the objects to be at –Inf.
Watershed segmentation CS 484, Spring 2012©2012, Selim Aksoy48 A cell image.Gradient of the cell image.
Watershed segmentation CS 484, Spring 2012©2012, Selim Aksoy49 Multi-scale watershed segmentation of the cell image.
CS 484, Spring 2012©2012, Selim Aksoy50 Graph-based segmentation An image is represented by a graph whose nodes are pixels or small groups of pixels. The goal is to partition the nodes into disjoint sets so that the similarity within each set is high and across different sets is low.
CS 484, Spring 2012©2012, Selim Aksoy51 Graph-based segmentation Let G = (V,E) be a graph. Each edge (u,v) has a weight w(u,v) that represents the similarity between u and v. Graph G can be broken into 2 disjoint graphs with node sets A and B by removing edges that connect these sets. Let cut(A,B) = w(u,v). One way to segment G is to find the minimum cut. u A, v B
Graph-based segmentation Problem with minimum cut: weight of cut proportional to number of edges in the cut; tends to produce small, isolated components. CS 484, Spring 2012©2012, Selim Aksoy52
CS 484, Spring 2012©2012, Selim Aksoy53 Graph-based segmentation Shi and Malik proposed the normalized cut. cut(A,B) cut(A,B) Ncut(A,B) = assoc(A,V) assoc(B,V) assoc(A,V) = w(u,t) u A, t V How much is A connected to the graph as a whole Normalized cut
CS 484, Spring 2012©2012, Selim Aksoy54 Graph-based segmentation Ncut(A,B) = A B
CS 484, Spring 2012©2012, Selim Aksoy55 Graph-based segmentation Shi and Malik turned graph cuts into an eigenvector/eigenvalue problem. Set up a weighted graph G=(V,E). V is the set of (N) pixels. E is a set of weighted edges (weight w ij gives the similarity between nodes i and j). Length N vector d: d i is the sum of the weights from node i to all other nodes. N x N matrix D: D is a diagonal matrix with d on its diagonal. N x N symmetric matrix W: W ij = w ij.
CS 484, Spring 2012©2012, Selim Aksoy56 Graph-based segmentation Let x be a characteristic vector of a set A of nodes. x i = 1 if node i is in a set A x i = -1 otherwise Let y be a continuous approximation to x Solve the system of equations (D – W) y = D y for the eigenvectors y and eigenvalues. Use the eigenvector y with second smallest eigenvalue to bipartition the graph (y x A). If further subdivision is merited, repeat recursively.
CS 484, Spring 2012©2012, Selim Aksoy57 Graph-based segmentation Edge weights w(i,j) can be defined by w(i,j) = e * where X(i) is the spatial location of node I F(i) is the feature vector for node I which can be intensity, color, texture, motion… The formula is set up so that w(i,j) is 0 for nodes that are too far apart. -||F(i)-F(j)|| 2 / I 2 e if ||X(i)-X(j)|| 2 < r 0 otherwise -||X(i)-X(j)|| 2 / X 2
CS 484, Spring 2012©2012, Selim Aksoy58 Graph-based segmentation
Pros: Generic framework, flexible to choice of function that computes weights (“affinities”) between nodes Does not require model of the data distribution Cons: Time complexity can be high Dense, highly connected graphs many affinity computations Solving eigenvalue problem CS 484, Spring 2012©2012, Selim Aksoy59 Adapted from Kristen Grauman
Image Segmentation Selim Aksoy Department of Computer Engineering Bilkent University
Segmentácia farebného obrazu. Image segmentation.
Image Segmentation Image segmentation is the operation of partitioning an image into a collection of connected sets of pixels. 1. into regions, which usually.
Region Segmentation Readings: Chapter 10: 10.1 Additional Materials Provided K-means Clustering (text) EM Clustering (paper) Graph Partitioning (text)
Segmentation CSE P 576 Larry Zitnick Many slides courtesy of Steve Seitz.
Image segmentation. The goals of segmentation Group together similar-looking pixels for efficiency of further processing “Bottom-up” process Unsupervised.
Lecture 30: Segmentation CS4670 / 5670: Computer Vision Noah Snavely From Sandlot ScienceSandlot Science.
Computer Vision - A Modern Approach Set: Segmentation Slides by D.A. Forsyth Segmentation and Grouping Motivation: not information is evidence Obtain a.
EE 7730 Image Segmentation. Bahadir K. Gunturk EE Image Analysis I 2 Image Segmentation Group similar components (such as, pixels in an image,
Announcements Project 2 more signup slots questions Picture taking at end of class.
Thresholding Otsu’s Thresholding Method Threshold Detection Methods Optimal Thresholding Multi-Spectral Thresholding 6.2. Edge-based.
Segmentation Divide the image into segments. Each segment: –Looks uniform –Belongs to a single object. –Have some uniform attributes. –All the pixel related.
Lecture 6 Image Segmentation Slides by: David A. Forsyth Clark F. Olson Steven M. Seitz Linda G. Shapiro.
Announcements Project 3 questions Photos after class.
1 Image Segmentation Image segmentation is the operation of partitioning an image into a collection of connected sets of pixels. 1. into regions, which.
Motion Segmentation at Any Speed Shrinivas J. Pundlik Department of Electrical and Computer Engineering, Clemson University, Clemson, SC.
Region Segmentation. Find sets of pixels, such that All pixels in region i satisfy some constraint of similarity.
CSE 185 Introduction to Computer Vision Pattern Recognition 2.
Chapter 14: SEGMENTATION BY CLUSTERING 1. 2 Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary.
CS 376b Introduction to Computer Vision 04 / 04 / 2008 Instructor: Michael Eckmann.
Segmentation. Terminology Segmentation, grouping, perceptual organization: gathering features that belong together Fitting: associating a model with observed.
EECS 274 Computer Vision Segmentation by Clustering II.
Announcements vote for Project 3 artifacts Project 4 (due next Wed night) Questions? Late day policy: everything must be turned in by next Friday.
Digital Image Processing, 2nd ed. © 2002 R. C. Gonzalez & R. E. Woods Chapter 10 Image Segmentation Chapter 10 Image Segmentation.
Segmentation & Grouping Tuesday, Sept 23 Kristen Grauman UT-Austin.
CS 376b Introduction to Computer Vision 04 / 08 / 2008 Instructor: Michael Eckmann.
CS654: Digital Image Analysis Lecture 26: Image segmentation.
Machine Learning Computer Vision James Hays, Brown Slides: Isabelle Guyon, Erik Sudderth, Mark Johnson, Derek Hoiem Photo: CMU Machine Learning Department.
CS654: Digital Image Analysis Lecture 30: Clustering based Segmentation Slides are adapted from:
CS 2750: Machine Learning Clustering Prof. Adriana Kovashka University of Pittsburgh January 25, 2016.
Segmentation and Grouping Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 02/23/10.
Algorithms & Applications in Computer Vision Lihi Zelnik-Manor Lecture 11: Structure from Motion.
Chapter 7 – Segmentation II 7.1. Mean Shift Segmentation 7.2. Active Contour Models – Snakes 7.3. Geometric Deformable Models – Level Sets and Geodesic.
Image Segmentation Shengnan Wang
Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...
Clustering & image segmentation Goal::Identify groups of pixels that go together Segmentation.
Image Segmentation Today’s Readings Forsyth & Ponce, Chapter 14 (plus lots of optional references in the slides) From Sandlot ScienceSandlot Science.
Computer Vision Lecture 5. Clustering: Why and How.
Brent M. Dingle, Ph.D Game Design and Development Program Mathematics, Statistics and Computer Science University of Wisconsin - Stout Segmentation.
Normalized Cuts and Image Segmentation Jianbo Shi and Jitendra Malik, Presented by: Alireza Tavakkoli.
Image Segmentation Image segmentation (segmentace obrazu) –division or separation of the image into segments (connected regions) of similar properties.
Chapter 5 Segmentation Presenter: 蘇唯誠
Segmentation Course web page: vision.cis.udel.edu/~cv May 7, 2003 Lecture 31.
Color Image Segmentation Mentor : Dr. Rajeev Srivastava Students: Achit Kumar Ojha Aseem Kumar Akshay Tyagi.
Machine Vision ENT 273 Regions and Segmentation in Images Hema C.R. Lecture 4.
Perceptual Organization: Segmentation and Optical Flow.
© 2017 SlidePlayer.com Inc. All rights reserved.