National Center for Supercomputing Applications University of Illinois at Urbana-Champaign Image Feature Descriptors Kenton McHenry, Ph.D. Research Scientist

Raster Images [Hoiem, 2012] image(234, 452) = 0.58

Neighborhoods of Pixels For nearby surface points most factors do not change much Local differences in brightness [Hoiem, 2012]

Features Interest points Points or collections of points that are somehow relevant to identifying the contents of an image Subset of the image Far less data than the total number of pixels Repeatability A feature detector should be likely to detect the feature in a scene under a variety of lighting, orientation, and other variable conditions. Descriptors A relatively short numerical representation that can be used to identify a specific feature instance

Descriptors, Why? Matching: Match an object in two images based on similar features 3D Reconstruction: Stereopsis Tracking: Follow an object in a video by following its features Object Recognition: Find objects based on known features it will posses Segmentation: Break an image up into more meaningful regions based on the seen features.

Feature Descriptors

Feature Descriptors for Edges Lines: (m, b) (r,  ) Circles: r 2 = (x 2 - a 2 ) + (y 2 - b 2 ) (a, b, r) Other shapes Follow lines to make curves

Feature Descriptors for Curves

Curve length Curvature Curves smoothness

Feature Descriptors for Curves

Curve length Curvature Curves smoothness Corners Concave polygons Convex polygons

Feature Descriptors for Curves Curve length Curvature Curves smoothness Corners Concave polygons Convex polygons Periodicity Frequency …

Feature Descriptors for Regions

Matching Regions Two views of the same object Find matching regions

Matching Regions

Template Matching If regions are the same size we can again treat them as vectors and compare them Sum of Squared Differences (SSD) d = (a 1 -b 1 ) 2 + (a 2 -b 2 ) 2 + … Euclidean Distance without the square root Slightly faster and closer things still have smaller values Normalized Cross Correlation d = a/|a|. b/|b| More invariant to changes in intensity Values between -1 and 1

Template Matching: SSD >r0 = mean(imread(‘scar1_region2.png’), 3); >imagesc(r0); >r1 = mean(imread(‘scar2_region1.png’), 3); >r2 = mean(imread(‘scar2_region2.png’), 3); >r3 = mean(imread(‘scar2_region3.png’), 3); >sum((r1(:)-r0(:)).^2) >sum((r2(:)-r0(:)).^2) >sum((r3(:)-r0(:)).^2)

Template Matching: NCC >r0 = r0(:) / norm(r0(:)); >r1 = r1(:) / norm(r1(:)); >r2 = r2(:) / norm(r2(:)); >r3 = r3(:) / norm(r3(:)); >dot(r0,r1) >dot(r0,r2) >dot(r0,r3)

Color Average Color Treat the entire regions as one color Coarse representation Regions don’t have to be the same size

Color >r0 = imread(‘scar1_region2.png’); >image(r0); >whos >mean(mean(r0, 1), 2) >c0 = reshape(mean(r0, 1), 2), 1, 3); >image(uint8(reshape(c0, 1, 1, 3)));

Color >r1 = imread(‘scar2_region1.png’); >image(r1); >c1 = reshape(mean(r1, 1), 2), 1, 3); >image(uint8(reshape(c1, 1, 1, 3))); >r2 = imread(‘scar2_region2.png’); >image(r2); >c2 = reshape(mean(r2, 1), 2), 1, 3); >image(uint8(mean(r2, 1), 2))); >r3 = imread(‘scar2_region3.png’); >image(r3); >c3 = reshape(mean(r3, 1), 2), 1, 3); >image(uint8(mean(r3, 1), 2)));

Color >sqrt(sum((c1-c0).^2)) >sqrt(sum((c2-c0).^2)) >sqrt(sum((c3-c0).^2))

Color Average Color Treats the entire regions as one color Coarse representation Regions don’t have to be the same size Color Distributions Build a distribution of colors in a region A bit more detail Regions still don’t have to be the same size

Color >r0 = imread(‘scar1_region2.png’); >r = r0(:,:,1); >g = r0(:,:,2); >b = r0(:,:,3); >hr = hist(double(r(:)), 10); >hg = hist(double(g(:)), 10); >hb = hist(double(b(:)), 10); >h0 = [hr hg hb]; >h0 = h0 / norm(h0);

Color >r1 = imread(‘scar2_region1.png’); >… >r2 = imread(‘scar2_region2.png’); >… >r3 = imread(‘scar2_region3.png’); >… >sqrt(sum((h1-h0).^2)) >sqrt(sum((h2-h0).^2)) >sqrt(sum((h3-h0).^2))

Color Spaces L  Human studies to model color similarity Euclidean distances between values better matches our perception HSV Hue: The colors Saturation: The purity of the color Value: The brightness, a black and white version of the image Matlab: rgb2hsv, hsv2rgb

Texture

Break the universe of textures into individual elements of texture, textons Use these textons as patterns to look for in the image and use convolution to find them Gaussian Filters In particular variations of Gaussian filters

Texture: Orientations >filter = fspecial(‘guassian’, 50, 5); >filter = diff(filter, 1, 2); >filter = filter / norm(filter); >imagesc(filter); >filter = imrotate(filter, 45); >imagesc(filter);

Texture: Elongations >f = fspecial(‘guassian’, 50, 5); >f = diff(f, 1, 2); >f = imresize(f, [2*size(f,1) size(f,2)]); >f = f / norm(f); >imagesc(f);

Texture: Miscellaneous >filter = fspecial(‘guassian’, 50, 5); >filter = diff(filter, 1, 2); >filter = filter / norm(filter); >imagesc(filter); >filter = imrotate(filter, 45); >imagesc(filter); >filter = fspeical(‘log’, 50, 5); Bars Spots

Texture: Scales >filter = … >filter = imresize(filter, 0.5); >filter = filter / norm(filter); >imagesc(filter);

Texture Filter bank Create many filters spanning: Orientations Elongations Types Scales

Texture >Ig = mean(imread(‘scar1.jpg’), 3); >Ig = Ig / norm(Ig); >f = fspecial(‘gaussian’, 16, 2); >f1 = diff(f, 1, 2); >f1 = f1 / norm(f1); >f2 = imrotate(f1, 45, ‘bilinear’, ‘crop’); >f2 = f2 / norm(f2); >f3 = imrotate(f2, 45, ‘bilinear’, ‘crop’); >f3 = f3 / norm(f3); >Ir1 = abs(conv2(Ig, f1)); >Ir2 = abs(conv2(Ig, f1)); >Ir3 = abs(conv2(Ig, f1));

Texture >Ig = mean(imread(‘textures.png’), 3); >Ig = Ig / norm(Ig); >f = fspecial(‘gaussian’, 16, 2); >f1 = diff(f, 1, 2); >f1 = f1 / norm(f1); >f2 = imrotate(f1, 45, ‘bilinear’, ‘crop’); >f2 = f2 / norm(f2); >f3 = imrotate(f2, 45, ‘bilinear’, ‘crop’); >f3 = f3 / norm(f3); >Ir1 = abs(conv2(Ig, f1)); >Ir2 = abs(conv2(Ig, f1)); >Ir3 = abs(conv2(Ig, f1));

Texture >r1 = Ir1(1:100,1:100); >r2 = Ir2(1:100,1:100); >r3 = Ir3(1:100,1:100); >h1 = hist(r1(:), 10); >h2 = hist(r2(:), 10); >h3 = hist(r3(:), 10); >ht1 = [h1 h2 h3]; >ht1 = ht1 / norm(ht1); >r1 = Ir1(1:100,100:200); >… >r1 = Ir1(1:100,200:300); >…

Texture >sqrt(sum((ht1-ht2).^2)) >sqrt(sum((ht1-ht3).^2)) >sqrt(sum((ht2-ht3).^2))

Miscellaneous Size Count the number of pixels Statistics Calculate mean and standard deviation in x and y directions Neighbors …

Image and Spatial Data Analysis Group Questions?