Recognizing Surfaces using Three-Dimensional Textons Thomas Leung and Jitendra Malik Computer Science Division University of California at Berkeley

ICCV '99, Corfu, Greece Traditional Texture Recognition Assume texture to be planar; Assume constant illumination and viewing directions; Ignore 3D nature of natural materials, i.e. no shadowing, occlusions, etc… E.g. Puzicha et al, Jain et al, Greenspan et al, etc….

ICCV '99, Corfu, Greece Example Natural Materials TerryclothRough PlasticPlaster-b SpongeRug-aPainted Spheres Columbia-Utrecht Database (

ICCV '99, Corfu, Greece Materials under different illumination and viewing directions Different illumination and viewing directions Plaster-aCrumpled Paper ConcretePlaster-b (zoomed)

ICCV '99, Corfu, Greece Task Felt? Polyester? Terrycloth? Rough Plaster? Leather? Plaster? Concrete? Crumpled Paper? Sponge? Limestone? Brick? ? ?

ICCV '99, Corfu, Greece 3D Texture Models Analytical models: –Simple parametric surface height distribution; –compute image statistics; –Dana & Nayar 97, 98, 99; Koenderink et al 96, 98; Leung & Malik 97; Chantler et al 97, 98; Computer graphics models: –bump maps, displacement maps, point clouds, etc. –difficult to obtain for natural materials;

ICCV '99, Corfu, Greece Problem Formulation Image Database Recognize new sample of different light/view Task

ICCV '99, Corfu, Greece Main Idea Natural materials are made up of local features (geometric and photometric); There exists a universal set of local features for all materials; How these local features change appearance with different illumination and viewing directions determine how the materials look.

ICCV '99, Corfu, Greece Outline Learning the universal vocabulary of local structures Material models Results

ICCV '99, Corfu, Greece Outline Learning the universal vocabulary of local structures –Introduce 2D textons for planar texture; –Extend to 3D textons for natural materials; Material models Results

ICCV '99, Corfu, Greece 2D Textons Julesz suggests a universal vocabulary for such features --- textons [Julesz 81]; crossings, line-ends, junctions, etc… Define textons for real images.

ICCV '99, Corfu, Greece 2D Textons Goal: find canonical local features in a texture; 1) Filter image with linear filters: 2) Vector quantization on filter outputs; 3) Quantization centers are the textons. Spatial distribution of textons defines the texture;

ICCV '99, Corfu, Greece 2D Textons (cont’d)

ICCV '99, Corfu, Greece 3D Textons Consider textures with 3D features, e.g. bumps, grooves, ridges, etc… Want textons to capture local 3D geometric and photometric features; One image is ambiguous: different features can look the same under certain illumination and viewing conditions; More images will discriminate between the different cases.

ICCV '99, Corfu, Greece Learning 3D Textons Rough Plastic Concrete Light/view 1 Light/view 2 Light/view N 3D textons Texton 1 Texton K Texton 2

ICCV '99, Corfu, Greece Algorithm for Learning Vocabulary Register all 20 images for each material; Filter images with filter bank of 48 kernels; Concatenate filter responses of the 20 images; Each pixel becomes a 960 (20x48) dimensional feature vector; Apply K-means to the feature vectors of all materials together; Resulting centers are the 3D textons.

ICCV '99, Corfu, Greece Algorithm for 3D Textons

ICCV '99, Corfu, Greece Universal 3D Texton Vocabulary Columbia-Utrecht Database (60 materials, each with 205 images) Vocabulary of textons learned from 20 training materials; Use 20 different light/view images for each material.

ICCV '99, Corfu, Greece Examples of 3D Textons Texton 1 Texton 2 Texton 3 Texton 4 Texton 5 Texton 6 Texton 7 Different illumination and viewing directions

ICCV '99, Corfu, Greece Quantization Errors Reconstruct images after quantization; SSD error within 5%.

ICCV '99, Corfu, Greece Outline Learning the universal vocabulary of local structures; Material models; –Image to texton representation; –Material representation using textons; Results.

ICCV '99, Corfu, Greece Texton Labeling Each pixel labeled to texton i (1 to K) which is most similar in appearance; Similarity measured by the Euclidean distance between the filter responses;

ICCV '99, Corfu, Greece Material Representation Each material is now represented as a spatial arrangement of symbols from the texton vocabulary; Recognition ---ignore spatial arrangement, use histogram (K=100);

ICCV '99, Corfu, Greece Histogram Models for Recognition Terrycloth Rough Plastic Pebbles Plaster-b

ICCV '99, Corfu, Greece Similarity of materials Similarity between histograms measured using chi-square difference:

ICCV '99, Corfu, Greece Similarity Matrix Plaster-aPlaster-b Aluminum Foil Cork

ICCV '99, Corfu, Greece Outline Learning the universal vocabulary of local structures Material models Results –Material recognition from single image; –Synthesis of novel images.

ICCV '99, Corfu, Greece Recognition from Single Image 4 images to build histogram for model; 1 image of novel illumination and/or viewing directions to be recognized; Image Database ? Novel image

ICCV '99, Corfu, Greece Novel Image from Material i? Build texton histogram for novel image. Compare with texton histogram for material i. However, texton labeling from 1 image is difficult, because in 1 light/view, several textons may have same appearance. Each pixel has N possible texton labels; Need to find the labeling that maximizes Similarity(novel image, material i)

ICCV '99, Corfu, Greece Markov chain Monte Carlo for finding labeling Randomly label each pixel to one of N possibilities. Call this the initial state x(t),t=0 Compute P(x(t)|material i); Obtain x’ by randomly changing M labels of x(t); Compute P(x’|material i); Compute If, the x’ is accepted, otherwise, accept with probability.

ICCV '99, Corfu, Greece P(detection) vs P(false alarm)

ICCV '99, Corfu, Greece Synthesis of images with novel illumination and viewing directions Map each pixel to textons Textons tell us how appearance changes

ICCV '99, Corfu, Greece Synthesis of novel light/view images Keep exact spatial arrangement of textons

ICCV '99, Corfu, Greece Synthesis Results Texture Mapping Ground Truth 3D Texton Model Texture Mapping Ground Truth 3D Texton Model Plaster-aConcrete

ICCV '99, Corfu, Greece Synthesis Results Texture Mapping Ground Truth 3D Texton Model Texture Mapping Ground Truth 3D Texton Model Crumpled paperPlaster-b (zoomed)

ICCV '99, Corfu, Greece Synthesis Results Texture Mapping Ground Truth 3D Texton Model Texture Mapping Ground Truth 3D Texton Model Rough PlasticSponge

ICCV '99, Corfu, Greece Similarity to Appearance-based Object Recognition Object Recognition: objects are represented by a collection of images under different illumination and viewing conditions; Material Recognition: materials are represented by 3D textons, each of which is represented by the appearances under different illumination and viewing conditions.

ICCV '99, Corfu, Greece Conclusions Model natural materials through images; Learn a universal vocabulary of 3D textons; Use the vocabulary to –recognize materials from a single image of novel illumination and viewing directions; –synthesize materials at novel illumination and viewing directions.