Digital Camera and Computer Vision Laboratory Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R.O.C. Computer Vision Chapter 9 Texture Presented by 盧毅 and 傅楸善教授 Cell phone:

DC & CV Lab. CSIE NTU Copyright Mentioned Some slides adapted from 455/09wi/Lects/lect12.pdf 455/09wi/Lects/lect12.pdf Some slides about LBP and variants adapted from XianBiao Qi

DC & CV Lab. CSIE NTU Introduction What does texture mean? Formal approach or precise definition of texture does not exist! Texture discrimination techniques are for the part ad hoc.

DC & CV Lab. CSIE NTU What is Texture?

DC & CV Lab. CSIE NTU “Definition” of Texture Non-local property, characteristic of region larger than its size Repeating patterns of local variations in image intensity which are too fine to be distinguished as separated objects at the observed resolution

DC & CV Lab. CSIE NTU “Definition” of Texture (cont.) For humans, texture is the abstraction of certain statistical homogeneities from a portion of the visual field that contains a quantity of information grossly in excess of the observer’s perceptual capacity

DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

Texture Classification Samples of Brodatz, CUReT, and KTH-TIPS

DC & CV Lab. CSIE NTU Texture Analysis Issues Pattern recognition: given texture region, determine the class the region belongs to Generative model: given textured region, determine a description or model for it Texture segmentation: given image with many textured areas, determine boundaries

DC & CV Lab. CSIE NTU Statistical Texture-Feature Approaches Autocorrelation function Spectral power density function Edgeness per unit area Spatial gray level co-occurrence probabilities Graylevel run-length distributions Relative extrema distributions Mathematical morphology

DC & CV Lab. CSIE NTU Image Texture Analysis Give a generative model and the values of its parameters, one can synthesize homogeneous image texture samples associated with the model and the given value of its parameters.

DC & CV Lab. CSIE NTU Image Texture Analysis (cont.) Verification: verify given image textures sample consistent with model Estimation: estimate values of model parameters based on observed sample examples of model-based techniques

DC & CV Lab. CSIE NTU Some Model-Based Techniques Autoregressive, moving-average, time-series models (extended to 2D) Markov random fields Mosaic models

DC & CV Lab. CSIE NTU Nonfigurative and Cellular Texture It is decomposable It has two basic dimensions First dim: gray level primitives Second dim: spatial organization of the gray level primitives

DC & CV Lab. CSIE NTU Nonfigurative and Cellular Texture Gray level primitives: Regions with gray level properties Spatial organization: Random, Pairwise dependence, Dependence of n primitives at a time Dependence: structural, probabilistic, or functional

DC & CV Lab. CSIE NTU Texel Texture element The basic textural unit of some textural primitives in their defining spatial relationships A texture is a set of texture elements or texels occurring in some regular or repeated pattern

DC & CV Lab. CSIE NTU Texel

DC & CV Lab. CSIE NTU Characterizing Texture An image texture is described by the number and types of its primitives and their spatial organization or layout. Image texture can be qualitatively evaluated as some properties.

DC & CV Lab. CSIE NTU Some Texture Features Fineness Coarseness Contrast Directionality Roughness Regularity Smoothness Granulation

DC & CV Lab. CSIE NTU Some Texture Features (cont.) Randomness Lineation Mottled Irregular Hummocky

DC & CV Lab. CSIE NTU Characterizing Texture (cont.) Each of these qualities translates into some property of the gray level primitives and the spatial interaction between them. Open issue: no attempts to map semantic meaning into precise properties of gray level primitives and their spatial distribution.

DC & CV Lab. CSIE NTU Aspects of texture

DC & CV Lab. CSIE NTU Take a Break

DC & CV Lab. CSIE NTU Texture and Scale For any textural surface, there exists a scale at which, when the surface is examined, it appears smooth and textureless. (see from infinite distance) As resolution increases, the surfaces appears as a fine texture and then a coarse one, and for multiple-scale textural surface the cycle of smooth, fine, and coarse may repeat.

DC & CV Lab. CSIE NTU Texture and Scale (cont.) Thus, texture cannot be analyzed without frame of reference on scale or resolution. Texture is a scale-dependent phenomenon.

DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU First-Order Gray-Level Statistics Statistics of single pixels E.g. Histogram, mean, median, variance

DC & CV Lab. CSIE NTU Second-Order Gray-Level Statistics The combined statistics of gray levels of pairs of pixels in which each two pixels in a pair have a fixed relative position E.g. co-occurrence Gray level spatial dependence: characterize texture by co-occurrence

DC & CV Lab. CSIE NTU Co-Occurrence Matrix The gray level co-occurrence can be specified in a matrix of relative frequencies P ij with which two neighboring pixels separated by distance d occur on the image, one with gray level i and the other with gray level j Symmetric matrix Function of angle and distance between pixels

DC & CV Lab. CSIE NTU 2)

DC & CV Lab. CSIE NTU Co-Occurrence Matrix (cont.) Probability of horizontal, d pixels apart pixels P(i, j, d, 0°) = #{[(k, l), (m, n)] | k-m = 0, |l-n| = d, I(k, l) = i, I(m,n) = j} Probability of 45 °, d pixels apart pixels P(i, j, d, 45°) = #{[(k, l), (m, n)] | (k-m = d, l-n = -d) or (k-m = -d, l-n = d), I(k, l) = i, I(m,n) = j}

DC & CV Lab. CSIE NTU Co-Occurrence Matrix (cont.) Probability of 90 °, d pixels apart pixels P(i, j, d, 90°) = #{[(k, l), (m, n)] | |k-m| = d, l-n = 0, I(k, l) = i, I(m,n) = j} Probability of 135 °, d pixels apart pixels P(i, j, d, 135°) = #{[(k, l), (m, n)] | (k-m = d, l-n = d) or (k-m = -d, l-n = -d), I(k, l) = i, I(m,n) = j}

DC & CV Lab. CSIE NTU 0

DC & CV Lab. CSIE NTU Co-Occurrence Matrix (cont.) Matrix symmetric: P(i, j, d, a) = P(j, i, d, a)

DC & CV Lab. CSIE NTU Variants of co-occurrence matrix Gray level difference probability: The probability of small contrast d for a coarse texture will be much higher than for a fine texture.

DC & CV Lab. CSIE NTU Variants of co-occurrence matrix (cont.) Another gray level difference probability: Can be used to compute a variety of features, such as entropy and energy.

DC & CV Lab. CSIE NTU Take a Break

DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU Matrix with Highest Entropy When all entries in P ij are equal Image where no preferred gray-level pairs exist features calculated from the co- occurrence matrix

DC & CV Lab. CSIE NTU Generalized Gray Level Spatial Dependence Models for Texture Simple generalization: consider more than two pixels at a time

DC & CV Lab. CSIE NTU Generalized Co-Occurrence Primitives and the spatial relationships between primitives. Difference from gray level co-occurrence: Pixel vs. Primitives Strong texture measures take into account the co-occurrence between texture primitives.

DC & CV Lab. CSIE NTU Texture Primitive Connected set of pixels characterized by attribute set Simplest primitive: pixel with gray level attribute More complicated primitive: connected set of pixels homogeneous in level, characterized by size, elongation, orientation, and average gray level

DC & CV Lab. CSIE NTU Spatial Relationship We have a list of primitives, their center coordinate, and their attributes after the primitives have been constructed.

DC & CV Lab. CSIE NTU Spatial Relationship (cont.)

DC & CV Lab. CSIE NTU Take a Break

DC & CV Lab. CSIE NTU Texture feature State-of-the-art works: "Pairwise rotation invariant co-occurrence local binary pattern“ ECCV 2012 "Multi-scale Joint Encoding of Local Binary Patterns for Texture and Material Classification“ BMVC 2013

DC & CV Lab. CSIE NTU Local Binary Pattern Classical image feature Successfully used in many CV applications Firstly proposed in > TPAMI 2002 > TPAMI 2006

Local Binary Pattern LBP 8,1

Local Binary Pattern Formula:

Rotation Invariant LBP Pattern LBP describes image microstructures There are 256 LBP 8,1 patterns Rotationpermutation Divide LBP patterns into several subgroups

Rotation Invariant LBP Pattern LBP describes image microstructures There are 256 LBP 8,1 patterns Rotationpermutation LBP 8,1 pattern has 36 subgroups under rotation permutations

Uniform LBP Pattern At most two transitions from 0 1 Most patterns are uniform pattern in images Sometimes >90% Samples of non-uniform patterns Samples of uniform patterns

Uniform LBP Pattern At most two transitions from 0 1 Most patterns are uniform pattern in images Sometimes >90% Microstructures defined by uniformed patterns Bright spot (0) Flat area or dark spot (8) Edges of varying positive/negative curvature (1-7)

DC & CV Lab. CSIE NTU Take a Break

DC & CV Lab. CSIE NTU State-of-the-art works: "Pairwise rotation invariant co-occurrence local binary pattern“ ECCV 2012 "Multi-scale Joint Encoding of Local Binary Patterns for Texture and Material Classification“ BMVC 2013

Multi-Scale Joint encoding Local Binary Patterns (MSJ-LBP)

LBP and Multi-scale LBP The LBP depicts the local structures of the images such as plat regions, edge, contours and so on. Contrast to LBP, multi-scale LBP describes larger and stronger structures, as shown in Fig. 1. Meanwhile, in the real image, the LBP patterns in different scales has strong relationship as shown in the right of Fig.1. LBPLBP on multiple scales To illustrate the LBP and Multi-scale LBP. MS-LBP depicts stronger structures than LBP. 1 0 A A

LBP and Multi-scale LBP

Dense Sampling 1 0

To illustrate the rotation invariant property of the proposed MSJ-LBP. Rotation Invariant Property Fig. 2 illustrates the rotation invariant property of the proposed MSJ-LBP. As shown in Fig. 2, the left MSJ-LBP pattern is [( )ru, ( )u]. Similarly, we can obtain the same pattern from the right of Fig. 2.

Drawback of MS-LBP

Experiment

Experiment (cont.)

DC & CV Lab. CSIE NTU Take a Break

Pairwise Rotation Invariant Co- occurrence Local Binary Pattern

Co-LBP pattern space is large 2 16 Co-occurrence Local Binary Pattern A B CoLBP(AB) ：

Co-LBP pattern space is large 2 16 Co-LBP under rotation permutation. Co-occurrence Local Binary Pattern AB CoLBP(AB) ： A* B* CoLBP(A * B * ) ： o

Co-LBP pattern space is large 2 16 Co-LBP under rotation permutation. Pairwise rotation invariant CoLBP Co-occurrence Local Binary Pattern AB CoLBP pri (AB) ： CoLBP ri (A) ：

Co-LBP pattern space is large 2 16 Co-LBP under rotation permutation. Pairwise rotation invariant CoLBP Uniformed pairwise–rotation invariant COLBP 10*59 patterns Pairwise Rotation Invariant Co- occurrence Local Binary Pattern

The Relative Angle Information The relative angle information is informative and rotation invariant.

Rotation Invariant Pairwise Rotation Invariant encoding just promises that for the same A and B, the encoding strategy is rotation invariant. But we must keep that for same point A in different images, the point B could be found. The gradient orientation is a relative invariant between point A and B in different images.

The difference between the invariant properties of co-occurrence feature and traditional features Same pairs should be promised. Invariant encoding strategy should be used

Encoding Multi-Scale and Multi- Orientation Information AB 2 46 C D E F G CoLBP(A B) CoLBP(A C) CoLBP(A D) CoLBP(A E) CoLBP(A F) CoLBP(A G) Vector dimension : 590*6 MSMO-CoLBP: MSMO- CoLBP R MSMO- CoLBP G MSMO- CoLBP B Vector dimension : 590*6*3 Color MSMO-CoLBP Vector dimension : 150 PCA dimension Reduction across images Encoding color information

DC & CV Lab. CSIE NTU Autocorrelation Function Texture relates to the spatial size of the gray level primitives on an image Gray level primitives of larger size are indicative of coarser texture Gray level primitives of smaller size are indicative of finer texture

DC & CV Lab. CSIE NTU Autocorrelation Function (cont.) Autocorrelation function describes the size of gray level primitives

DC & CV Lab. CSIE NTU Autocorrelation Function (cont.)

DC & CV Lab. CSIE NTU Autocorrelation Function (cont.)

DC & CV Lab. CSIE NTU Autocorrelation Function (cont.)

DC & CV Lab. CSIE NTU Autocorrelation Function (cont.)

DC & CV Lab. CSIE NTU Autocorrelation Function (cont.) - 1D Example from Wiki

DC & CV Lab. CSIE NTU Autocorrelation Function (cont.) - Interpreting autocorrelation

DC & CV Lab. CSIE NTU Autocorrelation Function (cont.) If the gray level on image is relatively large: texture is coarse, autocorrelation drops off slowly with distance If the gray level on image is relatively small: texture is fine, autocorrelation drops off quickly with distance Periodic

DC & CV Lab. CSIE NTU Autocorrelation Function (cont.) The gray level primitive in the autocorrelation model is the gray level. The spatial organization is characterized by the correlation coefficient that is a measure of the linear dependence one pixel has on another. The relationship between the autocorrelation function and the power spectral density function: Fourier transforms

DC & CV Lab. CSIE NTU Take a Break

DC & CV Lab. CSIE NTU Digital Transform Methods and Texture In the digital transform method of texture analysis, the digital image is typically divided into a set of non-overlapping small square subimages. The vectors is rexpressed in a new coordinate system. Fourier transform uses the complex sinusoid basic set, Handamard transfer uses the Walsh function basic set, …..

DC & CV Lab. CSIE NTU Digital Transform Methods and Texture (cont.) The key point: The basis vectors of the new coordinate system have an interpretation that relates to spatial frequency or sequency. Since frequency is a close relative of texture, such transformations can be useful.

DC & CV Lab. CSIE NTU Digital Transform Methods and Texture (cont.) In general, features based on Fourier power spectra have been shown to perform more poorly than features based on second-order gray level co-occurrence statistics or those based on first-order statistics of spatial gray level differences.

DC & CV Lab. CSIE NTU Take a Break

DC & CV Lab. CSIE NTU Texture Energy The image is first convolved with a variety of kernels Then each convolved image is processed with a nonlinear operator to determine the total textural energy in each pixel’s neighborhood

DC & CV Lab. CSIE NTU Texture Energy (cont.) The textural energy approach is very much in the spirit of the transform approach. But it uses smaller windows or neighborhood supports.

DC & CV Lab. CSIE NTU Texture Energy (cont.)

DC & CV Lab. CSIE NTU Texture Energy (cont.) - Law’s texture

DC & CV Lab. CSIE NTU Texture Energy (cont.) - Law’s texture

DC & CV Lab. CSIE NTU Texture Energy (cont.) - Law’s texture

DC & CV Lab. CSIE NTU Texture Energy (cont.) - Law’s texture

DC & CV Lab. CSIE NTU Texture Energy (cont.) - Using texture energy for segmentation

DC & CV Lab. CSIE NTU Take a Break

DC & CV Lab. CSIE NTU Texture Edgeness Autocorrelation function and digital transform both reference texture to spatial frequency Texture Edgeness: conceive texture in terms of edgeness per unit area

DC & CV Lab. CSIE NTU Texture Edgeness (cont.) Use small neighborhood to detect microedge Use large neighborhood to detect macroedge

DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU Texture Edgeness (cont.)

DC & CV Lab. CSIE NTU Texture Edgeness (cont.)

DC & CV Lab. CSIE NTU Texture Edgeness (cont.)

DC & CV Lab. CSIE NTU Conclusion of Statistical Approach Co-occurrence probability Autocorrelation Function Texture Energy Texture Edgeness

DC & CV Lab. CSIE NTU Conclusion of Statistical Approach

DC & CV Lab. CSIE NTU Take a Break