Texture We would like to thank Amnon Drory for this deck הבהרה : החומר המחייב הוא החומר הנלמד בכיתה ולא זה המופיע / לא מופיע במצגת.

Syllabus Textons TextonsBoost

Textons Run filter bank on images Build Texton dictionary using K-means Map texture image to histogram Histogram Similarity using Chi-square

TextonBoost Build Texton dictionary Texture Layout (pixel, rectangle, Texton) Count number of textons in rectangle Use Integral Image Generate multiple Texture layouts (Features) For each class do 1-vs-all classifier: – For each pixel in class Train GentleBoost Classifier Map strong classifier to probability Take Maximum value

CRF/MRF How to ensure Spatial Consistency? Bayes LikelihoodPosterior ML MAP Prior

Semantic Texton Forest Decision Trees Forest and Averaging Split decision to minimize Entropy Two level STF to add spatial regularization Works well when there is ample data, does not generalize well

(1) Textons B. Julesz, Leung, Malik M. Varma, A. Zisserman (II) TextonBoost J. Shotton, J. Winn, C. Rother, A. Criminisi (III) Semantic Texton Forests J. Shotton, M. Johnson, R. Cipolla (IV) Pose Recognition from Depth Images J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, A. Blake

Textures

Filter Bank

K-means

Texton Histogram

Classification

Results

TextonBoost : Joint Appearance, Shape and Context Modeling for Multi-Class Object Recognition and Segmentation J. Shotton *, J. Winn †, C. Rother †, and A. Criminisi † * University of Cambridge † Microsoft Research Ltd, Cambridge, UK

TextonBoost Simultaneous recognition and segmentation Simultaneous recognition and segmentation Explain every pixel Explain every pixel

TextonBoost Input: Input: 1. Training: Images with pixel level ground truth classification MSRC 21 Database

TextonBoost Input: Input: 1. Training: Images with pixel level ground truth classification. 2. Testing: Images Output: Output: A classification of each pixel in the test images to an object class.

Conditional Random Field Unary TermBinary Term Unary TermBinary Term

Textons Shape filters use texton maps ( [Varma & Zisserman IJCV 05], [Leung & Malik IJCV 01]) Convolve with 17D filter bank (Gaussians, Derivatives of Gaussians, DoGs, LoGs) – Can use Gabor instead Use k-means to create 400 clusters Compact and efficient characterisation of local texture Texton map Colors  Texton Indices Input image  Clustering Filter Bank

CRF: Unary Term Probability of class c_i given feature vector x

Texture-Layout Filters Pair: Feature responses v(i, r, t) Large bounding boxes enable long range interactions rectangle rtexton t (, ) v(i 1, r, t) = a v(i 3, r, t) = a/2 up to 200 pixels (, ) (, )

Texture Layout (Toy Example)

CRF: Binary Term

Potts model Potts model encourages neighbouring pixels to have same label encourages neighbouring pixels to have same label Contrast sensitivity Contrast sensitivity encourages segmentation to follow image edges encourages segmentation to follow image edges

Accurate Segmentation? Boosted classifier alone – effectively recognises objects – but not sufficient for pixel- perfect segmentation Conditional Random Field (CRF) – jointly classifies all pixels whilst respecting image edges unary term only CRF

The TextonBoost CRF Texture-LayoutColorlocation edge Unary Term Binary Term

Location Term Capture prior on absolute image location Capture prior on absolute image location treeskyroad Texture-LayoutColorlocation edge

Color Term Texture-LayoutColorlocation edge

Texture-Layout Term

Texton Boost - Summary Performs per-pixel classification using: 1. Statistics learned from Training Set: - Absolute location statistics - Configuration of textured areas around pixel of interest. 2. Cues from the Test Image: - Edges - Object Colors 3. Priors.

Results on 21-Class Database building

Effect of Model Components Shape-texture potentials only:69.6% + edge potentials:70.3% + Color potentials:72.0% + location potentials:72.2% shape-texture + edge + Color & location pixel-wise segmentation accuracies