1. Non-Local Characterization of Scenery Images: Statistics, 3D Reasoning, and a Generative Model Tamar Avraham and Michael Lindenbaum Technion

2. Characterization of Scenery Images: Overview ●Statistical Characterization ●Rough shape of regions ●Relative location of regions ●Shape of boundaries ●3D Reasoning ●Why are background contours horizontal? ●A Generative Model ●Provides a prior on scenery image annotation ●Generates image sketches, capturing the gist of scenery images manual segmentation and region annotation Given the above segmentation (without texture), which region labeling is more likely ? sky mountain sea scenery images (LabelMe) rocks ground trees mountain sand OR ?

3. Horizontalness Property 1 : Horizontalness Most background objects exceed the image width Background objects are wide and of low height while foreground objects’ shape tend to be isotropic background objects : sky, mountain, sea, trees, field, river, sand, ground, grass, land, rocks, plants, snow, plain, valley, bank, fog bank, desert, lake, beach, cliff, floor foreground objects: all others

4. The relative top-bottom locations of types of background are often highly predictable The probability for a background region of identity A to appear above a background region with identity B, summarized in a histogram for various background identity pairs Topological ordering of background identities can be defined: this DAG is associated with the reachability relation R : {(A,B)| p(A above B) > 0.7} sky mountain desert trees field valley riverlake land sand sea plants ground grass rocks Order / Relative Location Property 2: Order / Relative Location

5. Boundary Shape Property 3: Boundary Shape The characteristics of the upper region’s contour correlate with the region’s identity Chunks of upper boundaries as 1D signals: Curves associated with sea, grass or field resemble DC signals. Curves associated with trees and plants are high frequency signals. Curves associated with mountains resemble signals with low frequency and high amplitude A sample of contour segments associated with background object classes mountain, sea, and trees

6. FlatlandFlatland Why are background regions horizontal? 3D Reasoning: Why are background regions horizontal? Land regions whose contour tangents in aerial images are uniformly distributed appear with strong horizontal bias in images taken by a photographer standing on the ground Θ - the set of tangent angles for contours in aerial images (relative to an arbitrary 2D axis on the surface) Θ’ - the set of angles that are the projections of the angles in Θ on the camera’s image plane ”Place a penny on the middle of one of your tables in Space... look down upon it. It will appear a circle....gradually lower your eyes... and you will find the penny becoming more and more oval to your view....” From Flatland, by Edwin A. Abbott, 1884 lake sand grass flora soil weed p p X1X1 X2X2 X3X3 A schematic illustration of an aerial image An image is taken by a photographer standing on the ground The distribution Θ’, assuming Θ=U[0,180°), h~2[m], z~U[0,1000[m], x~U[0,500[m]] θ’ Θ’ θ Θ

7. 3D Reasoning cont. Ground elevation and slope statistics Two landscape image contour types: 1) The contours between different types of regions on the terrain 2) The contours of mountains associated with occluding boundaries (e.g., skylines) Ground elevation and slope statistics Two landscape image contour types: 1) The contours between different types of regions on the terrain 2) The contours of mountains associated with occluding boundaries (e.g., skylines)

8. 3D Reasoning cont. Ground elevation and slope statistics The contours between different types of regions on the terrain Ground elevation and slope statistics The contours between different types of regions on the terrain The distribution Θ’ assuming Θ =U[0,180°), ϕ~slope statistics, ω~[-90°,90°]. H’s distribution was estimated from sampling an elevation map in pair locations up to 9km apart A point p lies on a boundary between land regions, located on an elevated surface with gradient angle ϕ. The plane is rotated at an angle ω relative to X1 axis X1X1 X2X2 X3X3 O ω Estimated terrain slope distribution using the IIASA-LUC dataset

9. 3D Reasoning cont. Ground elevation and slope statistics The contours of mountains associated with occluding boundaries Tangents in images bounded by the max-slope-over-land-regions statistics Ground elevation and slope statistics The contours of mountains associated with occluding boundaries Tangents in images bounded by the max-slope-over-land-regions statistics Estimated distribution of the maximum slope over land regions, each covering approx. 9 square kilometers * The paper also discussed the effect of land cover and points out other factors that should be considered in a more complete analysis.

10. modeled by PCA of “1D” signals The Generative Model top-bottom order region height A normal distribution for the height covered by each region type upper boundaries top bottom skytreesgroundsea

11. The generative nature of the model makes it possible to: 1) Generate image sketches, capturing the gist of scenery images  ECCV10 2)Obtain priors for region annotation  more recent work The Generative Model: advantages

12. – Given a set of manually segmented and annotated scenery images: Top-bottom order: estimate the transaction matrix M counting number of occurrences of the different ‘moves’. Relative region coverage: estimate mean and variance for the relative average height of each type of region Upper boundary: for each background region type, collect 64-pixels length chunks. Find the first k principle components and Eigen values so that 95% of the variation in the training set is modeled. ( ) – Possible to train different models for different scenery categories. here we trained together of 3 categories: coast, mountain, open country. The Generative Model: Training top bottom skytreesgroundsea

13. – randomly selecting the top-bottom sequence by a random walk on the Markov network, starting at ‘top’, stopping at the sink ‘bottom’. – randomly select the relative average height of each region – randomly generate the boundaries: For each generate 4 chunks The Generative Model: Generating Sketches Sky mountain trees Trees sky mountain trees

14. typical scenery images (LabelMe) manual segmentation and region annotation semantic sketches of scenery images generated by our model typical scenery images (LabelMe) manual segmentation and region annotation semantic sketches of scenery images generated by our model The Generative Model: Generated Results

15. The Generative Model: (More) Generated Results

16. Region Classification Q: Can the new cues contribute to region classification/annotation? A: Complimentary to textural & color cues Goal: to show that region classification using global + local descriptors is better than only local descriptors sky mountain sea? ground? rocks?plants? only layout sky? sea? mountain? ground? sea rocks only color&texture + = sky mountain sea rocks

17. Region Classification - HMM Marginals by the sum-product message passing algorithm Classification by max

18. Region Classification - Discussion General object annotation and detection using context: G. Csurka and F. Perronnin. An efficient approach to semantic segmentation. IJCV, 2010. C. Desai, D. Ramanan, and C. Fowlkes. Discriminative models for multi-class object layout. ICCV, 2009. C. Galleguillos and S. Belongie. Context based object categorization:A critical survey. Comput. Vis. Image Understand, 2010. X. He, R. S. Zemel, and D. Ray. Learning and incorporating top-down cues in image segmentation. ECCV, 2006. S. Kumar and M. Hebert. A hierarchical field framework for unified context-based classification. ICCV, 2005. A. Rabinovich, A. Vedaldi, C. Galleguillos, E. Wiewiora,and S. Belongie. Objects in context. ICCV, 2007. J. Shotton, J. Winn, C. Rother, and A. Criminisi. Textonboost for image understanding: multi-class object recognition and segmentation by jointly modeling appearance, shape and context. IJCV, 81(1):2–23, 2009. Approximated inference needed (e.g., greedy iterative methods, loopy belief propagation) Background region classification of scenery images: a 1D problem Enables exact inference

19. Region Classification - Details Textural & color features: as in Vogel&Scheile IJCV 07: HSV Color histograms Edge direction histograms Gray-level co-occurrences (GCLM, Haralick et al. 73). 4 offsets. For each, contrast, energy, entropy, homogeneity, inverse difference moment, and correlation. and are each modeled with a multiclass probabilistic SVM (LibSVM, Wu, in, Weng 04 ), RBF kernel. 5-fold cross validation at image level. Each training includes parameter selection by inter-training set cross validation. Dataset of 1144 images (LabelMe: coast, open country, mountains). Regions: skymountainseatreesfieldriversandgroundgrasslandrocksplantssnowplateauvalleybanklakebeachcliffTotal Regions 11201489401622366150182943641201143502820 9344979

20. Region Classification – Results 1 ground truthInput imagerelative locationboundary shapecolor&textureall cues

21. Region Classification – Results 2 ground truthInput imagerelative locationboundary shapecolor&textureall cues

22. Accuracy per class: Color&texture: higher accuracy for trees, field, rocks, plants, snow New cues: better for sky, mountain, sea, sand Other classes performance: very low due to their number. Discussion We achieved the goal of showing that the new cues improve texture&color only based region classification. Many classifications counted as errors are actually correct Related to recent work on object categorization with huge amount of categories (Deng, Berg, Li, Fei-Fei ECCV10, Fergus, Weiss, Torralba ECCV10) Work in progress. Region Classification – Results 3 CueAccuracy Color&Texture0.615 Relative Location0.503 Boundary Shape0.452 Relative Loc. + Boundary Shape0.573 Color&Texture + Relative Loc.0.676 Color&Texture + Boundary Shape0.641 All (ORC)0.682 19 categories

23. Summary Focus of characterization of scenery images Intuitive observations regarding the statistics of co- occurrence, relative location, and shape of background regions were explicitly quantified and modeled Some 3D reasoning Non-local properties can capture the gist of images Contextual background region classification with exact inferences. The new cues improve local-descriptors based region classification

24. Future & General Discussion A better way to evaluate region classification: work in progress Use the layout cues for better top-down segmentation (Felzenszwalb&Veksler, CVPR 10). Shape prior to address “shrinking bias” (Vicente, Kolmogorov, Rother, CVPR 08) Use the layout cues to improve scene categorization Augment foreground objects into the model. Extend model to other domains. Use the cues to align pictures. Generated sketches as a basis for rendering. Scenery : too simple? Lets first succeed in understanding those images, following the biological visual system evolution

25. Thank You For Your Time