1. A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu

2. David Martin - UC Berkeley - ICCV 20012 Motivation Berkeley Segmentation Dataset  Groundtruth for image segmentation of natural images App#1: A segmentation benchmark App#2: Ecological statistics

3. David Martin - UC Berkeley - ICCV 20013 Benchmark Example for Recognition MNIST handwritten digit dataset [LeCun, AT&T] http://www.research.att.com/~yann/exdb/mnist/index.html Training set, test set, evaluation methodology, algorithm ranking

4. David Martin - UC Berkeley - ICCV 20014 The Image Dataset 1000 Corel images –Photographs of outdoor scenes –Texture is common –Large variety of subject matter –481 x 321 x 24b

5. David Martin - UC Berkeley - ICCV 20015

6. 6

7. 7

8. 8

9. 9 Establishing Groundtruth Def: Segmentation = Partition of image pixels into exclusive sets Manual segmentation by human subjects –Custom Java tool to facilitate task Currently: 1000 images, 5500 segmentations, 20 subjects Naïve subjects (UCB undergrads) given simple, non-technical instructions

10. David Martin - UC Berkeley - ICCV 200110 Directions to Image Segmentors You will be presented a photographic image Divide the image into some number of segments, where the segments represent “things” or “parts of things” in the scene The number of segments is up to you, as it depends on the image. Something between 2 and 30 is likely to be appropriate. It is important that all of the segments have approximately equal importance.

11. David Martin - UC Berkeley - ICCV 200111

12. David Martin - UC Berkeley - ICCV 200112

13. David Martin - UC Berkeley - ICCV 200113 The segmentations are not identical. But are they consistent??

14. David Martin - UC Berkeley - ICCV 200114 Perceptual organization forms a hierarchy image backgroundleft birdright bird grassbush headeye beakfar body headeye beak body Each subject picks a slice through this hierarchy.

15. David Martin - UC Berkeley - ICCV 200115 Quantifying inconsistency S1S1 S2S2 How much is S 1 a refinement of S 2 at pixel  ?

16. David Martin - UC Berkeley - ICCV 200116 Segmentation Error Measure One-way Local Refinement Error: Segmentation Error allows refinement in either direction at each pixel:

17. David Martin - UC Berkeley - ICCV 200117 Human segmentations are consistent Distribution of segmentation error over the dataset.

18. David Martin - UC Berkeley - ICCV 200118 ColorGrayInvNeg

19. David Martin - UC Berkeley - ICCV 200119 InvNeg

20. David Martin - UC Berkeley - ICCV 200120 ColorGrayInvNeg

21. David Martin - UC Berkeley - ICCV 200121 Gray vs. Color vs. InvNeg Segmentations SE (gray, gray) = 0.047 SE (gray, color) = 0.047 Color may affect attention, but doesn’t seem to affect perceptual organization SE (gray, gray) = 0.047 SE (gray, invneg) = 0.059 InvNeg interferes with high-level cues (2500 gray, 2500 color,200 invneg segmentations)

22. David Martin - UC Berkeley - ICCV 200122 Benchmark Methodology Separate training and test datasets with no images in common Generate computer segmentation(s) of each image in test set –Determine error of each computer segmentation using SE measure –Algorithm scored by mean SE Example: –SE (human, human) = 0.05 –SE (NCuts, human) = 0.22 –SE (different images) = 0.30

23. David Martin - UC Berkeley - ICCV 200123 Ecological Statistics of Image Segmentations Validating and quantifying Gestalt grouping factors [Brunswik 1953] Priors on region properties Recent work on natural image statistics: –Filter outputs [Ruderman 1994, Olshausen & Field 1996, Yuille et. al. 1999] –Object sizes [Alvarez, Gousseau, Morel 1999] –Shape [Zhu 1999] –Contours [August & Zucker 2000, Geisler et al. 2001]

24. David Martin - UC Berkeley - ICCV 200124 Relative power of cues Pairwise grouping cues –Proximity –Luminance similarity –Color similarity –Intervening contour –Texture similarity

25. David Martin - UC Berkeley - ICCV 200125 P (Same Segment | Proximity)

26. David Martin - UC Berkeley - ICCV 200126 P (Same Segment | Luminance)

27. David Martin - UC Berkeley - ICCV 200127 Bayes Risk for Proximity Cue

28. David Martin - UC Berkeley - ICCV 200128 Bayes Risk for Various Cues Conditioned on Proximity

29. David Martin - UC Berkeley - ICCV 200129 Mutual Information for Various Cues Conditioned on Proximity

30. David Martin - UC Berkeley - ICCV 200130 Priors on Region Properties Area Convexity

31. David Martin - UC Berkeley - ICCV 200131 Empirical Distribution of Region Area y = Kx -   = 0.913 Compare with Alvarez, Gousseau, Morel 1999.

32. David Martin - UC Berkeley - ICCV 200132 Empirical Distribution of Region Convexity

33. David Martin - UC Berkeley - ICCV 200133 Conclusion Large new database of segmentations of natural images by humans A segmentation benchmark Ecological statistics –Relative power of grouping cues –Priors on region properties http://www.cs.berkeley.edu/~dmartin/segbench