1. City Forensics: Using Visual Elements to Predict Non-Visual City Attributes Sean M. Arietta, Alexei A. Efros, Ravi Ramamoorthi, Maneesh Agrawala Presented by: Manu Agarwal

2. Outline Introduction Related Work Methodology proposed in the paper Results Applications Discussion

3. One of these is from Paris This is Paris Clap if… Slide credits: Doersch, Singh

4. Clap if… Slide credits: Doersch, Singh

6. Modeling predictive relationships Higher greenery City appearance Cases of depression Higher housing prices Broken Windows Theory! Broken glass, graffiti, trashIncreased crime rate

7. Violent crime rate High housing prices

8. Related Work Doersch et al. – Method for identifying visual elements of a city that differentiate it from other cities – Binary classification Koller et al., Srinivasan et al. – Use video to track crowds for detecting flow patterns – Availability of such videos is limited

9. Problem Input: a set of measured (location, attribute- value) pairs and a set of (location, street-side panorama) pairs Output: Predictor that can estimate the value of a non-visual city attribute based on visual appearance

10. Methodology Spatially interpolate the input (location, attribute-value) data Build a bank of SVMs Build an attribute predictor from the resulting bank of SVMs using SVR

11. Interpolating Non-Visual City Attribute Values Use the radial basis function (RBF) r is the Euclidean distance between locations Authors use ɛ=2

12. Methodology Spatially interpolate the input (location, attribute-value) data Build a bank of SVMs Build an attribute predictor from the resulting bank of SVMs using SVR

13. Constructing the Visual Element Detectors 100k200k300k400k500k600k700k800k900k1M1.1M PositiveNegative Negative set Positive set

14. Constructing the Visual Element Detectors

15. Extract the set of image features in each panorama projection HOG+color features

16. Constructing the Visual Element Detectors Compute 100 nearest neighbors of features sampled randomly from the positive set PatchMatches

17. Constructing the Visual Element Detectors Compute an SVM for each nearest neighbor set Keep the top 100 SVMs

18. Methodology Spatially interpolate the input (location, attribute-value) data Build a bank of SVMs Build an attribute predictor from the resulting bank of SVMs using SVR

19. Computing the Predictor Features SVM scores

20. Computing the Predictor Retain the top 3 detection scores for each of the 100 SVMs

21. Computing the Predictor Estimate parameters w and b such that the following loss is minimized ɛ controls the magnitude of error that we can tolerate

22. Results: Analysis of Prediction Accuracy

24. Results: Prediction Maps

26. Results: Deep Convolutional Neural Network Features

27. Applications: Defining Visual Boundaries of Neighborhoods

29. Applications: Attribute-Sensitive Wayfinding

30. Applications: Validating Visual Elements for Prediction

32. Discussion Presence of other dominant factors controlling non-visual attributes Retain top 5 SVM detection scores Image patch level features vs whole image features Combining predictors from two or more cities Experiment with smaller cities Use fc7 features or finetune CNN

33. Thank You!