1. Putting Objects in Perspective Derek Hoiem Alexei A. Efros Martial Hebert Carnegie Mellon University Robotics Institute

2. Understanding an Image

3. Today: Local and Independent

4. What the Detector Sees

5. Local Object Detection True Detection True Detections Missed False Detections Local Detector: [Dalal-Triggs 2005]

6. Work in Context Image understanding in the 70’s Guzman (SEE) 1968 Hansen & Riseman (VISIONS) 1978 Barrow & Tenenbaum 1978 Yakimovsky & Feldman 1973 Brooks (ACRONYM) 1979 Marr 1982 Ohta & Kanade 1973 Recent work in 2D context Kumar & Hebert 2005 Torralba, Murphy, Freeman 2004 Fink & Perona 2003 He, Zemel, Cerreira-Perpiñán 2004 Carbonetto, Freitas, Banard 2004 Winn & Shotton 2006

7. Real Relationships are 3D Close Not Close

8. Recent Work in 3D [Torralba, Murphy & Freeman 2003] [Han & Zu 2003] [Oliva & Torralba 2001] [Han & Zu 2005]

9. Biederman’s Relations among Objects in a Well-Formed Scene (1981): – Support – Size – Position – Interposition – Likelihood of Appearance Objects and Scenes Hock, Romanski, Galie, & Williams 1978

10. Biederman’s Relations among Objects in a Well-Formed Scene (1981): Hock, Romanski, Galie, & Williams 1978 – Support – Size – Position – Interposition – Likelihood of Appearance Contribution of this Paper

11. Object Support

12. Surface Estimation Image SupportVerticalSky V-Left V-Center V-RightV-PorousV-Solid [Hoiem, Efros, Hebert ICCV 2005] Software available online Object Surface? Support?

13. Object Size in the Image Image World

14. Input Image Object Size ↔ Camera Viewpoint Loose Viewpoint Prior

15. Input Image Object Size ↔ Camera Viewpoint Loose Viewpoint Prior

16. Object Position/Sizes Viewpoint Object Size ↔ Camera Viewpoint

17. Object Position/Sizes Viewpoint Object Size ↔ Camera Viewpoint

18. Object Position/Sizes Viewpoint Object Size ↔ Camera Viewpoint

19. Object Position/Sizes Viewpoint

20. What does surface and viewpoint say about objects? Image P(object) P(object | surfaces) P(surfaces) P(viewpoint) P(object | viewpoint)

21. Image P(object | surfaces, viewpoint) What does surface and viewpoint say about objects? P(object) P(surfaces)P(viewpoint)

22. Scene Parts Are All Interconnected Objects 3D Surfaces Camera Viewpoint

23. Input to Our Algorithm Surface EstimatesViewpoint Prior Surfaces: [Hoiem-Efros-Hebert 2005] Local Car Detector Local Ped Detector Object Detection Local Detector: [Dalal-Triggs 2005]

24. Scene Parts Are All Interconnected Objects 3D Surfaces Viewpoint

25. Our Approximate Model Objects 3D Surfaces Viewpoint

26. s1s1 o1o1 θ onon... snsn … Local Object Evidence Local Surface Evidence Local Object Evidence Local Surface Evidence Viewpoint Objects Local Surfaces Inference over Tree Easy with BP

27. Viewpoint estimation Viewpoint Prior Horizon Height Horizon Likelihood Viewpoint Final

28. Object detection 4 TP / 2 FP 3 TP / 2 FP 4 TP / 1 FP Ped Detection Car Detection Local Detector: [Dalal-Triggs 2005] 4 TP / 0 FP Car: TP / FP Ped: TP / FP Initial (Local)Final (Global)

29. Experiments on LabelMe Dataset Testing with LabelMe dataset: 422 images – 923 Cars at least 14 pixels tall – 720 Peds at least 36 pixels tall

30. Each piece of evidence improves performance Local Detector from [Murphy-Torralba-Freeman 2003] Car Detection Pedestrian Detection

31. Can be used with any detector that outputs confidences Local Detector: [Dalal-Triggs 2005] (SVM-based) Car Detection Pedestrian Detection

32. Accurate Horizon Estimation Median Error: 8.5 % 4.5 % 3.0 % 90% Bound: [Murphy-Torralba- Freeman 2003] [Dalal- Triggs 2005] Horizon Prior

33. Qualitative Results Initial: 2 TP / 3 FPFinal: 7 TP / 4 FP Local Detector from [Murphy-Torralba-Freeman 2003] Car: TP / FP Ped: TP / FP

34. Qualitative Results Local Detector from [Murphy-Torralba-Freeman 2003] Car: TP / FP Ped: TP / FP Initial: 1 TP / 14 FPFinal: 3 TP / 5 FP

35. Qualitative Results Car: TP / FP Ped: TP / FP Local Detector from [Murphy-Torralba-Freeman 2003] Initial: 1 TP / 23 FPFinal: 0 TP / 10 FP

36. Qualitative Results Local Detector from [Murphy-Torralba-Freeman 2003] Car: TP / FP Ped: TP / FP Initial: 0 TP / 6 FPFinal: 4 TP / 3 FP

37. Summary & Future Work meters Ped Car Reasoning in 3D: Object to object Scene label Object segmentation

38. Conclusion Image understanding is a 3D problem – Must be solved jointly This paper is a small step – Much remains to be done

39. Thank you

41. A Return to Scene Understanding Guzman (SEE), 1968 Hansen & Riseman (VISIONS), 1978 Barrow & Tenenbaum 1978 Brooks (ACRONYM), 1979 Marr, 1982 Ohta & Kanade, 1978 Yakimovsky & Feldman, 1973 [Ohta & Kanade 1978]

42. Images