1. Modeling Perspective Effects in Photographic Composition Zihan Zhou, Siqiong He, Jia Li, and James Z. Wang The Pennsylvania State University

2. What is Photo Composition? Everyone wants to take good pictures… 2 In photography, it’s not just what you shoot that counts – how you organize them within the frame is crucial, too.

3. The Use of Perspective Effects in Photographic Composition In photography, experienced photographers often make use of the linear perspective effect to emphasize the sense of 3D space in a 2D photo Can we teach computer to understand the use of perspective effects in photo composition? 3 Photo credit: Flickr

4. Real World Applications Image enhancement Image summarization and retrieval for large-scale database On-site composition feedback to photographers 4 Does my photo look good?

5. How Should We Model the Perspective Effects? Our Approach: A geometric image segmentation Partition an image into regions (approx. planar structures) according to the dominant vanishing point. Holistic: it encodes accurate, global geometric information about the scene Compact: it can be efficiently employed in real-world applications 5

6. Challenge 1 How to partition the image into photometrically and geometrically consistent regions? Existing image segmentation methods do NOT respect the perspective effects in photos P. Arbelaez, et al. Contour detection and hierarchical image segmentation. TPAMI, 2011 6

7. Challenge 2 How to detect the dominant VP in an arbitrary image? Clustering line segments? 7

8. Our Contributions 1.The first work to model the perspective effects in photographic composition via a geometric image segmentation framework Beyond photometric cues 2.The first work to detect the dominant VP in arbitrary images Without relying on edges 3.Applications to on-site composition feedback 8

9. Review of Hierarchical Image Segmentation P. Arbelaez, et al. Contour detection and hierarchical image segmentation. TPAMI, 2011 9 An over- segmentation of the image Find two regions with minimum distance Merge them and update the distances of relevant pairs Repeat Original ImageInitial Over-segmentationFinal Result

10. Geometric Distance Measure Intuition: If the boundary between two regions is parallel to the dominant direction, these two regions are likely to lie on different planes. 10

11. Geometric Distance Measure Our approach: measures the similarity of angle distribution of each region w.r.t. the dominant VP in a polar coordinate system 11

12. Combining Photometric and Geometric Cues 12

13. Quantitative Evaluation: Image Segmentation 200 images from Flickr Manually labeled ground truth segmentation Compared against gPb-owt-ucm P. Arbelaez, et al. Contour detection and hierarchical image segmentation. TPAMI’11 13

14. Detecting the Dominant Vanishing Point A simple exhaustive search: For each candidate location on a uniform grid mesh, compute: 14 Correct HypothesisIncorrect Hypothesis Consensus Scores for All Hypotheses

15. Quantitative Evaluation: VP Detection 400 images from Flickr Manually labeled vanishing points Compared against [Tardiff, 2009] and [Tretiak et al., 2012] 15

16. Application: On-Site Feedback via Composition-Sensitive Image Retrieval Exemplar dataset: 3,728 images collected from Flickr by querying the keyword “vanishing point”. Similarity measure: Geometric segmentation map VP location 16 Does my photo look good?

17. Image Retrieval Results 17 Queries

18. Summary We propose to model the perspective effects in photographic composition via a novel geometric image segmentation framework We develop a novel method to detect the dominant VP in arbitrary images We demonstrate the applications of our model to on-site composition feedback As future work, we plan to extend our method to more complex scenes More than one VPs Irrelevant foreground objects 18

19. Thank you! 19 Photo credit: Flickr