1. Photo-Quality Enhancement based on Visual Aesthetics S. Bhattacharya*, R. Sukthankar**, M.Shah* *University of Central Florida, **Intel labs & CMU

2. Motivation

3. Related work Quality enhancement framework Visual aesthetics Aesthetic appeal assessment Enhancement through recomposition Experimental results Conclusions Future directions Outline

4. Low-level (dehazing etc.) Related Work Domain-specific (face beautification etc.) T. Leyvand,et al., “Data-Driven Enhancement of Facial Attractiveness”, ACM SIGGRAPH 08 K. He, J.Sun X. Tang, “Single Image Haze Removal using Dark Channel Prior”, CVPR 09

5. Overview Enhanced Image Input Image Enhancement Engine Assessment Engine Aesthetic Features Image Semantics, Aesthetic Features Appeal Prediction Recomposition Aesthetic Model

6. Visual Aesthetics: Rule of Thirds Motivated by Renaissance Paintings… Rule of thirds: Subject of interest is aligned to one of the stress points Professional photographs also abide this: http://howtophotography.org/wp-content/uploads/2010/06/rule-of-thirds-photo2.jpg http://hoocher.com/Joseph_William_Turner/Joseph_William_Turner.htm

7. Visual Aesthetics: Golden Ratio http://hoocher.com/Joseph_William_Turner/Joseph_William_Turner.htm Divine proportion: Horizon divides sky and sea/land according to golden ratio. http://www.dptips-central.com/rules-of-composition.html An example professional photographic composition: ~1.618k Sky Sea Sky Land k

8. Single subject Compositions (384) Modeling Aesthetics: Dataset Landscapes/Seascapes (248) http://www.flickr.com

9. Single subjects Modeling Aesthetics: User study Landscapes/Seascapes http://www.flickr.com 121514 … Rank Assignment between 1-5 Ground Truth Appeal Factors

10. Modeling Aesthetics: User study 1.76 4.23 Poorly rated images Best rated images

11. Modeling Aesthetics: User study Appeal Factor Intervals User Agreements Good Compositions Poor compositions

12. Modeling Aesthetics: Features (a) Relative Foreground Location (Rule of Thirds) Visual Attention Center Stress Point

13. Modeling Aesthetics: Features (b) Visual weight deviation from Golden Ratio (Divine Proportion) YkYk YgYg

14. Experiments (Assessment) Learn Support Vector Regression models Prediction accuracy: ◦ Single subject compositions ~ 87% ◦ Landscapes/Seascapes ~ 91% Smooth mapping between Appeal factor and Aesthetic Features Relative Foreground Location Visual Weight Deviation

15. Spatial Recomposition

16. Why Cropping does not work? Optimal Crop

17. Recomposition: Algorithm I Input Image Labeled Elements Semantic Segmentation Single Subject? Optimal Object Placement Spatial Recomposition In-painting

18. Semantic Segmentation Input Image Geometric Context Classifier* *D. Hoiem, A.A. Efros, and M. Hebert, "Geometric Context from a Single Image", ICCV 2005 Post Processing Sky Support Horizon Segmented Foreground

19. Optimal Object Placement Find x that Maximizes Appeal Intensity Term Labeled Image Support Neighborhood Gradient Term s.t. neighbors stay “like neighbors” +

20. Optimization (Example) PAF = 3.31PAF = 3.68 Semantic constraint prevents this PAF = 3.22 Original Image PAF = 4.53 Optimal Solution X

21. Perspective Scaling Scaling Factor Vanishing Point Optimal location Visual Attention Center Scaled Foreground

22. Inpainting Foreground Hole Inpaint Hole Yunjun Zhang. Jiangjian Xiao. Mubarak Shah, “ Region Completion in a Single Image”, EUROGRAPHICS 04

23. Recomposition: Algorithm 2 Input Image Labeled Elements Semantic Segmentation Land/Sea scape? Visual Weight Balancing Optimally Crop/Expand

24. Ratio of Current extents Balancing Visual Weights h = vertical extent of the balanced image Solve for h (sign of h determines crop/expansion) YkYk YgYg Y k +h YgYg

25. Experimental Results Horse is moved to a more visually pleasing location Scaled appropriately Appeal increases by 64% Single Subject Composition Before RecompositionAfter Recomposition

26. Results BeforeAfter PAF = 2.45 PAF = 4.29 PAF = 3.98PAF = 4.46

27. Results BeforeAfter PAF = 3.13PAF = 4.19 PAF = 4.02PAF = 4.34

28. Results Before After PAF = 3.77PAF = 4.25 PAF = 3.92PAF = 4.11

29. Results BeforeAfter PAF = 4.06 PAF = 4.68 PAF = 2.71PAF = 3.26

30. Optimally cropped support region to increase weights for sky Appeal factor increased by 51% Visual weight balancing Results Before RecompositionAfter Recomposition

31. Balancing Visual weights BeforeAfter PAF = 3.83 PAF = 4.02 PAF = 3.92 PAF = 4.38

32. Balancing Visual weights BeforeAfter PAF = 4.02 PAF = 4.71 PAF = 4.17 PAF = 4.49

33. Not Perfect Algorithm says nice, humans: otherwise PAF = 2.34Fa = 2.41 (Ground Truth) Before PAF = 3.63Fa = 2.54 (Ground Truth) After

34. Summary: Optimal Placement BeforeAfter Increased # of Highly rated Images Decreased # of Poorly rated Images

35. Summary: Visual Weights BeforeAfter Increased # of Highly rated Images Decreased # of Poorly rated Images

36. Conclusion Intelligent photo recomposition Can also be used for aesthetic filtering Easy to use practical tool

37. Future Work Synthesizing ideal image from many photos of the same scene Recomposition for videos

38. Questions?