Unsupervised Automation of Photographic Composition Rules Serene Banerjee and Brian L. Evans Computer Engineering Area Dept. of Electrical and Computer Engineering The University of Texas at Austin

1/19/2004 Automation of Composition Rules 2 Motivation Problem: Amateur photographers often take low-quality pictures with digital still cameras Personal use Professionals who need to document (e.g.. realtors and architects) Goal: Automate photographic composition rules and find alternatives to the picture being acquired Analyze scene, including detection of main subject Develop algorithms to automate rules Main subject cropped Too much background

1/19/2004 Automation of Composition Rules 3 Solution Solution #1: Automatically detect main subject Independent of indoor/outdoor setting or scene Low implementation complexity, fixed-point computation Solution #2: Automate a few photograph composition rules Rule of thirds for placing the main subject Simulated background blur for motion pictures or depth-of-field Following rule-of-thirds Blur background for action pictures

1/19/2004 Automation of Composition Rules 4 1: Main subject 2: Lenses 3: CCD 4: Imaging device 5: Raw data Digital Still Cameras Converts optical image to electric signal using charge coupled device (CCD) Software control Zoom Focus, e.g. auto-focus filter Shutter aperture and speed White balance: Corrects color distortions Settings that can be controlled (with added hardware) Camera angle Aspect ratio: Landscape or portrait mode Produces JPEG compressed images

1/19/2004 Automation of Composition Rules 5 Main Subject Detection Methods Two differently focused photographs [Aizawa, Kodama, Kubota; ] One has foreground in focus, and other has background in focus Significant delay involved in changing the focus Bayes nets based training [Luo, Etz, Singhal, Gray; ] Bayesian network trained on example set and tested later Training time involved: suited for offline applications Multi-level wavelet coefficients [Wang, Lee, Gray, Wiederhold; ] Expensive to compute and analyze wavelet coefficients Iterative classification from variance maps [Won, Pyan, Gray; 2002] Optimal solution from variance maps and refinement with watershed Suitable for offline applications involving iterative passes over image

1/19/2004 Automation of Composition Rules 6 Proposed Main Subject Detection User starts image acquisition Focus main subject using auto-focus filter Partially blur background and acquire resulting picture Open shutter aperture (by lowering f-stop) which takes about 1 s Foreground edges stronger than background edges While acquiring user-intended picture, process blurred background picture to detect main subject Generate edge map (subtract original image from sharpened image) Apply edge detector (Canny edge detector performs well) Close boundary (e.g. gradient vector flow or proposed approximation)

1/19/2004 Automation of Composition Rules 7 Symmetric 3 x 3 sharpening filter For integer  and , coefficients are Integer when dropping 1/(1 +  ) term Fractional when -1 – 2    and 1/(1 +  ) is power-of-two Generate edge map Subtract original image from sharpened image Main subject region now has sharper edges Generate Edge Map f(x,y) g(x,y)f sharp (x,y) Smoothing filter f smooth (x,y) k + Model for an image sharpening filter Sharpening filter

1/19/2004 Automation of Composition Rules 8 Boundary Closure Gradient vector flow method [Xu, Yezzi, Prince; ] Compute gradient Outer boundary of detected sharp edges is initial contour Change shape of initial contour, depending on gradient Shape converges in approximately 5 iterations Disadvantage: computationally and memory intensive Approximate lower complexity method Select leftmost & rightmost ON pixel and make row between them ON Can detect convex regions but fails at concavities

1/19/2004 Automation of Composition Rules 9 Automation of Rule-of-Thirds Goal: Center of mass of the main subject at 1/3 or 2/3 of the picture width (or height) from the left (or top) edge Solution: For n-D, define function that attains minimum when center of mass placed as desired and increases otherwise Shift picture so that minimum is attained Implementation: For 2-D, sum of Euclidean distance from the 4 points Measure which of the 4 points is closest to the current position of the center of mass Shift picture so that the rule-of-thirds is followed

1/19/2004 Automation of Composition Rules 10 Simulated Background Blurring Goal: Filter the image background and add artistic effects keeping the main subject intact Solution: Original image masked with detected main subject mask Region of interest filtering performed on masked image Possible motion blurs Linear blur: subject or camera motion Radial blur: camera rotation Zoom: change in zoom Applications Enhance sense of motion where the main subject is moving Digitally decrease the depth-of-field of the photograph

1/19/2004 Automation of Composition Rules 11 Proposed Module Measure how close rule-of-thirds followed Auto-focus filter Lower f-stop for blur Filter to generate edge map Detect sharper edges Close boundary Original Image Automate rule-of-thirds Simulate background blur Binary Main Subject Mask Generated Picture with Rule-of-Thirds Generated Picture with Blur

1/19/2004 Automation of Composition Rules 12 Implementation Complexity Number of computations and memory accesses per pixel Main subject detection: convolution with symmetric 3x3 filter, edge detection, approximate boundary closure Rule-of-thirds: center of mass (1 division, 4 compares), shift pixels Background blurring: convolution with symmetric 3x3 filter Digital still cameras use ~160 digital signal processor instruction cycles per pixel Processing stepMultiply-Accumulates /pixel Comparisons/ pixel Memory accesses/pixel Main subject detection Rule of thirds 211 or 3 Background blurring 94

1/19/2004 Automation of Composition Rules 13 Results (1) Original image with main subject(s) in focus Detected strong edges with proposed algorithm Detected main subject mask Rule-of-Thirds: Main subject repositioned Simulated background blur

1/19/2004 Automation of Composition Rules 14 Results (2) Original image with main subject(s) in focus Detected strong edges with proposed algorithm Detected main subject mask Rule-of-Thirds: Main subject repositioned Simulated background blur

1/19/2004 Automation of Composition Rules 15 Results (3) Original image with main subject(s) in focus Detected strong edges with proposed algorithm Detected main subject mask Rule-of-Thirds: Main subject repositioned Simulated background blur

1/19/2004 Automation of Composition Rules 16 Conclusion Developed automated low-complexity one-pass method for main subject detection in digital still cameras Processes picture taken with blurred background All calculations in fixed-point arithmetic Automates selected photographic composition rules Rule-of-thirds: Placement of the main subject on the canvas Simulated background blur: motion and depth-of-field Applications: digital still cameras, surveillance, constrained image compression, and transmission and display Copies of MATLAB code, poster, and paper, available at