Digital Photography with Flash and No-Flash Image Pairs By: Georg PetschniggManeesh Agrawala Hugues HoppeRichard Szeliski Michael CohenKentaro Toyama, Microsoft Corporation Presented by: Yael Amsterdamer Advanced Topics in Visual Computing, Spring 2012

The Dilemma: to Flash or not to flash? 2Digital Photography with Flash and No-Flash Image Pairs Natural lighting Low signal-to-noise ratio (SNR) Loss of details Longer exposure – motion blur Harsh, unnatural lighting High SNR More details May cause unwanted artifacts (red eye, shadows, specularities)

Why not both? The idea: use the good features of each photo to create a better image 3Digital Photography with Flash and No-Flash Image Pairs

Other works on combining images 4Digital Photography with Flash and No-Flash Image Pairs Different exposures Panoramic view Relighting Current work handles 2 images with specific characteristics, Envisioned for camera firmware Current work handles 2 images with specific characteristics, Envisioned for camera firmware

Different Applications Denoising Detail transfer – Side requirement: handling flash induced shadows and specularities White Balancing Continuous Flash Adjustment Red Eye Removal 5Digital Photography with Flash and No-Flash Image Pairs

Camera Adjustment to light A certain amount of light is needed for a photo Taking pictures of a dark scene – Increasing exposure time – Opening the aperture – Increasing ISO (sensor sensitivity) The problem – high ISO  noise affects high-frequency details 6Digital Photography with Flash and No-Flash Image Pairs

The Input F – the photo with flash A – the no-flash (ambient) photo – Composed of 3 RGB intensity values for each pixel – A p – one pixel in A Same camera position, focus, aperture – Otherwise, registration is required Adjusted ISO and exposure time Undergo non-linear tone-mapping operations 7Digital Photography with Flash and No-Flash Image Pairs

1st Application: Denoising A classic problem in image processing – The target: analytically reduce SNR One type of solution: smoothing filters Bilateral Filter [Tomasi and Manduchi 1998] – Fast, non-iterative – Averaging close-by pixels – Low-pass filter, combined with – Edge-stopping function 8Digital Photography with Flash and No-Flash Image Pairs

Bilateral filter However, results still have noise or blur (or both) 9Digital Photography with Flash and No-Flash Image Pairs ambient flash Bilateral filter

Improvement: Joint Bilateral Filter In the flash image there are much more details Why not use F to find edges? 10Digital Photography with Flash and No-Flash Image Pairs Bilateral filter Joint Bilateral filter The difference

Not all edges are real … May cause over- or under-blur in joint bilateral filter We need to eliminate their effect 11Digital Photography with Flash and No-Flash Image Pairs

Detecting shadows Observation: the pixels in the flash shadow should be similar to the ambient image Not identical: – Noise – Inter-reflected flash Compute a shadow mask Take pixel p if is manually adjusted Mask is smoothed and dilated 12Digital Photography with Flash and No-Flash Image Pairs Pixels for which we will not use F

Detecting specularities Take pixels where sensor input is close to maximum (very bright) – Over fixed threshold Create a specularity mask Also smoothed and dilated M – the combination of shadow and specularity masks – Where M p =1, we use A Base. For other pixels we use A NR. 13Digital Photography with Flash and No-Flash Image Pairs

There is so much denoising can do It cannot add details missing in the ambient image Exist in flash image because of high SNR We use a quotient image: Multiply with A NR to add the details Masked in the same way 14Digital Photography with Flash and No-Flash Image Pairs Bilateral filtered Reduces the effect of noise in F

Unified process Denoising + detail transfer + masking shadows and specularities 15Digital Photography with Flash and No-Flash Image Pairs

More examples 16Digital Photography with Flash and No-Flash Image Pairs

The opposite of preserving the atmosphere The goal: simulate white light in the scene – What color is the light in the scene? Flash image can help us: The difference between F and A is only the flash light Assume it is proportional to the surface albedo – Inherent property of the surface – Assume the surface reflects light in the same color (will not work for plastic, glass…) 17Digital Photography with Flash and No-Flash Image Pairs

Computing the ambient light We cancel the color of the surface The average c of C p(col) for pixels with certain properties is the ambient light color Then we scale A to cancel the ambient light 18Digital Photography with Flash and No-Flash Image Pairs

Example 19Digital Photography with Flash and No-Flash Image Pairs Estimated ambient light Computed color Ambient original White- balanced

Combining flash and no-flash images Providing complementary details about the scene Use for denoising and detail transfer Use for white balancing Future work – More robustness – Automatization of configuration 20Digital Photography with Flash and No-Flash Image Pairs

Thank you!