1 A Markov Random Field Framework for Finding Shadows in a Single Colour Image Cheng Lu and Mark S. Drew School of Computing Science, Simon Fraser University, Vancouver (CANADA)

2 Objective – finding shadows Many computer vision algorithms, such as segmentation, tracking, and stereo registration, are confounded by shadows. Finding shadows

3 Shadows stem from what illumination effects? Changes of illuminant in both intensity and colour Region Lit by Sky-light only Region Lit by Sunlight and Sky-light Intensity — sharp intensity changes Colour — shadows exist in the chromaticity image

4 Colour of illuminants Wien’s approximation of Planckian illuminants: How good is this approximation? 2500 Kelvin Kelvin 5500 Kelvin

5 Invariant Image Concept For narrow-band Sensors: n aiai Lambertian Surface The responses: Planckian Lighting x Finlayson et al.,ECCV2002 k = R, G, B Shading and intensity term

6 Band-ratio chromaticity G R B Plane G=1 Perspective projection onto G=1 Let us define a set of 2D band-ratio chromaticities: p is one of the channels, (Green, say) [or could use Geometric Mean]

7 Let’s take log’s: Band-ratios remove shading and intensity with Gives a straight line: Shading and intensity are gone.

8 Calibration: find illuminant direction Log-ratio chromaticities for 6 surfaces under 14 different Planckian illuminants, HP912 camera Macbeth ColorChecker: 24 patches Illuminant direction Invariant direction

9 A real image containing shadows The red line refers to the changes of illuminants: same surface lit by two different lights Two lights: Shadows : lit by sunlight and sky-light Non-shadows : lit by sky-light

10 Illuminant discontinuity Illuminant discontinuity pair Illuminant discontinuity pair: Two neighbouring pixels of a single surface, under two different lights

11 Illuminant discontinuity measure Using the means of two neighboring blocks of pixels better than using two neighbouring pixels because of noise and diffuse shadow edges. Illuminant discontinuity angle: Cos of the two vectors

12 Finding Shadows First order neighbors Label image pixels with label l ={shadow, nonshadow} Model this labelling problem using Markov Random Field The label of a pixel depends only on its neighbours

13 Markov Random Field l is a Markov Random Field: l follows a Gibbs distribution: Z=normalizing constant, and U( l ) is an energy function defined with respect to neighbours labelling minimizing energy U( l )

14 Energy function D ij =wQ ij +(1-w)R ij Combining intensity difference Q ij and illuminant discontinuity angle R ij (weight=w) if (l i = l j ) if Roughly, In full,

15 Implementation Gibbs Sampler can be used to minimize the energy: optimization technique. Texture and noise may confuse the discontinuity measure, so the Mean Shift method is used to filter (segment) the image first.

16 Experiments