1. Shadow Detection In Video Submitted by: Hisham Abu saleh

2. Why detecting shadows? Moving objects detection: Shadows can cause object merging, object shape distortion or object loses. Shadow points are detected as foreground They differ from background. They move the same as the object.

3. Why detecting shadows? cont.

4. Shadow Detection Algorithms Algorithms are divided into: Deterministic approaches Model based Best results. Too Complex. Non-Model based Statistical approaches (use probabilistic functions). Parametric methods. Nonparametric methods. Selecting parameters is critical issue.

5. Features presented by algorithms Spectral Gray level Color (next lecture) Spatial Pixel level Region level Frame level Temporal Static Dynamic

6. Algorithm Taxonomy

7. Algorithm Comparison We will compare between four algorithms: Statistical Nonparametric Statistical Parametric Deterministic Nonmodel-based (DNM1) Deterministic Nonmodel-based (DNM2) The deterministic model-based class has not been considered due to its complexity and due to its reliance on very specific task domain assumptions.

8. Statistical Nonparametric Approach Horprasert Color, Pixel level, Static. Shadows have similar chromaticity, but lower brightness than the background model. A statistical learning procedure is used to automatically determine the appropriate thresholds.

9. Statistical Parametric Approach Mikie’ Color, Region level, Dynamic. The probability of a pixel belonging to shadow is computed by assuming the color of the pixel not shadowed, and by using an approximated linear transformation to estimate the color of the point covered by a shadow. Manual segmentation of a certain number of frames has to be done to collect statistics and to compute the values of matrix of the transformation.

10. Deterministic Nonmodel-based Approach (DNM1) Cucchiara, Color, Pixel level, Static. This algorithm works in the HSV color space (closer to the human perception of color and it has more accuracy in distinguishing shadows). A shadow cast on a background does not change its hue significantly.

11. Deterministic Nonmodel-based Approach (DNM2) Stauder Gray level, Frame level, Dynamic. The only algorithm that deals with penumbra. Reference frame Previous frame Has limitations: Object speed, Noise sensetive. Background frame

12. Deterministic Nonmodel-based Approach (DNM2) cont. The image luminance: If illuminated If penumbra If umbra Describes the transition inside the penumbra p(x,y) – The reflectance of the object surface. E(x,y) – irradiance. C p – Intensity of light source. C A – ambient light.

13. Deterministic Nonmodel-based Approach (DNM2) cont. Assumptions: 1. light source intensity c p is high. The frame difference at a pixel changed by a moving cast shadow will be large. If pixel is inside the umbra of a cast shadow at time instant k and outside the shadow at time instant k + 1. The reflectance of a static background does not change with time, thus the frame difference will be then:

14. Deterministic Nonmodel-based Approach (DNM2) cont. 2. Camera and background are static. 3. Background is plane, light source position is distant from background. 4. Light source size and distance between moving object and background are not negligible compared to the distance between light source and object.

15. Deterministic Nonmodel-based Approach (DNM2) cont. A. Detection of Static Background Edges edges in the current and in the previous image are detected and classified into moving and static edges.

16. Deterministic Nonmodel-based Approach (DNM2) cont. B. Detection of Uniform Changes of Shading Frame Ratio: If the frame ratio is locally spatially constant, a moving cast shadow is assumed at position x,y.

17. Deterministic Nonmodel-based Approach (DNM2) cont. A shadow is decided to be in a region if it there is “darker” uniform region. Static edges hint at a static background and can be exploited to detect nonmoving regions inside the frame difference.

18. Deterministic Nonmodel-based Approach (DNM2) cont. C. Penumbra Detection According to assumption 4 the cast shadow has penumbra. The penumbra causes a soft luminance step at the contour of a shadow that is characterized by its step height, step width and its gradient.

19. Deterministic Nonmodel-based Approach (DNM2) cont. According to a table we put a candidate as a penumbra or not.

20. Algorithm comparison Two important quality measures: good detection Low probability of misclassifying a shadow point. Minimizing the shadow points classified as background/foreground. good discrimination the probability of classifying nonshadow points as shadow should be low.

21. Algorithm comparison We define two metrics: TP – True Positive FN – False Negative S – Shadow F – Foreground (object)

22. Algorithm comparison The Results: SNP – Very effective, sensitive to dark objects and noise. SP – Not very effective since the assumption that the transformation matrix is constant on the entire image. DNM1 – Good accuracy, stable performance. DNM2 – fails when the background is not planar.

23. Example

24. Physics-Based Shadow Detection This step detects moving pixels (object, shadow, and some erroneous pixels).

25. Physics-Based Shadow Detection cont. pixels on a detected surface cannot be shadow if they have higher intensity than the actual background. This test does not reduce the shadow areas, but may successfully reduce object areas.

26. Physics-Based Shadow Detection cont.

27. Shadow pixels falling on neutral surfaces, such as asphalt roads, tend to be more blue. This test is not applied to all the pixels, but to neutral or gray surfaces that have low saturation (< 0.3)

28. Physics-Based Shadow Detection cont.

29. This step performs surface segmentation. The connectivity C of two neighboring pixels p1 and p2 with intensities u and v:

30. Physics-Based Shadow Detection cont. H(u,v) is defined as follows: If two neighboring pixels belong to the same surface they will have temporal ratios that are close together.

31. Physics-Based Shadow Detection cont.

32. This step removes the effect of sky illumination. We subtract the foreground pixel values from the background over the masked area.

33. Physics-Based Shadow Detection cont. The user selects a patch of shadow region from a frame during the training. The materials colors are stored to the verification step.

34. Physics-Based Shadow Detection cont. This step performs matching of body color of various surfaces with the stored body color of materials that we expect to see in the scene. The algorithm has training and testing phases.

35. Physics-Based Shadow Detection cont.

36. References A. Prati, I. Mikic, M. Trivedi, R. Cucchiara, “Detecting Moving Shadows: Algorithms and Evaluation”, IEEE Transactions on pattern analysis and machine intelligence, vol. 25, no. 7, July 2003. S. Nadimi, B. Bhanu, “Physical Models for Moving Shadow and Object Detection in Video”, IEEE Transactions on pattern analysis and machine intelligence, vol. 26, no. 8, August 2004. J. Stauder, R. Mech, J. Ostermann, “Detection of Moving Cast Shadows for Object Segmentation”, IEEE Transactions on multi media, vol. 1, no. 1, March 1999.

37. References cont. A. Prati, I. Mikic, C. Grana, M. Trivedi, “Shadow Detection Algorithms for Traffic Flow Analysis: a Comparative Study”.