1. Robust Foreground Detection in Video Using Pixel Layers Kedar A. Patwardhan, Guillermoo Sapire, and Vassilios Morellas IEEE TRANSACTION ON PATTERN ANAYLSIS AND MACHINE INTELLIGENCE VOL.30, NO, 4, APRIL 2008

2. Outline A typical Nonparametric background modeling : kernel Density estimation Introduction Automatic image layering Foreground detection in video Implementation details and experimental results

3. Nonparametric kernel Density estimation Estimate the pdf directly from the data without any assumptions about the underlying distributions.

4. Introduction A framework for robust foreground detection that works under difficult conditions such as dynamic background and moderately moving camera. The proposed method includes two components: Coarse scene representation as the union of pixel layers Foreground detection in video by propagating these layers using a maximum-likelihood assignment.

5. Introduction Detection challenges: Dynamic background (water ripples, swaying trees, etc) Camera motion Real-time detection requirements

6. Introduction Modeling a video scene as a group of layers instead of single pixels.

7. Introduction Main contributions A principled way of extracting and automatically computing the number of scene-layers Different detection thresholds for different “ layers ” using a principled approach for threshold computation that allows for less Conversion of foreground layers to background and vice-versa based on global layer models Notion of background memory for each pixel, which helps to reduce false detections when foreground disoccludes the background

8. Introduction The main reasons for modeling the scene as a group of layers instead of individual pixels: 1) To exploit spatial-temporal correlation between pixels 2) Use other similar pixels in the scene to model a pixel x, giving a better nonparametric estimate of the process the generated x 3) Handling nominal camera motion without explicit registration since we are not constrained to look at instances of x at exactly the same spatial location in every frame.

9. d

10. Automatic image layering

11. Extracting a Layer: Initial Guess Compute local maximum of the image histogram ( at the gray value ) a radius ( ) which is the square root of the trace of the global covariance matrix. All pixels with gray-value between and form our initial guess or “ layer-candidate ” (L C ) of the layer to be extracted.

12. Automatic image layering

13. Extracting a Layer: Refinement Step By Sampling-Expectation (SE) approach

14. Automatic image layering 3 main steps in this refining steps: Initial step Start with an initial (spatial) probability distribution on the image pixel, where pixels belonging to have high and pixels not in get low values. S-step The image is uniformly sampled to get a set of samples A sample size about 10 to 20 percent of the pixels in the image has been found to be satisfactory.

15. Automatic image layering The likelihood of a pixel belonging to one of the two processes is refined using a weighted Kernel Density Estimation (KDE).

16. Automatic image layering E-step The distribution and are re-estimated.

17. Automatic image layering Extracting a layer: multiple layers and validation In order to ascertain that the extracted layer is both meaningful and of significant size Kullback-Leibler (KL) divergence is proposed. As long as the data (image) supports the initial guess and the refined layers is meaningful, the condition will hold.

18. Automatic image layering : before beginning the layer extraction process, assume that the entire image is the candidate layer (i.e., ), After a few refinement steps, compute the KL divergence. : find the real candidate layer (finite ), perform the refinement and after this layer is stable, compute the KL divergence.

19. Automatic image layering Some pixels being classified as belonging to multiple layers. These pixel along with the residual un-assigned pixels are assigned to one of the layers in their spatial vicinity using maximum-likehood. The video scene, where N are automatically computed.

20. Foreground Detection in video Assign all incoming pixels to : One of the existent layers Identify them as outliers/foreground (assign to layer ) Identify them as part of a temporally persistent (or uninteresting) foreground object and assign them to a completely new background layer.

21. Foreground Detection in video Density estimation

22. Foreground Detection in video s

23. A pixel can be detected as a foreground pixel (outlier) in two case: When the pixel does not belong to any other background layer. When the pixel belongs to the layer of outliers (L 0 ) by maximum-likelihood assignment.

24. Foreground Detection in video Automatic threshold computation Depending upon the homogeneity and integrity of the pixel belonging to a layer, we believe that each layer will need to have a different threshold to achieve the same “ Number of False Alarms ”

25. Foreground Detection in video

26. Temporal persistence and region background model Most state-of-art algorithms change the background model to accommodate persistent object after a particular time has elapsed.

27. Temporal persistence and region background model Persistent foreground After a particular time threshold t persit, we convert the persistent foreground object to a completely new background layer

28. Temporal persistence and region background model Region level background model

29. Temporal persistence and region background model Background memory Let pixel y at spatial location (x,y) be assigned to layers If y is assigned to a persistent outlier which occludes the background layers, we keep testing the pixel y against the layer models.

30. Implementation details and experimental results