Performance Evaluation of Object Detection Algorithms for Video Surveillance Jacinto C. Nascimento, Member, IEEE, and Jorge S. Marques IEEE Transactions On Multimedia VOL.8, NO.4, AUGUST 2006

Outline Introduction Related Work Segmentation Algorithms Proposed Framework Tests on PETS2001 Dataset Conclusions

Introduction (1/4) Video surveillance systems rely on the ability to detect moving objects in the video streams. It should be reliable and effective. unconstrained environments non stationary background different motion patterns…etc

Introduction (2/4) Approaches to characterize the performance of video segmentation: Pixel based methods Template based methods Object based methods Three major drawback: Several types of error should be considered. Some methods are based on the selection with or without persons. It is not possible to define a unique ground truth.

Introduction (3/4) Five segmentation algorithms are considered as examples and evaluated. BBS, W4, SGM, MGM, and LOTS. Several types of errors are considered. Correct Detections, Detection Failures, Splits, Merges, Splits/Merges, and False Alarms. Provide segmentation results of these algorithms on the PETS2001 sequence. We also consider multiple interpretations.

Introduction (4/4) Segmentation Algorithms Proposed Framework (BBS, W4, SGM, MGM, LOTS) Proposed Framework (User Friendly Interface) Segmentation of video images Create the ground truth Performance Evaluation (CD, DF, , Split, Merge, S/M, and FA)

Related Work (1/3) Background subtraction is simple to detect moving objects in video sequences. by comparing the difference with a threshold Several difficulties arise when background image is corrupted by noise. camera movements fluttering objects (e.g., tree waving) illumination changes clouds, shadows

Related Work (2/3) Some works use a deterministic background model. admissible interval for each pixel maximum rate of change in consecutive images, …etc Most works rely on statistical models of the background. Each pixel is a random variable with a probability distribution. e.g., Pfinder system uses a Gaussian Model. mixture of Gaussian Models.

Related Work (3/3) For shadows and non-stationary backgrounds: show changes (e.g., sun motion) and rapid changes (clouds, rain, or abrupt changes…etc) recursively update the background parameters and thresholds Presence of ghosts Static objects suddenly starts to move. Combining background subtraction with frame differencing or by high level operation.

Segmentation Algorithms Basic Background Subtraction W4 detection algorithm used in the W4 system [17] Single Gaussian Model Multiple Gaussian Model Lehigh Omnidirectional Tracking System Used to detect small non cooperative targets [18] [17] “W4: real time surveillance of people and their activities” [18] “Into the woods: Visual surveillance of non-cooperative camouflaged targets in complex outdoor settings”

Segmentation Algorithms BBS: Basic Background Subtraction Computing the difference between the current frame and the background image. Classify each pixel as foreground region if For pixels associated with the same object by connected component analysis (threshold) intensity (current frame, 3x1 vector) mean intensity (background)

Segmentation Algorithms Algorithm used in W4 System modified threshold! designed for grayscale images Three features: Min: minimum intensity Max: maximum intensity D: maximum intensity difference between consecutive frames Classify the pixel I(x,y) as a foreground pixel if modified! [17] W4: real-time surveillance of people and their activities

Segmentation Algorithms SGM: Single Gaussian Model The mean and covariance of each pixel: updated recursively Classify each pixel as active or background α: constant, I(x,y): pixel of the current frame (YUV) If l(x,y) is small, the pixel is classified as active! [1] Pfinder: real-time tracking of the human body

Segmentation Algorithms MGM: Multiple Gaussian Model MGM models each pixel I(x,y) as a mixture of N (N=3) Gaussian Distributions. I(x,y) is a 3x1 vector (R,G,B) The mixture model is dynamically updated. N Gaussian Distributions with respective weights weight: non match components of the mixture are not modified [2] Learning Patterns of Activity Using Real-Time Tracking

Segmentation Algorithms LOTS: Lehigh Omnidirectional Tracking System Use two gray level background images B1, B2. initialized using a set of T consecutive frames Targets are detected using two thresholds high threshold, low threshold User Specified!

Segmentation Algorithms LOTS: Lehigh Omnidirectional Tracking System Each pixel is considered as active if A target is a set of connected active pixel that a subset of them verifies: , high threshold (TH), and low threshold (TL) are updated recursively! [18] Into the woods: Visual surveillance of non-cooperative camouflaged targets in complex outdoor settings

Error Analysis and Classification Proposed Framework Principles: Select a set sequences 1 frame/second Object Detection By Automatic Procedure Manually Correction User Friendly Interface Detection Failure, False Alarm…etc Error Analysis and Classification Statistics Performance Evaluation!

Proposed Framework Interface used to create ground truth manually. output of the detector four active regions 4 false alarms! User can easily edit it and provide the correct segmentation!

Proposed Framework Compare the output of the algorithm with the ground truth segmentation. Region Matching Region Overlap Area Matching Multiple Interpretation

Proposed Framework Several cases are considered: Correct Detection (CD): 1-1 match False Alarm (FA): 0-1 match Detection Failure (DF): 1-0 match Merge Region (M): many-1 match Split Region (S): 1-many match Split-Merge Region (SM) Correspondence: ground truth – detector output

Proposed Framework Region Matching Binary Correspondence Matrix: Defines correspondence between active regions.

Different matching cases: Correct Detection Detection Failure! False Alarm! Merge Split Split-Merge

Proposed Framework Region Overlap Overlap Requirement = 20% DF! (Overlap < 20%) CD! (Overlap > 20%) 2 DF! (Overlap < 20%) Split (Overlap > 20%)

Proposed Framework Area Matching higher percentage of match size, better active regions produced by the algorithm. For correctly detected regions, Characterize the performance of the detector!

Proposed Framework Multiple Interpretations Correct Split Example: Should be considered as valid! manual segmentation SGM segmentation

Proposed Framework Multiple Interpretations Wrong Split Example: Wrong Segmentation! manual segmentation W4 segmentation

Proposed Framework Multiple Interpretations different merged regions groups Labeling Matrix M Region Linking Procedure with three objects A, B, C

Tests on PETS2001 Dataset Evaluate several object detection algorithms using PET2001 dataset. Training (3064 frames) and test sequences (2688 frames) are used. The first 100 images were used to build the background model. The algorithm were evaluated using 1 frame/second. Area of 25 pixel was chosen, and overlap requirement is 10%. Ground Truth vs. Detector Output

Tests on PETS2001 Dataset Choice of the Model Parameter (BBS) ROC for different value of α : BBS Performance of BBS is independent of α. α = 0.05 α = 0.1 α = 0.15 T=0.2 is the best value!

Tests on PETS2001 Dataset Choice of the Model Parameter (SGM) ROC for different value of α : SGM see for -400 < T < -150 seems less DF and FA! α = 0.01 α = 0.05 α = 0.15 Choose α = 0.05, T=-400!

Tests on PETS2001 Dataset Choice of the Model Parameter (MGM) ROC for different value of α : MGM Performance of MGM is strongly depend on the value of T. α = 0.008 α = 0.01 α = 0.05 Choose α = 0.008, T > 0.9!

Tests on PETS2001 Dataset Choice of the Model Parameter (LOTS) ROC for different background update rate : LOTS variation of sensitivity from 10% to 110% Background update at every: 1024th Frame 256th Frame 128th Frame

Tests on PETS2001 Dataset Performance Evaluation (Case I.) Performance of five object detection algorithms If a moving object stops and remains still, it is considered an active region.

Tests on PETS2001 Dataset Performance Evaluation (Case II.) Performance of five object detection algorithms If a moving object stops and remains still, it is integrated in the background after one minute.

Tests on PETS2001 Dataset Complexity vs. Performance by Appendix BBS, LOTS, W4, SGM have a similar computational complexity. MGM requires higher computational cost! MGM requires higher complexity, but the performance is not as good as the LOTS and SGM.

Conclusion This paper proposes a framework for the evaluation of object detection algorithms. Detector Output vs. Ground Truth consider multiple interpretations Measuring the percentage of each type of error. The best results were achieved by the LOTS and SGM algorithms.