Background Subtraction for Urban Traffic Monitoring using Webcams Master Graduation Project Final Presentation Supervisor: Rein van den Boomgaard Mark Smids December 12 th 2006

Overview Introduction Background Subtraction Shadow Detection Video Summarization Demo’s Background Subtraction in action Shadow Detector in action Smart Surveillance using Video Summarization Evaluation Conclusions

Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Traditional ways of traffic monitoring using magnetic loops

Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Traditional ways of traffic monitoring using magnetic loops Limitations: These systems only count, very costly

Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Traditional ways of traffic monitoring using magnetic loops Limitations: These systems only count, very costly For extended traffic monitoring we want to measure: road density, queue detection, vehicle speed, exact location of vehicles

Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Traditional ways of traffic monitoring using magnetic loops Limitations: These systems only count, very costly For extended traffic monitoring we want to measure: road density, queue detection, vehicle speed, exact location of vehicles Solution: use cameras to monitor traffic automatically

Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Why focus on an urban setting? Most research focused on a highway setting More challenging tasks

Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Why focus on an urban setting? Most research focused on a highway setting More challenging tasks Components of a vision based traffic monitoring system: cameras, calibration, background subtraction, tracking, shadow detection, parameter extraction, video summarization, …

Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Why focus on an urban setting? Most research focused on a highway setting More challenging tasks Components of a vision based traffic monitoring system: cameras, calibration, background subtraction, tracking, shadow detection, parameter extraction, video summarization, …

Background Subtraction Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Deterministic approach Create an initial background model from the first N frames For each new frame, subtract it from the background model to obtain a binary mask for all x,y: if I(x,y) – B(x,y) > T then M(x,y) = 1 else M(x,y) = 0 Update the background model: for all x,y: if M(x,y) = 0 then B(x,y) = I(x,y)

Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Statistical approach Model each pixel in the background model by a mixture of Gaussians Background Subtraction

Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Statistical approach Model each pixel in the background model by a mixture of Gaussians How to determine those components that model the background? Observation: these Gaussians have the most supporting evidence and lowest variances Order the K distributions in the mixture by the value of The first B distributions are chosen as the background model, where: Background Subtraction

Shadow Detection Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Shadows: cast and self shadows

Shadow Detection Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Shadows: cast and self shadows Elimination of cast shadows can improve background subtraction results very much…

Shadow Detection Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Shadows: cast and self shadows Elimination of cast shadows can improve background subtraction results very much…

Shadow Detection Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Consider the set of pixels classified as foreground pixels

Shadow Detection Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Consider the set of pixels classified as foreground pixels A pixel is a candidate shadow pixel when the pixel value has a significant lower value than it’s corresponding background value

Shadow Detection Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Consider the set of pixels classified as foreground pixels A pixel is a candidate shadow pixel when the pixel value has a significant lower value than it’s corresponding background value Extend this idea: let c = (R,G,B) and Rate of similarity:

Shadow Detection Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Consider the set of pixels classified as foreground pixels A pixel is a candidate shadow pixel when the pixel value has a significant lower value than it’s corresponding background value Extend this idea: let c = (R,G,B) and Rate of similarity: If tau < < 1 then pixel is a shadow pixel

Video Summarization Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Application: smart vision based surveillance system

Video Summarization Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Application: smart vision based surveillance system Record only frames which includes relevant foreground objects

Video Summarization Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Application: smart vision based surveillance system Record only frames which includes relevant foreground objects How to guarantee that a full trajectory of a vehicle is recorded?

Demos Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions 1.Shadow Detector in action – 1 | Background Subtraction in action det 1 | stat 1 - det 2 | stat 2det 1stat 1det 2stat 2 3.Smart Surveillance using Video Sum. - 11

Evaluation Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Test videos: three different weather conditions (5 minutes each) Goal: test both background subtraction algorithms on these videos Limitation: no ground truth available!

Evaluation Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Evaluation on another level: using the video summarization component. A frame level ground truth is created For each algorithm a score can be computed

Evaluation Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions

Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Score S (deterministic approach) Score S (statistical approach) Total number of Frames Video A (wind/cloudy) 85.6%88.3%4581 Video B (sunny) 88.5%94.6%4163 Video C (rain) 83.4%93.4%3024 Evaluation

Conclusions For all weather conditions: the statistical approach outperforms the deterministic approach (5-10%) Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions

For all weather conditions: the statistical approach outperforms the deterministic approach (5-10%) Wind is the hardest problem from both algorithms Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions

For all weather conditions: the statistical approach outperforms the deterministic approach (5-10%) Wind is the hardest problem from both algorithms Statistical approach performs much better in the sunny settings Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions

For all weather conditions: the statistical approach outperforms the deterministic approach (5-10%) Wind is the hardest problem from both algorithms Statistical approach performs much better in the sunny settings Future work: create a pixel-level ground truth and evaluate both algorithms Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions

Questions? Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions Questions?

MoG details Update Equations: Z. Zivkovic, “Improved Adaptive Gaussian Mixture Model for Background Subtraction” MoG : Introduction Background Subtraction Shadow Detection Video Summarization Demos Evaluation Conclusions