1. Video Introduction Background Models Tracking Performance
Static Background, Frame difference, Running Average, Selectivity, Median, Running Gaussian Average, GMM
Shadow Detection
Tracking
Exhaustive Search, Mean Shift, Optical Flow, Feature Point Tracking
Performance
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

2. Introduction – Objects of interest
Reliably detecting moving objects of interest in a scene.
Motion detection
Moving object detection & location
Derivation of 3D object properties
When is an object of interest?
Size
Max and min velocity and acceleration
Assumptions:
Mutual correspondence
Common motion
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

3. Introduction – Common problems
Illumination & appearance changes
Gradual (e.g. time of day)
Sudden (e.g. clouds, lights)
Shadows
Weather (e.g. rain, snow)
Background changes
Objects becoming part of the background
Objects leaving the background
Background objects oscillating slightly
Setup
Camera motion
Frame rate
Field of view
Distance to objects
Location of camera
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

4. Introduction – Difference images
Image subtraction ..
Colour image version:
Process each channel separately?
Just process hue?
False Positives
False Negatives
The threshold (ε) may be…
Too low or too high
Just right?
High contrast
© Reproduced by permission of
Dr. James Ferryman, University of Reading
absdiff( frame, background, difference );
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

5. Background Models – Static Background
| fn(i,j) - b(i,j) | > T
Simplest approach
Sensitive to the Threshold T
absdiff( current_frame, first_frame, difference );
cvtColor( difference, moving_points, CV_BGR2GRAY );
threshold( moving_points, moving_points, 30, 255,
THRESH_BINARY );
Mat display_image = Mat::zeros( moving_points.size(),
CV_8UC3 );
current_frame.copyTo( display_image, moving_points );
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

6. Background Models – Static Background
| fn(i,j) - b(i,j) | > T
Background b(i,j) = first frame?
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

7. Background Models – Frame Difference
| fn(i,j) – fn-1(i,j) | > Threshold
Depends on the speed of the objects and the frame rate
Sensitive to the Threshold
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

8. Background Models – Running Average
Adapts to changing scene
Incorporates moving objects…
© Reproduced by permission of Dr. James Ferryman, University of Reading
accumulateWeighted( current_frame_gray,
running_average_background, 0.01 );
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

9. Background Models – Selectivity
Selectivity (with running average)
Avoids incorporating moving objects
Adapts to scene changes?
© Reproduced by permission of Dr. James Ferryman, University of Reading
accumulateWeighted( current_frame_gray,
running_average_background, 0.01, foreground_mask );
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

10. Background Models – Colour
Running Average
Selectivity
Selectivity (with
running average)
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

11. Background Models – Colour
vector<Mat> input_channels(3);
split(current_frame,input_channels);
vector<Mat> channels(3);
background.convertTo(temp_background,CV_8U);
absdiff(temp_background,current_frame,difference);
split(difference,channels);
temp_sum = (channels[0] + channels[1] + channels[2])/3;
threshold(temp_sum,foreground,30,255,THRESH_BINARY_INV);
split(background,channels);
accumulateWeighted(input_channels[0], channels[0], alpha, foreground);
accumulateWeighted(input_channels[1], channels[1], alpha, foreground);
accumulateWeighted(input_channels[2], channels[2], alpha, foreground);
invertImage(foreground,foreground);
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

12. Background Models – Colour
if (update_foregrond) {
accumulateWeighted(input_channels[0], channels[0], alpha /3.0,foreground);
accumulateWeighted(input_channels[1], channels[1], alpha /3.0,foreground);
accumulateWeighted(input_channels[2], channels[2], alpha /3.0,foreground); }
merge(channels,background);
if (clean_binary_images)
morphologyEx(foreground, closed_image, MORPH_CLOSE, Mat());
morphologyEx(closed_image,f oreground, MORPH_OPEN, Mat());
foreground_image.setTo(Scalar(0,0,0));
current_frame.copyTo(foreground_image, foreground);
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

13. Background Models – Selectivity
Selectivity (with running average)
Reduce the incorporation of moving objects
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

14. Background Models – Median
Median: Middle value (from an ordered list)
No. of frames (m)
Histogram quantisation?
Computational expense
Adding, storing and removing frames
Change in median can be tracked inexpensively from frame to frame
Can be approximated using aging
Can also use selective update
Could use the Mode instead… (Most common value)
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

15. Background Models – Median Results
Learning Rate = 1.001
Learning Rate = 1.005
Learning Rate = 1.02
© Original Video reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

16. Background Models – Median Algorithm
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

17. Background Models – Running Gaussian Average
Fit a Gaussian distribution over the histogram
Background PDF (μ, σ)
Update using running average
Foreground test
Threshold is a multiple of the standard deviation
Can also use selective update
e.g. Pfinder (Wren et al. 1997)
Problems?
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

18. Background Models – Gaussian Mixture Model
How to deal with multi-modal background pixels
e.g. from trees, water
Stauffer & Grimson, 2000
Algorithm presented is based on that in Sonka (3rd edition) pp
Model multiple values (3-5) at each point.
Unsupervised learning…
Most popular method for background modelling
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

19. Background Models – GMM Model details
Model each pixel fn(i,j) using k Gaussian distributions Nk = N(μk, Σk) πn μn Σn
Covariance between two channels a and b for pixel (i,j) at frame t:
Σt ((X t(i,j,a) - μk(i,j,a)) * ((Xt(i,j,b) - μk(i,j,b))
Approximate using σ2k
Assume different components independent and of equal variance
Set a learning constant α
in the range 0.01 … 0.1
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

20. Background Models – GMM Model Update
For a new sample fn(i,j)
Select the best close Gaussian distribution
close = within 2.5 σn (i,j,m) of μn (i,j,m)
If there is a best close Gaussian l
If there is no close Gaussian (replace one…)
x = argmink(πn(i,j,k))
μn+1(i,j,x) = fn(i,j)
σ 2n+1(i,j,x) = 2.maxk σ 2n(i,j,k)
πn+1(i,j,x) = ½.mink πn(i,j,k)
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

21. Background Models – GMM Moving Points
Identifying background distributions
Define T, a proportion of the frames in which background pixels should be visible.
Order the Gaussians by πn+1(i,j,k) / σ n+1(i,j,k)
Gaussians 1..B are considered background where
B = argminb( (Σb=1..k πn+1(i,j,k) ) > T )
Just check if best close Gaussian (or the new Gaussian distribution) is a background distribution
Finally use morphological dilations and erosions to remove small regions and fill in holes.
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

22. Background Models – Shadow Detection
Compare current frame with background image… (Prati 2003)
Intensity / luminance / value drops
Intensity / luminance / value has not dropped too much (below λ)
Saturation does not increase too much
Hue does not change too much
Hue unpredictable & Change can decrease a little (Tattersall 2003)
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

23. Background Models – Shadow Detection
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

24. Background Models – Shadow Detection – Code
BackgroundSubtractorMOG2 gmm;
gmm( current_frame, foreground_mask );
threshold( foreground_mask, moving_points, 150, 255,
THRESH_BINARY ); …
threshold( foreground_mask, changing_points, 50, 255,
absdiff( moving_points, changing_points, shadow_points );
Mat mean_background_image;
gmm.getBackgroundImage( mean_background_image );
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

25. Tracking – Introduction
Used in video surveillance, sports video analysis, vehicle guidance systems, etc.
A hard task because objects
may be undergoing complex motion
may change shape
may be occluded
may change appearance due to lighting/weather
may physically change appearance
Approaches considered:
Exhaustive search
Mean Shift
Optical Flow
Feature based tracking
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

26. Tracking – Exhaustive Search
Extract object to be tracked from frame
Compare in all possible positions in future frame(s)
Use a similarity metric
E.g. normalised cross correlation
Pick the best match
Need extra degrees of freedom for scale and orientation.
May fail if object motion is too complex.
Template matching and chamfer matching support this type of tracking.
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

27. Tracking – Mean Shift
Back-projects a histogram of the object into the current frame
Searches for a region of the same size within the back projection looking for the highest (weighted) sum.
Uses hill climbing to iteratively look for the (local) maximum
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

28. Tracking – Mean Shift Rect position(starting_position);
TermCriteria criteria( cv::TermCriteria::MAX_ITER, 5, 0.01);
meanShift( back_projection_probabilities, position, criteria);
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

29. Tracking – Dense Optical Flow
Compute a motion field (known as optical flow) for the entire image
Direction & Magnitude
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

30. Tracking – Dense Optical Flow
Based on the brightness constancy constraint
Object points will have the same brightness over a short period of time
Need to find the displacement which will minimise the residual error
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

31. Tracking – Dense Optical Flow
To compute the optical flow , assuming that the displacement is small…
Hence (given our previous equation So
And reorganising
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

32. Tracking – Dense Optical Flow – Code
calcOpticalFlowFarneback (previous_gray_frame, gray_frame,
optical_flow, 0.5, 3, 15, 3, 5, 1.2, 0);
cvtColor (previous_gray_frame, display, CV_GRAY2BGR);
for (int row = 4; row < display.rows; row+=8)
for(int column = 4; column < display.cols; column+8) {
Point2f& flow = optical_flow.at<Point2f>(row,column);
line (display, Point(column,row), Point(
cvRound(column+flow.x), cvRound(row+flow.y)),
passed_colour); }
gmm.getBackgroundImage( mean_background_image );
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

33. Tracking – Feature-based Optical Flow
We cannot accurately compute optical flow for constant regions or along edges.
Often better to compute optical flow just for features… (e.g. Lucas Kanade feature tracker)…
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

34. Performance – Ground Truth Formats
Label pixel masks
Label all pixels corresponding to objects of interest (e.g. moving people).
Very time consuming.
Very hard to get agreement particularly near boundaries.
Bounding Boxes
Put a box around all objects of interest.
Much faster to do.
Much more inaccurate (e.g. size of bounding box vs. area of object).
Labelled events
Simply label a frame or range of frames to indicate when some event has taken place (e.g. car has stopped).
Much easier to compute.
May need a bounding box or labelled pixels as well for the frame.
Quite hard to agree on what frame an event occurs in.
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

35. Performance – Metrics For labelled pixel masks or labelled events
we can use metrics like Precision, Recall, F1 measure, etc.
For bounding boxes we must use other metrics such as
Dice coefficient
Overlap
Percentage of Lost Tracks
Lost Track = Overlap falls below a threshold
Threshold = 10% or 20% ???
Video
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014