Nathan Johnson1 Background Subtraction Various Methods for Different Inputs

Nathan Johnson2 Purpose of Background Subtraction Reduce problem set for further processing Only process part of picture that contains the relevant information Segment the image into foreground and background Add a virtual background

Nathan Johnson3 Encountered Problems Lighting Shadows Gradual/Sudden illumination changes Camouflage Moving objects Foreground aperture Foreground object becomes motionless Bootstrapping

Nathan Johnson4 Lighting and Shadows Weight the luminance with other characteristics Depth of object Region/Frame information Adjust the background model with time Store a history of previous backgrounds

Nathan Johnson5 Comparison of Two Techniques Wallflower Uses three different components Pixel, Region, and Frame levels Uses many different statistical models to anticipate various changes in the background Gordon, Darrell, Harville, Woodfill Subtraction Two or more cameras to measure distances Uses distance to determine foreground and falls back on luminance

Nathan Johnson6 Wallflower Method – Pixel Level Makes initial judgment whether a pixel is in the foreground Handles background model adaptation Addresses many of the classical problems Moved objects Time of day Camouflage Bootstrapping

Nathan Johnson7 Wallflower – Region & Frame Region level Refines the pixel level judgment Handles foreground aperture problem Frame level Sudden frame level change Uses previous models to figure out what caused the sudden change Light switching on/off

Images from Wallflower: Principles and Practice of Background Maintenance, Kentaro Toyama, John Krumm, Barry Brumitt, Brian Meyers 8 Results using Wallflower

Nathan Johnson9 Gordon, et al. Method Correctly identifies background depth and color when it is represented in a minority of the frames Addition of range solves many of the classic problems Shadows Bootstrapping Foreground object becomes motionless

Nathan Johnson10 Obtaining Initial Background Model Records the (R,G,B,Z) values at each pixel Attempts to determine background through the observed depth Marks a pixel as invalid if there is not enough information for the range valid pixel – range determines whether the pixel is in the background, without the aid of the (R,G,B) values invalid pixel – fall back on classic methods for background subtraction

Nathan Johnson11 Gordon, et al. Method (cont.) r m is invalid r i is valid and smoothly connected to regions with valid background data then a foreground decision can be made Solves the problem of the background being the same depth as part of the foreground Z-keying* methods fail in these cases *Kanade, Yoshida, Oda, Kano, and Tanaka, “A Video-Rate Stereo Machine and Its New Applications”, Computer Vision and Pattern Recognition Conference, San Francisco, CA, 1996.

Nathan Johnson12 Gordon, et al. Method (cont.) YValid(Y m ) = Y > Y min Shadows have a stronger effect on luminance than inter-reflections Separate ratio limits for shadows and reflections

Images from Background estimation and removal based on range and color, G. Gordon, T.Darrell, M. Harville, J. Woodfill 13 Problems Using Only Range or Color

Nathan Johnson14 Which is better? Wallflower over Gordon, et al. Doesn’t require extra cameras to record depth Gordon, et al. produces a “halo” around foreground objects Gordon, et al. over Wallflower Handles more problems Tree waving Bootstrapping

Nathan Johnson15 Other Innovative Methods Fast, Lighting Independent Background Subtraction Advantages Light has no basis on the decision of foreground Disadvantages Requires a known, static background Multiple cameras

Nathan Johnson16 Which Method to Use Type of background present Static or Dynamic Lighting Gradual/Sudden changes Lack of lighting Hardware used during recording Multiple cameras Speed required for application

Nathan Johnson17 Conclusion Record as much information as possible Background subtraction methods have mainly been looked at in particular situations Severe case: Fast, Lighting Independent Method A method to use in every case is still being researched Currently combinations of previously released methods offer the best results for background subtraction