Learning Image Statistics for Bayesian Tracking Hedvig Sidenbladh KTH, Sweden Michael Black Brown University, RI, USA Learning Image Statistics for Bayesian Tracking Hedvig Sidenbladh KTH, Sweden Michael Black Brown University, RI, USA 2. Why is it Hard? People have different appearance due to clothing, shape etc. Need to discriminate between “people” and “non-people”, but allow for variation in human appearance Occlusion 2D-3D projection ambiguities Unexpected position Similarity between different limbs 1. Introduction Goal: Tracking and reconstruction of people in 3D from a monocular video sequence. Articulated human model. Approach: Learn models of the appearance of humans in image sequences. What are the image statistics of human limbs? How do they differ for general scenes? 3. Previous Approaches Edge Detection: Probabilistic model? Under/over-segmentation, thresholds,... Image flow, templates, color, stereo,... Instead, model the appearance of people in terms of filter responses 4. Key idea #1 Steered edge responses Use the 3D model to predict the location of limb boundaries in the scene. Compute various filter responses steered to the predicted orientation of the limb. Compute likelihood of filter responses given 3D model, using a statistical model learned from examples. 10. Key idea #2 ForegroundBackground “Explain” the entire image Konishi et al. 00: p(image | foreground,background) p(image | foreground,background) = p(fgr part of image | background) p(fgr part of image | foreground) const  14. Related Work Statistics of natural images: S. Konishi, A. Yuille, J. Coughlan, and S. Zhu. Fundamental bounds on edge detection: An information theoretic evaluation of different edge cues. submitted: PAMI T. Lindeberg. Edge detection and ridge detection with automatic scale selection. IJCV, 30(2): , 1998 J. Sullivan, A. Blake, and J. Rittscher. Statistical foreground modelling for object localization. In ECCV, pp , 2000 S. Zhu and D. Mumford. Prior learning and Gibbs reaction-diffusion. PAMI, 19(11), 1997 Tracking: J. Deutscher, A. Blake, and I. Reid. Articulated motion capture by annealed particle filtering. In CVPR, vol 2, pp , 2000 M. Isard and J. MacCormick. BraMBLe: A Bayesian multi-blob tracker. In ICCV, 2001 M. Isard and A. Blake. Contour tracking by stochastic propagation of conditional density. In ECCV, pp , Conclusions Generic, learned, model of appearance Combines multiple cues, exploits work on image statistics Use the 3D model to predict position and orientation of features Model of foreground and background Exploits the ratio between foreground and background likelihood, improves tracking 5. Learning Distributions Training data: ForegroundBackground Statistical differences. P on /P off can be used for discriminating foreground from background. P on = probability of filter response on limbs P off = probability of filter response on general background Similar to Konishi et al. 00, but object specific - responses may be present or not. 6. Normalized Derivatives Image derivatives f x, f y, f xx, f xy, f yy normalized with a non-linear transfer function. High  1, low  0, Vertical derivative Normalized vertical derivative Parameters of transfer function optimized to maximize Bhattacharyya distance between P on and P off.: 7. Edge Steered edge response: P on /P off P on, P off 8. Ridge Scale specific! Steered ridge response: P on /P off P on, P off |2 nd derivative|  arm |2 nd derivative| // arm 9. Motion Motion response = I(x, t+1) - I(x+u, t) xx+u Training data: Consecutive images from sequence P on P off 11. Bayesian Formulation Posterior propagated with Condensation (Isard & Blake 96) Likelihood independence assumption: Posterior Likelihood Temporal prior 12. Results Last frame, EdgeRidgeMotionAll Tracking with self occlusion (see video) Tracking is improved using multiple cues (see video) Last frame