1. Visual Tracking CMPUT 615 Nilanjan Ray

2. What is Visual Tracking Following objects through image sequences or videos Sometimes we need to track a single object, sometimes a number of them Sometimes we just track the object centroid, sometimes entire object boundary

3. Theoretical Foundation Visual tracking is a “state” estimation problem Bayesian inference is at the heart of visual tracking; it is called sequential Bayesian estimation We form the posterior probability of the state, given all evidence or measurements up to the current time point Inference is performed from the posterior density

4. Setting The Stage Some notations: X t : unknown state we want to estimate at time point t; e.g., object centroid Z t : Measurement/observation made at time point t; e.g., image intensities The sequential estimation model assumes that we know the three probability densities: p(X 0 ): The initial state density p(X t |X t-1 ): State transition density or motion model p(Z t |X t ): Measurement/observation/likelihood density

5. Sequential Bayesian Estimation We want to recursively estimate the state X t given the observations Z 1:t = {Z 1, Z 2, …, Z t }

6. Sequential Bayesian Estimation… Filter: Bayes’ Rule: Current posterior Previous posteriorPrediction Likelihood/observation density

7. Bayes’ Rule Derivation Conditional probability rule Marginal density rule Also, because measurement Z t is conditionally independent on the current state X t : So, we have the sequential Bayes’ rule:

8. Filter Derivation Rule of marginal density Rule of conditional probability Also, note that X t is conditionally independent X t-1 (Markovianity), so: Thus we have the filter rule:

9. Important Assumptions Observation is conditionally independent on the current state Current state is conditionally independent on the immediate previous state

10. Computation Theory is all good, however we need to show people that it works in practice… We will study Particle filter is the framework that can compute the recursive state estimation, i.e., sequential Bayesian filter We will also study Kalman filter, popular sequential state estimation with some more assumptions

11. What is a Particle Filter? Let the particles represent the previous density So, the filter step is now: And the Bayes’ rule is now: We need to generate the current particle set from p(X t |Z t ): Particle filter

12. Factored Sampling Let h(x) = f(x)g(x) is a product of two functions, where say, g(x) is a density and f(x) is another non-negative function Factored sampling says that to represent h(x) non-parametrically by a set of particles, generate samples from g(x) and assign weights by f(x) i.e., {(s 1, w 1 ), …, (s n, w n )}, where s i are generated from g(x) and w i = f(s i ). This is closely related to another sampling method called important sampling.

13. Conditional Density Propagation (CONDENSATION) This a product of two functions: (1) and (2) Following the principle of factored sampling, CONDENSATION generates samples from (1) And assigns weight using (2)

14. Samples From a Mixture Density Notice that is a mixture density To generate samples from the mixture density these two steps are followed:

15. CONDENSATION Algorithm

16. How to estimate the state? OK, we generated samples, what do we d with them: estimate the current state: h is any function of the state, for example when h(x) = x the state estimation is done

17. Other PFs To date lots of particle filters have been proposed: – Sequential importance re-sampling (SIR) – Auxiliary particle filter (APF) – Likelihood particle filter – Rao-Blackwellized particle filter – A ton others Leading researchers in PF : Arnoud Doucet et al.

18. Some Points to Ponder about PF The good point about PF is that it can handle very general likelihood and motion models PF inherits a serious shortcoming from non- parametric density representation – curse of dimensionality – when the state space x is large, for example large multiple number of objects etc.