Presentation is loading. Please wait.

Presentation is loading. Please wait.

Relations as Context to improve Multi-Target Tracking and Activity Recognition. Cristina Manfredotti 12, Enza Messina 1, David Fleet 2 1 DISCo, University.

Similar presentations


Presentation on theme: "Relations as Context to improve Multi-Target Tracking and Activity Recognition. Cristina Manfredotti 12, Enza Messina 1, David Fleet 2 1 DISCo, University."— Presentation transcript:

1 Relations as Context to improve Multi-Target Tracking and Activity Recognition. Cristina Manfredotti 12, Enza Messina 1, David Fleet 2 1 DISCo, University of Milano-Bicocca 2 Computer Science Dept, University of Toronto

2 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 2 Relations to improve tracking

3 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 3 Complex activity recognition Y.Ke, R.Sukthankar, M.Hebert; Event Detection in Crowed Videos

4 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 4 Objectives Goals: 1.To model relations and 2.To maintain beliefs over particular relations between objects In order to simultaneously: Improve tracking with informed predictions and Identify complex activities based on observations and prior knowledge

5 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 5 Relational Domain Relational Domain: set of objects characterized by attributes 1 and with relations 1 between them Boat Id color position(t) velocity(t) direction(t) DecreasingVelocity(t) A SameDirection(t) distance(t) Before(t) Boat B Id color position(t) velocity(t) direction(t) DecreasingVelocity(t) SameDirection(t) distance(t) Before(t) 1 Attributes and relations are predicate in FOL.

6 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 6 A Parenthesis: To model uncertainty in a Relational Domain we will use Relational (Dynamic) Bayesian Networks

7 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 7 BN: the Alarm example

8 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 8 Thanks to Mark Chavira A large BN

9 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 9 The Alarm Relational Domain Relational Domain contains a set of objects with relations between them Objects e.g.: Relation neighbor alarm burglar toCall (the howner of the house) toHear (the alarm) neighbors attributes: capacity of hearing, attention,... alarms attributes: its volume, its sensibility,...

10 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Alarm RBN: Alarm.Volume NeighborCalls Earthquacke Neigh.DegOfDef Neigh.NoiseAround

11 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Closing the parenthesis... Syntax RBN: –a set of nodes, one for each variable –a directed, acyclic graph –a conditional distribution for each node given its parents This distribution must take into account the actual complexity of the nodes! Syntax RBN: –a set of nodes, one for each predicate –a directed, graph –a conditional distribution for each node given its parents,

12 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Relational State The State of a Relational Domain is the set of the predicates that are true in the Domain. Relational state State of attributes State of relations

13 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Dynamics The State of a Relational Domain is the set of the predicates that are true in the Domain. State evolves with time We extend a RBN to a RDBN as we are used to extend a BN to a DBN.

14 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Inference Markov assumption and Conditional independence of data on state. bel(s t ) = ® p(z t |s t ) s p(s t |s t-1 )bel(s t-1 )ds t-1 Bayesian Filter The problem of computing: bel(s t ) = p(s t |z 1:t )

15 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept p(x t-1 |z 1:t-1 )p(x t |z 1:t-1 )p(x t |z 1:t ) Bel(x t-1 ) Bel(x t ) Transition model Sensor model t = t+1 ~ Transition model Sensor model Inference Relations in the State result in correlating the State of different objects between them p(x t-1 |z 1:t-1 )p(x t |z 1:t-1 )p(x t |z 1:t ) Bel(x t-1 ) Bel(x t ) Transition model Sensor model t = t+1

16 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Sensor model (1 st assumption) part of the state relative to relations, s r, not directly observable p(z t |s t ) = p(z t |s a t ) observation z t independent by the relations between objects. Intuitively: Travelling Together vs Being Close

17 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Transition model: a trick p(s t |s t-1 ) = p(s a t,s r t |s a t-1, s r t-1 ) S a t-1 S r t-1 SatSat SrtSrt Intuitive

18 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept p(s a t,s r t |s a t-1,s r t-1 )= But s r t independent by s a t-1 given s r t-1 and s a t p(s a t,s r t |s a t-1,s r t-1 ) = p(s a t |s a t-1,s r t-1 ) p(s r t |s r t-1, s a t ) bel(s t ) = p(s t |z 1:t ) = p(s a t,s r t |z 1:t ) bel(s t )=αp(z t |s a t,s r t ) s p(s a t,s r t |s a t-1,s r t-1 )bel(s t-1 )ds t-1 p(z t |s a t,s r t ) = p(z t |s a t ) Relational Inference p(s a t |s a t-1,s r t-1 ) p(s r t |s a t-1,s r t-1, s a t ) Transition model (2 nd assumption)

19 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Conditional Probability Distribution FOPT: a Probabilistic Tree whose nodes are FOL formulas CPD relation t (x,y): relation t-1 (x,y) p(relation t (x,y)) x t, y t CPD y t : x, relation t-1 (x,y) p(y t |y t-1 ) T F p(y t |y t-1 ) p(x t |x t-1,y t-1,r t-1 )

20 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept * It is a technique that implements a recursive Bayesian Filter through a Monte Carlo simulation. The key idea is to represent the posterior pdf as a set of samples (particles) paired with weights and to filter the mesurament based on these weights.. Particle Filtering* (general case)

21 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Relational Particle Filter

22 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept RPF: extraction X a t,(m) X r t,(m) X a t,(m) ~ p(x a t,(m) |s a t-1,s r t-1 ) X a t,(m) ~ p(x r t,(m) |s a t = x a t,(m),s r t-1 ) X r t,(m)

23 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept RPF: weighting The consistency of the probability function ensures the convergence of the algorithm. X a t,(m) X r t,(m) Weight ( ) ~p(z t |x a t ) The weighting step is done according to the attributes part of each particle only, the relational part follows.

24 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Tracking AND activity Recognition X a t,(m) X r t,(m) X a t,(m) X r t,(m) X a t,(m) X a {t,(m)} X o {t,(m)} X r t,(m) X a t+1,(m) 1° step of sampling: prediction of the state of attributes X a t,(m) X a {t,(m)} X o {t,(m)} X r t,(m) X a t+1,(m) X a {t,(m)} X o {t,(m)} X r t+1,(m) 2° step of sampling: prediction of the state of relations Or activity prediction

25 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Canadian Harbor: rendezvous Same speed

26 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Canadian Harbor: Avoidance Constant speed

27 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Exp: attributestransition

28 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Exp: relationstransition

29 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Exp: Results methodTP ratioTN ratio Mean Tracking Error (km) RPF PF random

30 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept To conclude... Modeling Relations dynamically: –To improve multi target tracking –To recognize complex activities Inference in Dynamic Relational Domain – In theory complex BUT – Simplified by smart decomposition of the transition model non-relational sensor model Results are promising

31 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept BNs: a drawback Each node is a variable: Two different nodes If we would have 4 neighbors? We have to construct a graph with 2 more nodes.

32 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Related works Complex tracking tasks: –Heuristics M.Isard and J. MacCormick BraMBLe, A Bayesian Multi-Blob Tracker.

33 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Related works Complex tracking tasks: –Heuristics –Mixed-States models M.Isard and A.Blake A mixed-state condensation tracker with automatic model switching

34 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Related works Complex tracking tasks: –Heuristics –Mixed-States models Complex activity recognition: –Stochastic grammar Free Y.A.Ivanov and A.F.Bobick Recognition of Visual Activities and Interactions by Stochastic Parsing

35 C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept Related works Complex tracking tasks: –Heuristics, –Mixed-States models Complex activity recognition: –Stochastic grammar Free, –First Order Logic S. Tran and L. Davis, Visual Event Modeling and Recognition using Markov Logic Networks


Download ppt "Relations as Context to improve Multi-Target Tracking and Activity Recognition. Cristina Manfredotti 12, Enza Messina 1, David Fleet 2 1 DISCo, University."

Similar presentations


Ads by Google