1. Multi Object Tracking using Data Association
Ali Taalimi

2. Outline Problem Definition and Formulation Using Linear Programming
Using K Short Path
Using Network Flow
Using Energy Minimization

3. Definition
Data association is the process of partitioning a set of observations into trajectories.

4. Problem Formulation Assumptions: Graph formation:
sequence of k image frames where detections have been observed.
Each frame f = 1,…, k has nf observations, labeled by an augmented index set If
index 0 is a virtual/dummy index to reason over missed detections
Graph formation:
observations from frames 1 through k form a k-partite graph G = (V;E) with
set of all paths of length k through graph G by:
Each path represents one hypothesized target trajectory, while c(t) is a cost quantifying track quality.
path is written as an ordered list of its incident indices: (i1,i2,…,ik)

5. Problem Formulation Examples:
(0,0,a,b,c) : path that starts at frame 3
(a,b,c,0,0) : path that ends at frame 3
(0,0,a,0,0) : false positive detection in frame 3
(a,b,0,0,c) : missed detections or occlusion

6. Challenges challenges of multi-target tracking:
Track Linking Definition: as a batch process, is a generalization of traditional measurement-to-measurement association: here, the matching involves trajectory segments (tracklets).
challenges of multi-target tracking:
the naturally discrete problem of data association, i.e. assigning image observations to the appropriate target.
the naturally continuous problem of trajectory estimation, i.e. recovering the trajectories of all targets.
Short-term occlusion: easy to re-solve if objects have strong motion patterns.
Long-term occlusion: is difficult since motion information about the objects (i.e. linear dynamics) typically only characterizes them for a short time period.
long-term occlusions typically create “track-lost” or “track-switch” errors.
Linking Scheme: We should have local and global linking strategies to processes the track graph. If occlusion occurs for a short period of time or the object features are sufficiently discriminate, we can use a local linking strategy. If occlusion occurs during a long period of time or object features have to be monitored for several frames (e.g., motion smoothness or periodic motion patterns), we need to use a global

7. Linear Programming
round nodes object’s possible location and appearance states
rectangular nodes: occlusion nodes that provide a node to represent that an object is occluded and does not have a spatial location.
Source/Sink node (diamond nodes): start and end of the object tracking sequence.
A connected set of nodes between a source and sink = spatial trajectory of an object
Nodes in a spatial conflict set correspond to object states occupying the same spatial location.
H. Jiang, “A Linear Programming Approach for Multiple Object Tracking,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.

8. Linear Programming
A cost c(vn,m,i, vn,m+1,j) is assigned to each arc, which indicates the cost of state i at time m and state j at time m+1 being on the trajectory of object n.
g(.) is similarity function.
d(.) is spatial distance
ca,bconst = costs penalizing when an object disappears or reappears.
similarity of target node to the template
appearance similarity of detections in two frames
penalizes large spatial displacement between frames

9. Linear Programming Discrete Optimization:
 keep the spatial layout of objects stable over a short period of time.
pn,m is a 4-element vector of bounding box.
N is the set of neighboring objects

10. Linear Programming
Linear Programming Relaxation: Moving from discrete to continuous space by defining ζ
ζ(n,m,i),(n,m+1,j)=indicate if arc(vn,m,i,vn,m+1,j) is on the path
integer programming if 𝜻= 𝟎,𝟏 , But, it is hard.
K(n,m) = # of nodes for object n at time m.
objects in first frame are in one and only one location.
objects in last frame are in one and only one location.
for each node in each frame  input flow = output flow
yn,m,i = {0, 1}: whether node vn,m,i is on the path of obj n
Each location in each frame can be occupied by at most one object

11. K-Shortest Paths
To model occupancy over time = labeled directed graph with K T vertices, which represents every location at every instant.
ft i,j : discrete variable for the number of objects moving from location i at time t to location j at time t+1 for each edge
mti : discrete variable standing for the number of objects located at i at time t for each vertex. (mti = 𝑘ɛ𝑁(𝑗) ftj,k )
J. Berclaz, Multiple Object Tracking using K-Shortest Paths Optimization. IEEE Transactions on PAMI, 2011

12. K-Shortest Paths
set an upper bound of 1 to the sum of all outgoing flows from a given location
flows have to be nonnegative
all flows departing from vsource eventually end up in vsink
F(feasible map): set of flows ft i,j satisfying (1,2,3,4)
Mit = true presence of an object at location i at time t.

13. K-Shortest Paths Linear Programming Formulation:
constraints of (1), (2), (3), and (4) are expressed as inequalities, to have the linear program in canonical form.

14. K-Shortest Paths Formulation:
Definition: Given a pair of nodes, namely, the source vsource and the sink vsink, in a graph G, the k-shortest paths problem is to find the k paths {p1; ; pk} between these nodes such that the total cost of the paths is minimum.
Assumptions:
directed acyclic graph (DAG)
paths are node-disjoint: two separate paths cannot share the same node(condition2).
node-simple: a path visits every node in the graph at most once (conditions 1, 4).

15. Network flow
Data Association using maximizing the posteriori probability of T given the observation set X
X={xi}={pos,scale,appearance,frame index}
T= {Tk}
Bernoulli distribution for modeling likelihood

16. Network flow Min-cost flow solution:
0-1 indicator variables definition for couple the non-overlap constraints with the objective function

17. Network flow Network Optimization:
frame1
frame2
frame3 X3 X6 X1 X4 X7 X2 X5 X8
Finding optimal association hypothesis T∗ is equivalent to sending the flow from source s to sink t that minimizes the cost.
representation of multiple object tracking as cost-flow network formulation : each flow path can be interpreted as an object trajectory, the amount of the flow sent from s to t is equal to the number of object trajectories
for any observation xi that is more likely to be a true detection (βi < 0.5), the cost Ci of edge (ui, vi) is negative; this allows the optimal cost to become below zero by sending flows through these negative-cost edges.

18. Energy Minimization Problem Formulation:
Given a number of unlabeled object detections (a) and a number of possible trajectory hypotheses (b), method labels all detections (c) and re-estimates the trajectories (d) using an alternating discrete-continuous optimization scheme
identify a set of target trajectories T = {T1, , TN} given the set of target hypotheses D.
So we also need to search for a data association f, which for each detection d ∈ D assigns a label fd ∈ L = {1, , N} ∪ ∅.
Eventually, perform multi-target tracking by minimizing a joint energy E(T , f) w.r.t. the trajectories T and the data association f.
Given the set of target hypotheses D, our goal is to identify
a set of target trajectories T = {T1, , TN}. This
implies that we also need to search for a data association
f, which for each detection d ∈ D assigns a label
fd ∈ L = {1, , N} ∪ ∅. Thereby a detection is either
identified as belonging to one of the trajectories or, using
the additional outlier label ∅, identified as a false alarm.
We eventually aim to perform multi-target tracking by minimizing
a joint energy E(T , f) w.r.t. the trajectories T and
the data association f. To ease understanding, we first introduce
the component energies and unify them later.
Discrete-Continuous Optimization for Multi Target Tracking, CVPR, 2012

19. Energy Minimization Continuous trajectory model:
represent individual trajectories in continuous space using cubic B-splines.
Why? avoiding discretization artifacts and good trade-off between model flexibility and intrinsic motion smoothness.
Given a data association f, trajectory estimation problem as minimization of the energy will be:
aim to minimize the weighted Euclidean distance to each assigned target hypothesis in all valid frames:
A low confidence score of the object detector usually means one of two things: either the output is a false alarm, or the bounding box is not properly aligned with the object.

20. Energy Minimization Discrete data association: Problem Definition:
Formulate data association as a multi-labeling problem
Problem Definition:
Aim to estimate a labeling f that uniquely assigns each detection d ∈ D to one of the N trajectory hypotheses T = {T1, , TN}, or identifies it as a false alarm (label ∅).
Making Graph G(D,E):
D=Vertex(all detections)
E=Edges(nearby detections in sequential frames)
The motivation for E (Edges) is that nearby detections in adjacent
frames should be encouraged to have the same trajectory
label.
Neighborhood structure of the underlying pairwise Markov random field. Detections in adjacent frames are connected if their distance is below a certain threshold.

21. Energy Minimization Discrete Pairwise MRF Energy:
Discrete-continuous tracking with label costs: