KOUROSH MESHGI PROGRESS REPORT TOPIC To: Ishii Lab Members, Dr. Shin-ichi Maeda, Dr. Shigeuki Oba, And Prof. Shin Ishii 9 MAY 2014

KOUROSH MESHGI – ISHII LAB - DEC SLIDE 2 MAIN APPLICATIONS Surveillance Public Entertainment Robotics Video Indexing Action Recog.

KOUROSH MESHGI – ISHII LAB - DEC SLIDE 3 MAIN CHALLENGES Varying Scale Clutter Deformation Illumination Abrupt Motion

 Goal: Define p(X t |Y 1,…,Y t ) given p(X 1 ) KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 4 States: Target Location and Scale Observations: Sensory Information

PARTICLE FILTER TR.

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 6 Frame: t RGB Domain

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 7 Frame: t Depth Domain

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 8 Frame: t Sensory Information

Observation KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 9 Frame: t State w h (x,y)

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 10 Feature Set Color Shape Edge Texture

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 11 Frame: 1 Template f1f1 fjfj fMfM ……

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 12 Frame: 1 Particles Initialized Overlapped

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 13 Frame: t Motion Model → t + 1

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 14 Frame: t + 1 Feature Vectors f1f1 f2f2 fMfM X 1,t+1 X 2,t+1 X N,t+1 … …

KOUROSH MESHGI – ISHII LAB - MAR SLIDE 15 Frame: t Probability of Observation Each Feature Independence Assumption  !

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 16 Frame: t + 1 Particles Brighter = More Probable

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 17 Frame: t + 1 Feature Vectors

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 18 Frame: t + 1 New Model Model Update

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 19 Frame: t + 1 Proportional to Probability

PARTICLE FILTER TR.

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 21 Appearance ChangesModel DriftDeficient Feature SpaceUninformed SearchOptimized Feature SelectionApproximation of Target

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 22 Same Color Objects Background Clutter Illumination Change Shadows, Shades Use Depth!

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 23 Templates Corrupted! Handle Occlusion! (No Model Update During Them)

* Local Optima of Feature Space * Feature Noise * Feature Failures KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 24 Regularization Non-zero Values Normalization

Particles Converge to Local Optima / Remains The Same Region KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 25 Advanced Motion Models (not always feasible) Restart Tracking (slow occlusion recovery) Expand Search Area!

* The Search is not Directed * Neither of the Channels have Useful Information * Particles Should Scatter Away from Last Known Position KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 26 Occlusion!

do not address occlusion explicitly maintain a large set of hypotheses  computationally expensive direct occlusion detection robust against partial & temp occ.  persistent occ. hinder tracking GENERATIVE MODELS DISCRIMINATIVE MODELS Dynamic Occlusion: Pixels of other object close to camera Scene Occlusion: Still objects are closer to camera than the target object Apparent Occlusion: Due to shape change, silhouette motion, shadows, self-occ UPDATE MODEL FOR TARGET  TYPE OF OCCLUSION IS IMPORTANT  KEEP MEMORY VS. KEEP FOCUS ON THE TARGET Combine Them!

 PTO partial occlusion  SAO self- or articulation occlusion  TFO temporal full occlusion - shorter than 3 frames  PFO persistent full occlusion  CPO complex partial occlusion - including “split and merge” and permanent changes in a key attribute of a part of target  CFO complex full occlusion KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 28

[Zhao & Nevatia, 04] Occlusion Indicator: Ratio of FG/BKG [Wu & Nevatia, 07] Handle Occlusion using Appearance Model [de Villiers et al., 12] Switch Tracker in the case of Occlusion [Song & Xiao, 13] Occlusion Indicator: New Peak in HOD or Reduction of the Size of Main Peak Many other papers handle occlusions as the by- product of their novel trackers

OCCLUSION AWARE PFT

Motion Model Resampling Target Estimation Calculate Likelihood KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 31 Initialization Model Update Observation Occlusion Flag? Constant Likelihood Occlusion Estimation Occlusion Threshold > ? YES NO

 Occlusion Flag (for each particle)  Observation Model  No-Occlusion Particles  Same as Before  Occlusion-Flagged Particles  Uniform Distribution KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 32

 Position Estimation of the Target  Occlusion State for the Next Box KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE a x x

 Model Update (Separately for each Feature)  Modified Dynamics Model of Particle KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 34

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 35 Occlusion!

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 36 Occlusion! GOTCHA!

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 37 Quick Occlusion Recovery  Low CPE No Template Corruption No Attraction to other Object/ Background

COLOR (HOC) TEXTURE (LBP) EDGE (LOG) 2D PROJ. (BETA) 3D SHAPE (PCL Σ ) DEPTH (HOD) GRADIENT (HOG) KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 38

& DISCUSSION

Princeton Tracking Dataset 5 Validation Video with Ground Truth 95 Evaluation Video KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 40

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 41 OAPFT (Proposed, with different feature sets) State-of-the-art RGBD tracker OI+SVM (SVM tracker with Occlusion Indicator) Traditional Particle Filter tracker ACPF (Adaptive Color Particle Filter) State-of-the-art RGB tracker, Successful for Occlusion Handling STRUCK (Structured Output SVM Tracker)

 PASCAL VOC: Overall Performance toto Success Overlap Threshold Area Under Curve KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 42

KOUROSH MESHGI – ISHII LAB - MAR SLIDE Success Plot Overlap Threshold Success Rate 1 1

 Mean Central Point Error: Localization Success  Mean Scale Adaption Error KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 44 EstimatedGround Truth

Center Positioning Error Frames CPE (pixels)

Scale Adaptation Error Frames SAE (pixels)

 FP happens when a tracker doesn’t realize that the target is occluded.  MI happens when the target is visible but the tracker fails to track it as if the target is still in an occlusion state  MT the estimated bounding box has nothing in common with ground truth box  FPS execution time in frames per second KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 47

KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 48 Tracker AUCCPESAEMIFPMTFPS BCDEGST (proposed) ACPF (Nummiaro ‘03) STRUCK (Hare ‘11) OI+SVM (Song ‘13)

KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 49 More Resilient Features + Scale Adaptation Active Occlusion Handling Measure the Confidence of each Data Channel Adaptive Model Update

Q UESTIONS? Thank you for your time…