1. Kourosh MESHGI Shin-ichi MAEDA Shigeyuki OBA Shin ISHII 18 MAR 2014 Integrated System Biology Lab (Ishii Lab) Graduate School of Informatics Kyoto University meshgi-k@sys.i.kyoto-u.ac.jp IEICE NC Tamagawa’14

2. KOUROSH MESHGI – ISHII LAB - DEC 2013 - SLIDE 2 MAIN APPLICATIONS Surveillance Public Entertainment Robotics Video Indexing Action Recog.

3. KOUROSH MESHGI – ISHII LAB - DEC 2013 - SLIDE 3 MAIN CHALLENGES Varying Scale Clutter Non-Rigid Illumination Abrupt Motion

4. [Mihaylova et al., 07] RGB Color + Texture + Motion + Edge, Two PF [Spinello & Arras,11] HOG on RGB + HOG on Depth, SVM Classification [Shotton et al, 11] Skeleton from Depth, Random Forrest [Choi et al, 11] Ensemble of Detectors (upper body, face, skin, shape from depth, motion from depth), RJ-MCMC [Song et al., 13] 2.5D Shape + Motion + HOG on Color and Depth, Occlusion Indicator, SVM

5. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 5 Frame: t Observation

6. Image Patch KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 6 Frame: t State w h (x,y)

7. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 7 Feature Set Color Shape Edge Texture

8. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 8 Frame: 1 Template f1f1 fjfj fnfn ……

9. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 9 Frame: 1 Particles Initialized Overlapped

10. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 10 Frame: t Motion Model → t + 1

11. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 11 Frame: t + 1 Feature Vectors f1f1 f2f2 fnfn X 1,t+1 X 2,t+1 X N,t+1 … …

12. KOUROSH MESHGI – ISHII LAB - MAR 2014- SLIDE 12 Frame: t Probability of Observation Each Feature Independence Assumption  !

13. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 13 Frame: t + 1 Particles Brighter = More Probable

14. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 14 Frame: t + 1 Expectation

15. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 15 Frame: t + 1 New Model Model Update

16. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 16 Frame: t + 1 Proportional to Probability

17. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 17 Same Color Objects Background Clutter Illumination Change Shadows, Shades Use Depth!

18. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 18 Templates Corrupted! Handle Occlusion!

19. Particles Converge to Local Optima / Remains The Same Region KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 19 Advanced Motion Models (not always feasible) Restart Tracking (slow occlusion recovery) Expand Search Area!

20. do not address occlusion explicitly maintain a large set of hypotheses  computationally expensive direct occlusion detection robust against partial & temp occ.  persistent occ. hinder tracking GENERATIVE MODELS DISCRIMINATIVE MODELS Dynamic Occlusion: Pixels of other object close to camera Scene Occlusion: Still objects are closer to camera than the target object Apparent Occlusion: Due to shape change, silhouette motion, shadows, self-occ UPDATE MODEL FOR TARGET  TYPE OF OCCLUSION IS IMPORTANT  KEEP MEMORY VS. KEEP FOCUS ON THE TARGET Combine Them!

21. * The Search is not Directed * Neither of the Channels have Useful Information * Particles Should Scatter Away from Last Known Position KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 21 Occlusion!

22.  Occlusion Flag (for each particle)  Observation Model  No-Occlusion Particles  Same as Before  Occlusion-Flagged Particles  Uniform Distribution KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 22

23.  Probability of Occlusion for the Next Box  Modified Dynamics Model of Particle KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 23

24.  Model Update  Separately for each Feature KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 24

25. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 25 Occlusion!

26. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 26 Occlusion! GOTCHA!

27. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 27 Quick Occlusion Recovery  Low CPE No Template Corruption No Attraction to other Object/ Background

28. COLOR (HOC) TEXTURE (LBP) EDGE (LOG) DEPTH (HOD) 3D SHAPE (PCL Σ ) KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 28

29. Princeton Tracking Dataset 5 Validation Video with Ground Truth 95 Evaluation Video KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 29

30. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 30 A. Edge PFB. Edge + Color PFC. Edge + Color + Depth PFD. Edge + Color + Depth + Texture PFE. Edge + Color + Depth + Texture + 3D Shape PFF. Occlusion Aware PF

31. (Yellow Dashed Line is Ground Truth)

32.  PASCAL VOC toto Success Overlap Threshold 0 1 1 Area Under Curve KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 32

33. KOUROSH MESHGI – ISHII LAB - MAR 2014- SLIDE 33 Success Plot A D B E C F 1 1 Overlap Threshold Success Rate

34.  Mean Central Point Error: Localization Success  Mean Scale Adaption Error KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 34 EstimatedGround Truth

35. KOUROSH MESHGI – ISHII LAB - MAR 2014- SLIDE 35 Center Positioning Error A D B E C F 100 50 Frames CPE (pixels)

36. KOUROSH MESHGI – ISHII LAB - MAR 2014- SLIDE 36 Scale Adaptation Error KOUROSH MESHGI – ISHII LAB - MAR 2014- SLIDE 36 140 SAE (pixels) 50 Frames A D B E C F

37. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 37 Tracker AUCCPESAE A (edg) 15.72192.2741.62 B (edg+hoc) 29.8893.4746.71 C (edg+hoc+hod) 46.7434.6240.88 D (edg+hoc+hod+tex) 48.4930.1846.27 E (edg+hoc+hod+tex+shp) 58.0323.8429.62 F (all + occlusion handling) 63.5817.4625.07

38. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 38 More Resilient Features + Scale Adaptation Active Occlusion Handling Measure the Confidence of each Data Channel Adaptive Model Update

39. Q UESTIONS? Thank you for your time… Image Credit: http://www.engg.uaeu.ac.ae/