Presentation is loading. Please wait.

Presentation is loading. Please wait.

Silhouette Lookup for Automatic Pose Tracking N ICK H OWE.

Similar presentations

Presentation on theme: "Silhouette Lookup for Automatic Pose Tracking N ICK H OWE."— Presentation transcript:

1 Silhouette Lookup for Automatic Pose Tracking N ICK H OWE

2 Goal: 3D Pose Tracking Full 3D “motion capture” from 2D video Single camera Unmarked video Difficulties:  3D ambiguity  Self-occlusion  Foreshortening  Appearance changes  Shadowing ↑ (Uses hand-entered data)

3 The “Old” Way: Incremental Tracking Previous frame with Compare with Image Refine 2D Pose 2D Pose + Appearance NumericalOptimization Next Next frame

4 Creeping Error Incremental  Errors accumulate and grow. May be mitigated by:  Better motion models (more guidance)  Better appearance models (3D)  Better tracking (multiple hypotheses) [Sidenbladh, et. al.; Sminchisescu, et. al.] Intrinsic problems still remain. (initialization, error recovery)

5 Direct Pose Estimation Consider human abilities:  Estimate pose from still photo  Estimate pose from stick figure  Estimate pose from silhouette [Brand ’99; Rosales et. al. ’01)

6 Recognition/Retrieval Hypothesis: Humans can recognize pose by recalling similar examples.  Pose Recognition  Retrieval

7 Recognition/Retrieval Hypothesis: Humans can recognize pose by recalling similar examples.  Pose Recognition  Retrieval New Approach: 1. Store many silhouettes with known poses 2. Given video, extract silhouettes 3. Retrieve best candidate matches 4. Look for plausible series of poses over time

8 Some Related Work Estimating Human Body Configuration Using Shape Context Matching Mori & Malik, ECCV 2002 3D Tracking = Classification+Interpolation Tomasi, Petrov, & Sastry, ICCV 2003 Temporal Integration of Multiple Silhouette-based Body-part Hypotheses Kwatra, Bobick, & Johnson, CVPR 2001 3D Human Pose from Silhouettes by Relevance Vector Regression Agarwal & Triggs, CVPR 2004

9 Silhouette Comparison Turning angle (Captures morphology) Chamfer distance (Captures overlap) Combine using Belkin technique (score = sum of individual ranks)

10 Sample Retrievals (Hits from a small library of 1600 poses)

11 Coordination Between Frames Need to pick from top matches at each frame.  Want good image match at all frames  Want small change between frames  Markov chain minimization! Best local choices minimize global error etc. frame i-1frame iframe i+1

12 Too Much Coffee? Initial solution shows “twitches”

13 Smoothing it Out Jitters in motion parameters smoothed via polynomial splines

14 Making it Match Problem: poor overlap between observed silhouette & smoothed solution  Work with 11-frame splines  Optimize spline parameters to reduce chamfer distance Result: better match to observations, still smooth

15 Walking Sequence Result

16 Re-rendering Same scene, different viewpoint.

17 Another Example Tracked using library of ballet poses

18 Incremental Tracking Markov chain is best for offline use But: Convergence after ~10 frames  Incremental tracking with latency

19 Key Points Silhouette lookup provides set of potential poses for each frame Markov chain selects best temporal pose sequence (HMM) Smoothing & optimization based upon temporal splines Result: simple tracker, tolerates errors

20 Thank you! Questions?

21 Continuing Challenges Mistakes in rotational direction No data for parts not on silhouette  Incorporate optical flow Some unrealistic motions generated  Incorporate motion model Correct pose not always retrieved  Improve library coverage, retrieval

22 Future Research People carrying objects Multiple overlapping people (sports) Time considerations  Optimization slow  Chaining currently slow  Holy Grail: Real-time tracking

23 2. Identify best (least expensive) result Markov Chain Minimization Frame 1Frame 2Frame n... 1. Compute least expense to reach each state from previous frame (cost = estimate of plausibility) State 2A State 2C State 2B State 1A State 1C State 1B State n A State n C State n B 3. Backtrack, picking out path that gave best result.

24 Silhouette Extraction Many candidate approaches.  Moving & fixed camera This work:  Static camera  Graph-based segmentation

25 Making it Match Solution doesn’t match exactly yet.

Download ppt "Silhouette Lookup for Automatic Pose Tracking N ICK H OWE."

Similar presentations

Ads by Google