1. Human Posture Recognition with Convex Programming Hao Jiang, Ze-Nian Li and Mark S. Drew School of Computing Science Simon Fraser University Burnaby, BC, V5A 1S6

2. July, 11 th, 2005Simon Fraser University Human Posture Recognition  Recognizing human postures is very important in vision and multimedia.  It has many applications in surveillance, human computer interaction, image and video database analysis and retrieval.  At the same time, recognizing human postures is a hard problem.

3. July, 11 th, 2005Simon Fraser University The Challenges of Human Posture Recognition  It is hard to recognize human postures because: –Articulated nature of a human body –No segmentation schemes are available for general images or videos. –Strong background clutters. –Large appearance changes because of clothing  Different schemes have been studied.

4. July, 11 th, 2005Simon Fraser University Methods for Posture Recognition  Methods having been studied: –Silhouette based method with background subtraction –Multi-camera based methods –Tracking body movement –Chamfer matching based schemes –Shape context based schemes  These methods are not sufficient to address the problem robustly.

5. July, 11 th, 2005Simon Fraser University The Proposed Method  We will present a matching based scheme that has the following properties: –Based on a robust convex (linear) programming matching scheme –Work for cases where no background subtraction is available –Able to deal with strong background clutters –Able to deal with large appearance changes

6. July, 11 th, 2005Simon Fraser University Matching Distance Transform Canny Edge Detection Distance Transform Feature Point Selection Delaunay Triangulation Template Generation Canny Edge Detection Distance Transform Matching With LP Template Image Target Image Object Recognition result

7. July, 11 th, 2005Simon Fraser University Matching as a Labeling Problem p q fpfp fqfq Target p’ Target q’ Clutter Target Template Mesh Target Image

8. July, 11 th, 2005Simon Fraser University The Labeling Problem  The matching problem can be formulated as the following optimization problem: Matching cost Smoothing term

9. July, 11 th, 2005Simon Fraser University Convex Relaxation  The original problem is a hard non-convex problem. We convert it to LP: |fp-fq| c’(s,j)

10. July, 11 th, 2005Simon Fraser University Properties of the Relaxation 1. For convex problems, LP exactly solves the continuous extension of the original problem. 2. For general non-convex problems, LP solves the problem where each matching surface is replaced by the lower convex hull. 3. The “cheapest” basis set for each site corresponds to the lower convex hull’s vertices

11. July, 11 th, 2005Simon Fraser University The Effect of Covexification For site 0 For site M-1 … c(0,j) c(M-1,j) Label :Basic Labels Convexification Label c(i,j) : Lower Convex Hull Vertices Label For non-convex problems, the relaxation replaces each c(m,j) by its lower convex hull surface:

12. July, 11 th, 2005Simon Fraser University Searching Scheme of Simplex Method 4. Using simplex method, there are at most three adjacent non-zero weight basis labels: : Zero-weight basis label : Non-zero-weight basis label : non basis label: Continuous label Searching for one site

13. July, 11 th, 2005Simon Fraser University Successive Relaxation Scheme  Single relaxation may miss the global optimum because of convexification effect  An intuitive scheme is to shrink the trust region and reconvexify the data in the smaller region  This scheme is found to be able to greatly improve the matching results

14. July, 11 th, 2005Simon Fraser University The Trust Region Shrinking

15. July, 11 th, 2005Simon Fraser University Successive Relaxation Scheme (An Example) min C(1,  1 )+ C(2,  2 )+0.5|  1 -  2 |

16. July, 11 th, 2005Simon Fraser University Shape Recognition  We have to define the goodness of matching –Matching cost (M): Average difference of the template and target image in the ROI. –Deformation (D): Affine transformation compensated pairwise distance changes –Shape Context in the ROI (C).  Finally, we use M + a*D+b*C to quantify the matching

17. July, 11 th, 2005Simon Fraser University Random Dots Experiment Noise: 50% Random Disturbance: 5 Noise: 100% Random Disturbance: 5 Noise: 50% Random Disturbance: 10 Noise: 100% Random Disturbance: 10

18. July, 11 th, 2005Simon Fraser University Matching Synthetic Images Results (a): Template model showing distance transform; (b): Matching result of proposed scheme; (c): Matching result by GC; (d): Matching result by ICM. (e): Matching result by BP. : GC : ICM : BP : LP

19. July, 11 th, 2005Simon Fraser University Matching Leaves

20. July, 11 th, 2005Simon Fraser University Experiment Results An example where traditional methods fail. (a): Template image; (b): Target image; (c): Edge map of template image; (d): Edge map of target image; (e): Template mesh; (f): Matching result of the proposed scheme; (g): ICM matching result; (h): Sliding template search result.

21. July, 11 th, 2005Simon Fraser University Gesture Recognition Results Template Top match Second match

22. July, 11 th, 2005Simon Fraser University Gesture Recognition Results

23. July, 11 th, 2005Simon Fraser University Video Browsing Result

24. July, 11 th, 2005Simon Fraser University Video Browsing Result

25. July, 11 th, 2005Simon Fraser University Multiple Target Matching Results

26. July, 11 th, 2005Simon Fraser University Conclusion and Future Directions  We present a robust matching framework for human posture recognition  The method can be applied to multimedia data retrieval in image or video database, or human computer interaction applications  In future work: –We will add tempera information for behavior recognition

27. July, 11 th, 2005Simon Fraser University Future work –The successive reconvexification is in fact very general. It can be used to increase the robustness of many other matching schemes, such as BP and GC –The proposed matching can be used for many other applications, such as tracking, object recongnition, motion estimation etc.

28. July, 11 th, 2005Simon Fraser University The End