1. The Recognition of Human Movement Using Temporal Templates Liat Koren

2. - 2 - Lecture subjects Introduction Prior work The Temporal Templates Usage example

3. - 3 - Introduction Computer vision trends –Less image or camera motion –More on labeling of action Reasons –More computational power –Wireless application –Interactive environments

4. - 4 - Introduction – cont. Recent efforts are in Three Dimensional object reconstruction –Assuming it will have to be used in the recognition of human motion. This article claims otherwise –View-based approach –Direct recognition

5. - 5 - Motivating Example

6. - 6 - Motivating Example

7. - 7 - Motivating Example Static pictures –Hard to recognize. Sequence on motion –Human can recognize without three dimensional reconstruction. Conclusion –It is possible to recognize movement using only the motion itself.

8. - 8 - Process –Recover the pose of the person at each time instant using a 3D model. –The model’s projected image should be as close as possible to the object (e.g. edges of body in the image) Drawbacks –Complicated process –Human interference is usually required –Special imaging environment

9. - 9 - 2D Based recognition Action is a sequence of static poses of object. Requires –Normalization –Removal of background

10. - 10 - Wilson and Bobik’s approach Actions are usually hand gestures Representation –Actual image –Grayscale –No background Benefits: –Hand appearance is fairly similar over a wide range of people Problems –Actions that include the appearance of the whole body are not visually consistent across different people.

11. - 11 - Yamato’s et al. approach Representation –No background –Black and white silhouettes Matching –Vector quantize –Usage of a mathematical method Benefits –Help handling the variability between people Problems –Disappearance of movement inside the silhouette

12. - 12 - Summery of prior work Action is a sequence of static poses. Requires individual features or properties that can be extracted and tracked from each frame. Recognition of movement from a sequence of images is a complicated task. Usually requires previous recognition and segmentation of the person.

13. - 13 - Motion based recognition Attempt to characterize the motion itself without reference to the underlying static poses of the body. Possible approaches –Blob like representation –Tracking of predefined regions (e.g., legs, head, mouth) using motion. Face expression patches Whole body patches –Measure typical patterns of muscle activation

14. - 14 - Terms Movement –where – motion has occurred in image sequence. MEI – Motion Energy Image –how – the motion is moving. MHI – Motion History Image + Temporal Templates

15. - 15 - Temporal Templates Representation of movement –View specific –Movement is motion in time –Vector image that can be matched against stored representations of movements. Assumptions –Background is static –Camera movements can be removed –Motion of irrelevant objects can be eliminated

16. - 16 - M otion- E nergy I mages where did the movement occurred ….

17. - 17 - M otion- E nergy I mages Notice that: –If τ is very big, all the differences are accumulated –Τ has a vast influence on the temporal representation of a movement.

18. - 18 - M otion- E nergy I mages Smooth change in the viewing angle causes a smooth change in the viewed image, thus coarse sampling of the viewing circle is enough (30°)

19. - 19 - M otion- H istory I mages Intensity of a pixel represents the temporal history in that pixel. Newer movement is brighter.

20. - 20 - M otion- H istory I mages A time-window of size τ is used – movement older than τ is ignored. The results of the article uses a simple replacement and decay operator: Notice that MEI can be calculated out of MHI by painting in white any non-black pixel One may wonder, why not use only MHI ? Answers will be given later…

21. - 21 - MEI and MHI in a nutshell MEI and MHI are two vector images designed to encode a variety of motion properties. Benefits in this representation is that the calculation is recursive, thus only up-to- date information need to be stored, making the computation both fast and space efficient.

22. - 22 - Matching Temporal Templates Collect training examples of each movement from a variety of viewing angles. Compute statistical representation of the MHI/MEI images (Hu moments) Given an input movement: –Calculate a statistical representation –Use mahalanobis distance to find a stored movement, that is the nearest to the input.

23. - 23 - Mahalanobis Distance Example

24. - 24 - Reasoning for the algorithm Mahanobis distance provides: –Good matching as shown in the results of the article. –Simple calculation which makes real-time applications feasible. Hu moments allow representation of images, that is invariant to scale or translation. One problem with Hu moments is that: “Hu moments are difficult to reason about intuitively” (the authors)

25. - 25 - Testing the system 18 exercises performed by experienced aerobic instructor. MEIs are on the bottom rows.

26. - 26 - Why both MHI and MEI ? Because MHI and MEI perceive two different characteristics of the movement (the “where” and the “how”) they look different,and thus, both essential.

27. - 27 - First experiment Input 30° left of the subject Match against all seven views of all 18 moves 12 out of 18 are correctly recognized 0° 30° 60° 90° 120° 150° 180°

28. - 28 - Analyze the results of 1 st exp. Move 13 in 30 °Move 6 in 0 °The correct match input falsecorrect

29. - 29 - Combining multiple views Two cameras with orthogonal views Minimize the sum of the mahalanobis distance between the two input templates and two stored views of movement that have 90° between them. Hidden assumption: we know the angular relationship between the cameras.

30. - 30 - Second Experiment 0° 30° 60° 90° 120° 150° Input with two cameras: 30° left of the subject 60° right of the subject Match against all seven views of all 18 moves 15 out of 18 are correctly recognized

31. - 31 - Analyze the results of 2 nd exp. Move 16Move 15 The correct match input falsecorrect

32. - 32 - Segmentation and Recognition Problem : speed of performance is different among different people. Solution: Segmentation –When training the system, calculate τ max and τ min for each movement. – Use algorithm to match over a wide range of τ.

33. - 33 - Problems Problems with current system –One person partially occludes another Solution: Use several cameras –More than one person appears in the view point Solution: use a tracking bounding box

34. - 34 - More Problems Motion of part of the body is not specified during a movement –Possible solutions Automatically mask away regions of this type of motion Always include them Camera motion –Rather easy to eliminate since camera motion is limited. Person is performing the movement while locomotion

35. - 35 - The KidsRoom: An Application room is aware of the children (at most 4) The room takes the children to a story. The room’s reaction is influenced by the actions of the children. Current story : adventurous tour to monster land In the last scene the monsters teach the children to dance. Then, the monsters follow the children if they perform movements they “know” The narration coerces the children to room locations where occlusions is not a problem

36. - 36 -