1. CNN-based Action Recognition Using Adaptive Multiscale Depth Motion Maps And Stable Joint Distance Maps
Junyou He, Hailun Xia, Chunyan Feng, Yunfei Chu
Beijing University of Posts and Telecommunications
GlobalSIP 2018
Nov. 27, 2018 2

2. Adaptive Multiscale Depth Motion Maps(AM-DMMs)
OUTLINE
Motivations
The Proposed Method：
Adaptive Multiscale Depth Motion Maps(AM-DMMs)
Stable Joint Distance Maps(SJDMs)
Input Preprocessing
Network Training & Class Score Fusion
Experiments Results
Conclusions
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

3. Advantages of depth modality: providing 3D structural information
Motivations
Advantages of depth modality:
providing 3D structural information
insensitive to variations in lighting
Contains significant flicker noises
But
Depth map
Skeleton data :
more robust to noise
But
Not always reliable
Each modality can capture a certain kind of information
that is likely to be complementary to the other
Thus, integrating the information from depth and skeleton is expected to improve the recognition performance
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

4. The Spatio-Temporal Information
Motivations
Action Recognition
The Spatio-Temporal Information
Key
Handcrafted features: SIFT, color histogram, edge direction …
Domain knowledge
Shallow & Dataset-dependent
But
Difficult to memorize the entire sequence information
Difficult to extract high-level features
But
RNN-based methods
Thus, we propose a compact and effective CNN based method to capture the spatio-temporal information .
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

5. The Proposed Method Generate AM-DMMs Generate SJDMs
Input Preprocessing
Network Training & Class Score Fusion
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

6. Adaptive Multiscale Depth Motion Maps(AM-DMMs)
To capture more details of shape and motion information and cope with speed variations in actions
Suffer from loss of temporal information
DMMs
AM-DMMs
capture the detailed motion cues
cope with
speed variations
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

7. Adaptive Multiscale Depth Motion Maps(AM-DMMs)
AM-DMMs generated from a sample video of the action Swipeleft on three views
DMM of a depth video sequence with N frames
The motion energy E(i) of ith frame
returns the number of non-zero elements in a binary map
represents frame index
is the projected map of
Frame under projection view
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

8. The Proposed Method Generate AM-DMMs Generate SJDMs
Input Preprocessing
Network Training & Class Score Fusion
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

9. Stable Joint Distance Maps(SJDMs)
To avoid excessive noise, three reference joints which
are stable in most actions are used to compute relative distances of the other joints
The Euclidean distance at
frame t
is the joint indices
is one of three stable joints
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

10. Stable Joint Distance Maps(SJDMs)
The distances to different stable joints contain different spatial relationships and useful structural information of the skeleton
is expressed as follows:
corresponding to
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

11. The Proposed Method Generate AM-DMMs Generate SJDMs
Input Preprocessing
Network Training & Class Score Fusion
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

12. Input Preprocessing
Resized maps to make them compatible with the pre-trained CNN model and solve the variable-length problem
HSV-color coding has highlighted the differences in texture and edges
Sample color coded AM-DMMs and SJDMs generated by the proposed method on UTD-MHAD dataset
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

13. The Proposed Method Generate AM-DMMs Generate SJDMs
Input Preprocessing
Network Training & Class Score Fusion
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

14. Network Training and Class Score Fusion
A multi-channel CNN is adopted to exploit the discriminative features
Two fusion methods are expressed as follows
are score probability vectors
is the element-wise multiplication
are the accuracy of the corresponding network
is a function to find the index of the element having the maximum score
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

15. skeleton joint positions inertial sensor signals
Experiments: dataset
UTD-MHAD dataset:multimodal action dataset
RGB videos
depth videos
skeleton joint positions
inertial sensor signals
UTD-MHAD dataset: contains 27 different actions and each action is performed by 8 subjects (4 females and 4 males) and with up to 4 repetitions
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

16. Experiments Result
The effectiveness of different schemes and the results of individual CNN and two fusion methods
Comparisons of the different scheme on UTDMHAD dataset
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

17. Experiments Result
The performance of the proposed method and the results reported before on UTD-MHAD dataset
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

18. Experiments Result
Confusion matrix of proposed method on the UTD-MHAD dataset
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

19. The proposed AM-DMMs capture more shape clues and details of motion.
Conclusions
Presents an effective method for action recognition using a nine-channel CNN
The fusion of depth and skeleton modalities is proposed to improve the classification accuracy
The proposed AM-DMMs capture more shape clues and details of motion.
Transform one skeleton sequence into three SJDMs which describe different spatial relationships between joints
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

20. Thanks! Junyou He @BUPT 12211006@bupt.edu.cn
CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS