1. Video Surveillance for Human Emotion Identification(VSHEI)
Anku Adhikari, Hao Wu and Lingyong Wang (aadhikr2, haowu11,
Introduction
In this project we propose VSHEI, a system to extract heart pulse from human facial videos, that is suitable for varied environments and tolerant to natural motion and movements. We highly improve results from comparative works that use Eulerian Color Magnification and demonstrate usage of the results for emotion analysis by heart rate variability detection.
Goals:
Design a system able to extract human pulse under conditions of normal movement and environmental dynamics.
Investigate best methods to pre-process the video for facial tracking and stabilization.
Use current methods that magnify changes in human face video frames and improve and customize them.
Extract clear pulse signals and eliminate environmental noise.
Demonstrate the application of the extracted pulse information and multiple video footage for detecting heart rate variability and diagnosing heart rate change indicating emotion/physiological states.
Implementation
Results
System Framework:
Raw Video
Face Tracking Output
Video Stabilization Output
Eulerian Magnification Output
Extracted Pulse Waves
Pulse Rate, Diagnosis
Face Tracking:
Signal Processing:
Video Stabilization:
Emotion Analysis:
Contribution 
VSHEI has much improved results compared to Eulerian Color Magnification method(MIT) with results that are sharper, less noisy and more accurate.
It can extract heart pulses in different scenarios of lighting, indoor/outdoor condition, skin type and movement levels and video at different distance and facial angle from camera.
It preprocesses to stabilize the face in the video and handle different levels of motion in the human subject.
It is able to face track changes introduced by 2D motion in all direction and slight 3D face rotation and motion.
It uses feature points to tracking human faces. This approach out performs another algorithm that uses skin color tone.
VSHEI pulse rate and waveforms can be used to analyze pulse and facial changes for other human physiology applications directly.
Pulse rate = bpm
State: Resting state
Diagnosis: physiological stress not detected
Evaluation
Experimental Videos
Description
Level1(Motion test)
Stable subject
Level2(Motion test)
Slight movement
Level3(Motion test)
Subject moving forward
Low HR State (Variability test)
Subject before exercise
High HR State(Variability test)
State Subject immediately after exercise
Evaluation done using 7 samples of experimental video data. Wide variety of video shot samples collected for experimental analysis in conditions of:
Indoor and Outdoor lighting conditions
Different subjects and skin tones
Different types of movement
Different levels of physical exertions
Different types of background noise conditions
Future Work
Further optimization for real-time applications.
VSHEI can be applied for other applications that investigate physiological changes in the body: criminal investigation, athlete training, medical monitoring, etc.