1. Real-Time Human Posture Reconstruction in Wireless Smart Camera Networks Chen Wu, Hamid Aghajan Wireless Sensor Network Lab, Stanford University, USA IPSN 2008 Speaker Lawrence

2. Outline Background Motivation Goal Challenge Strategy for Camera Sensor Network System Overview Wireless Smart Camera (Hardware) Human Pose Estimation (Algorithm) Result Conclusion

3. Background Traditional Camera network for surveillance & security New applications of camera network for multimedia, video conference…etc Wireless Camera network –Scalability –Privacy preservation –Flexibility on collaboration scheme between cameras

4. Motivation As pervasive sensors the cameras can free the users from wearable devices. Lack of real-time vision algorithm to achieve moderate complexity, robustness and scalability.

5. Goal Implementation of human pose interpretation on a wireless smart camera network. Employing distributed processing –Real-time vision & scalability for complex vision algorithms.

6. Challenge A vision sensor network poses three key challenges: –High computation capacity for real-time performance. –Wireless links limit image transmission (bandwidth & energy) –Lack of established vision-based fusion mechanisms (by real time)

7. Strategy for Camera SN Difference between Camera network & Distributed vision processing strategy systems. –Employ cameras as a wireless sensor network. Strategy: 1. Video data reducing (Network bandwidth) 2. Level of vision analysis to different PHY processors

8. Strategy for Camera SN (cont.) Central PC Smart Camera Level of vision analysis to different PHY processors

9. Scalability : Spatial and functional parallelism Each camera video processes its own data(spatial) Running their own function modules(functional) Strategy for Camera SN (cont.)

10. Smart camera communicate with the central PC through ZigBee System Overview LCD display Smart camera Different ZigBee channels

11. Data flow in the system System Overview (cont.) Semaphore tech for DPRAM P.S. DPRAM allows multiple r or w to occur at the same time. Asynchronous

12. Wireless Smart Camera Hardware Platform –VGA color image sensor –SIMD processor(IC3D) –Embedded processor(8051) –ZigBee platform

13. Parallel arch power consumption LPA(320 PEs) data processing GCP control IC3D & DSP operations PE # video format, e.g., VGA(640*480) The main design factors of SIMD frequency & PE # Wireless Smart Camera (cont.) MP-SIMD

14. Data sharing between processors –PDRAM functions as an asynchronous connection between IC3D and 8051 –Semaphore tech to prevent mutual access Wireless communication –P2P structure offers direct camera to PC communication –Maximum data rate : 100 Kbit/sec Wireless Smart Camera (cont.)

15. Review (Algorithm) –Goal : 2D to 3D –Ambiguity: Perspective views of the camera or self- occlusion of human body Pose Estimation Approach –Top-down –Bottom-up Human Pose Estimation

16. Top-down Human Pose Estimation (cont.)

17. Bottom-up Human Pose Estimation (cont.)

18. Top-down vs Bottom-up Top-down –Strength Occlusion handling Contours & body part association –Weakness Search tech complexity(depth) Computational load(projection) Bottom-up –Strength Much less demands on 3D switch –Weakness Complex assemble Difficult to detect occlusions Human Pose Estimation (cont.)

19. Challenges & Method –Bandwidth constraint (100Kbits/sec)/(30frames/sec)/(8bits/Byte) ≈ 400B/frame solution: Detect body part cancroids coordinates –Limited image processing capability of the SIMD processor solution: Color segmentation –Robustness with varied environment solution: Auto-balancing filtering & combination Human Pose Estimation (cont.)

20. In-node processing Detect positions(x, y): –Head, shoulders and hands –2Bytes for x and y Detect mechanism: –Face -> face color model –Head -> skin color model –Shoulders -> shirt color model (low-pass filter) Human Pose Estimation (cont.)

21. The image processing program on IC3D

22. Human Pose Estimation (cont.)

23. Processing on the central PC –Noise filtering and 2D to 3D reconstruction Human Pose Estimation (cont.)

24. Results

25. Standard Deviation of detected body part coordinates in the smart cameras (in pixels) and those after noise filtering Results (cont.)

26. Original data from the smart cameras and data after noise filtering Head Left shoulderRight shoulder Results (cont.)

27. Left hand Right hand

28. Conclusion Propose an algorithmic strategy to approach vision problems in a wireless camera sensor network Major aspect of the strategy: –reduce video data locally through smart camera Implement a prototype system of 3D human reconstruction using a wireless smart camera. Wireless camera networks will offer potentials for user-centric applications.

29. Thanks for listening