1. Two Examples Of Indoor And Outdoor Surveillance Systems: Motivation, Design, And Testing Ioannis Pavlidis Vassilios Morellas Honeywell Laboratories

2. Graduate Seminar in CIS Video Processing and Mining CIS 750 – Spring 2003 Presented by: Ken Gorman

3. Agenda CCN – Cooperative Camera Network CCN – Cooperative Camera Network DETER - Detection of Events for Threat Evaluation and Recognition DETER - Detection of Events for Threat Evaluation and Recognition

4. Cooperative Camera Network (CCN) Network of cooperating cameras Network of cooperating cameras Controlled by computer vision software Controlled by computer vision software Features: Features: –mechanism for counting the people present in various parts of the building –An automatic or semi-automatic mechanism for tagging people. –Report tagged individuals whereabouts whenever they are within the field of view

5. Major Components COTS Hardware & Software Set-Up COTS Hardware & Software Set-Up Change Detection Change Detection Counting People Counting People Tracking People Tracking People

6. State of the Art Active badges Active badges –small, electronic devices worn by people –transmit an ID signal to receivers placed around the building –ID signal corresponds to the identity of the badge’s wearer –received signals are used to compute the wearer’s location

7. Badge Examples Infrared-transmitting badges at Olivetti Research and Xerox PARC, Infrared-transmitting badges at Olivetti Research and Xerox PARC, Olivetti ultrasonic badges at AT&T Laboratories in Cambridge, UK Olivetti ultrasonic badges at AT&T Laboratories in Cambridge, UK Radio frequency tags from PinPoint Radio frequency tags from PinPoint Wired and unwired motion trackers from Wired and unwired motion trackers from –Ascension Technology –Polhemus

8. Disadvantages Consumers unwilling to wear badges Consumers unwilling to wear badges Cumbersome Cumbersome

9. Alternatives ???? ????

10. Cameras Pro – Leaves users unencumbered Pro – Leaves users unencumbered Cons – Not as reliable as badge methods Cons – Not as reliable as badge methods

11. Camera Arrangement Overlapping Fields of View Overlapping Fields of View

12. Fundamentals Imaging Technologies for Surveillance Systems Imaging Technologies for Surveillance Systems –Image Segmentation –Tracking Mechanism –Multi-Camera Fusion Threat Assessment Threat Assessment

13. Multi-Normal Pixel Representation

14. Initialization Goal - provide statistically valid values for the pixels corresponding to the scene. Goal - provide statistically valid values for the pixels corresponding to the scene. Starting point for the dynamic process of foreground and background awareness Starting point for the dynamic process of foreground and background awareness

15. Initialization Methods Used: Methods Used: –K-Means – better for plazas and malls –Expectation-Maximization – better for changing weather conditions [1]

16. Image Segmentation Each pixel is considered as a mixture of five time-varying trivariate normal distributions Each pixel is considered as a mixture of five time-varying trivariate normal distributions

17. Image Segmentation The term represents a trivariate Normal distribution with vector mean and variance- covariance matrix The term represents a trivariate Normal distribution with vector mean and variance- covariance matrix

18. Image Segmentation The distributions are trivariate to account for the three component colors (red, green, and blue) of each pixel in the general case of a color camera. The distributions are trivariate to account for the three component colors (red, green, and blue) of each pixel in the general case of a color camera.

19. Image Segmentation For simplification, the variance-covariance matrix is assumed to be diagonal with x R,x G,x B, having identical variance within each Normal component, but not across all components For simplification, the variance-covariance matrix is assumed to be diagonal with x R,x G,x B, having identical variance within each Normal component, but not across all components

20. Update Cycle Distributions are ordered based upon their weights. Distributions are ordered based upon their weights. Member of a Distribution Member of a Distribution Distribution is in background or foreground Distribution is in background or foreground Jeffreys [2] algorithm used for “matching” pixel to distribution Jeffreys [2] algorithm used for “matching” pixel to distribution Distributions are Updated Distributions are Updated

21. Matching We use the Jeffreys divergence measure (J) to determine whether the incoming data point belongs to one of the existing distributions We use the Jeffreys divergence measure (J) to determine whether the incoming data point belongs to one of the existing distributions The Jeffreys number measures how unlikely it is that one distribution (g) was drawn from the population represented by the other (f) The Jeffreys number measures how unlikely it is that one distribution (g) was drawn from the population represented by the other (f) K* - prespecified cut-off value K* - prespecified cut-off value

22. Update – Match Found Incoming pixel state is labeled either background or foreground Incoming pixel state is labeled either background or foreground All the parameters of the matched distribution are updated according to the method of moments All the parameters of the matched distribution are updated according to the method of moments Only the weights of the other distributions are updated Only the weights of the other distributions are updated

23. Update – No Match Incoming pixel state is labeled foreground Incoming pixel state is labeled foreground Last distribution in the ordered list is replaced Last distribution in the ordered list is replaced All the parameters of the new distribution are updated All the parameters of the new distribution are updated Only the weights of the other distributions are updated Only the weights of the other distributions are updated

24. Jeffrey’s Algorithm Jeffreys number measures how unlikely it is that one distribution (g) was drawn from the population represented by the other (f). Jeffreys number measures how unlikely it is that one distribution (g) was drawn from the population represented by the other (f).

25. Broken Clouds Preferential, in orderNo Proference Preferential, in orderNo Proference

26. Segmentation of Moving Objects The form of some of the distributions could change The form of some of the distributions could change Some of the foreground states could revert to back-ground and vice versa. Some of the foreground states could revert to back-ground and vice versa. One of the existing distributions could be dropped and replaced with a new distribution. One of the existing distributions could be dropped and replaced with a new distribution.

27. Predictive Tracking On-line segmentation of foreground pixels On-line segmentation of foreground pixels Calculation of blob centroids Calculation of blob centroids Multiple-Hypotheses Tracking Algorithm Multiple-Hypotheses Tracking Algorithm

28. Multiple-Hypotheses Tracking Recursive Bayesian probabilistic procedure Recursive Bayesian probabilistic procedure Does NOT commit early to trajectory Does NOT commit early to trajectory

29. Multiple Hypothesis Tracking

30. Kalman filtering prediction based on constant velocity models Kalman filtering prediction based on constant velocity models K-best hypothesis trajectory tree generation, pruning and merging K-best hypothesis trajectory tree generation, pruning and merging Bayesian probability calculations for matching input data to track hypothesis Bayesian probability calculations for matching input data to track hypothesis See references [5] and [6] for exact algorithm See references [5] and [6] for exact algorithm

31. Multi-Camera Fusion Monitoring of large areas can only be accomplished using multiple cameras Monitoring of large areas can only be accomplished using multiple cameras Panoramic View is created by fusing individual camera FOVs Panoramic View is created by fusing individual camera FOVs Object motion registered against a global coordinate system Object motion registered against a global coordinate system

32. Multi-Camera Fusion Optimal Coverage Scheme is created Optimal Coverage Scheme is created Minimal use of cameras to minimize cost Minimal use of cameras to minimize cost

33. Multi-Camera Fusion Compute HOMOGRAPHY matrix H between two cameras based on CoG of moving objects appearing in the overlapping areas of the two fields of view Compute HOMOGRAPHY matrix H between two cameras based on CoG of moving objects appearing in the overlapping areas of the two fields of view Requirement: Information exchange between respective computers (e.g., pixel intensity data and CoG of moving objects in pixel coordinates) Requirement: Information exchange between respective computers (e.g., pixel intensity data and CoG of moving objects in pixel coordinates)

34. Homography Matrices Computation Least Squares Method Least Squares Method Very popular Very popular Relatively simple Relatively simple Defined in Reference [6] Defined in Reference [6]

35. Homography Matrix Used Kanatani Method Used Kanatani Method Based on a statistical optimization theory for geometric computer vision Based on a statistical optimization theory for geometric computer vision Cures the deficiencies exhibited by Least-Squares Cures the deficiencies exhibited by Least-Squares

36. Kanatani Method Epipolar constraint may be violated by various noise sources due to the statistical nature of the imaging problem Epipolar constraint may be violated by various noise sources due to the statistical nature of the imaging problem

37. Multi-Camera Fusion O 1 and O 2 are Optical Centers O 1 and O 2 are Optical Centers P(x,y,z) is a point in the scene that falls in the common area between the two camera P(x,y,z) is a point in the scene that falls in the common area between the two camera Vector O 1 p, O 2 q, and O 1 O 2 are co-planar Vector O 1 p, O 2 q, and O 1 O 2 are co-planar

38. References Paolo Remagnino, et al (Editors). Video-Based Surveillance Systems: Computer Vision and Distributed Processing. Kluwer Academic Publishers, 2002. Paolo Remagnino, et al (Editors). Video-Based Surveillance Systems: Computer Vision and Distributed Processing. Kluwer Academic Publishers, 2002. http://www.htc.honeywell.com/projects/deter/ http://www.htc.honeywell.com/projects/deter/ http://www.htc.honeywell.com/projects/deter/ [1] - A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the em algorithm (with discussion),” J. Roy. Stat. Soc. B, vol. 39, pp. 1–38, 1977. [1] - A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the em algorithm (with discussion),” J. Roy. Stat. Soc. B, vol. 39, pp. 1–38, 1977. [2] - J. Lin, “Divergence measures based on the Shannon entropy,” IEEE Trans. Inform. Theory, vol. 37, pp. 145–151, Jan. 1991. [2] - J. Lin, “Divergence measures based on the Shannon entropy,” IEEE Trans. Inform. Theory, vol. 37, pp. 145–151, Jan. 1991. [3] - C. Stauer and W.E.L. Grimson, “Adaptive background mixture models for real-time tracking," in Proceedings 1999 IEEE Conference on Computer Vision and Pattern Recognition, Fort Collins, CO, June 23-25 1999, vol. 2, pp. 246-252. [3] - C. Stauer and W.E.L. Grimson, “Adaptive background mixture models for real-time tracking," in Proceedings 1999 IEEE Conference on Computer Vision and Pattern Recognition, Fort Collins, CO, June 23-25 1999, vol. 2, pp. 246-252.

39. References (cont.) [4] D. B. Reid, “An algorithm for tracking multiple targets”, IEEE Transactions on Automatic Control, vol. 24, pp. 843{854, 1979. [4] D. B. Reid, “An algorithm for tracking multiple targets”, IEEE Transactions on Automatic Control, vol. 24, pp. 843{854, 1979. [5] I. J. Cox and S. L. Hingorani, “An efficient implementation of Reid's multiple hypothesis tracking algorithm and its evaluation for the purpose of visual tracking," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 18, no. 2, pp. 138-150, 1996. [5] I. J. Cox and S. L. Hingorani, “An efficient implementation of Reid's multiple hypothesis tracking algorithm and its evaluation for the purpose of visual tracking," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 18, no. 2, pp. 138-150, 1996. [6] L. Lee, R. Romano, and G. Stein, “Monitoring activities from multiple video streams: Establishing a common coordinate frame," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 758 { 767, 2000. [6] L. Lee, R. Romano, and G. Stein, “Monitoring activities from multiple video streams: Establishing a common coordinate frame," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 758 { 767, 2000.