1. Distributed Sensor Exploitation 6.1 Rapid Deployment of Smart Camera Networks Dr. B.S. Manjunath University of California, Santa Barbara manj@ece.ucsb.edu 805-893-7112 Thrust Technical Review March 2012

2. Distributed Sensor Exploitation 6.1 Rapid Deployment of Smart Camera Networks SCHEDULE: TTA: N/A TECH TRANSITION PATH: DYNAMIC WIKI/RESOURCE MANAGER (D&I) TECHNICAL APPROACH: l Geo-calibration using mobile sensors l Auto-calibration and network topology discovery using visual analysis l Detecting and tracking objects in a camera network through fusion of multiple views, despite wireless communication challenges l Analyzing events over non-overlapping camera views PERFORMER: University of California at Santa Barbara B. S. Manjunath, Professor RESEARCH QUESTIONS: How can we geo-calibrate a stationary network using mobile sensors and associated GPS and other information? What are the issues in discovering the geometric/spatial and temporal connections and constraints within a camera network with overlapping and non-overlapping views? What information needs to be computed at the smart sensor nodes for activity analysis without distributing the raw video? When and how to fuse multi-modal information, e.g., mobile, airborne, and fixed cameras, for robust object tracking and activity analysis? MILITARY RELEVANCE/OPERATIONAL IMPACT: l Rapid deployment of a network of smart cameras and UAVs can help monitor any potential area, especially in high risk and remote sites. NAVAL S&T FOCUS AREAs ADDRESSED: l Asymmetric & irregular warfare TASKS FY10 FY11FY12 Camera network setup Object detection & 2D tracking Object tracking in 3D Auto-calibration and topology Event & behavior detection Demos on campus testbed UAV: Provides intelligence from overhead Rapidly Deployed Smart Camera: Provides ground-based views and distributed image analysis 2012 Thrust Area Technical Review

3. 3 Principal Events & Activities FY2010FY2011FY2012 O N D J F M A M J J A S Camera Network Setup Auto Calibration & Toplology Geo-calibration Object Detection and 2D tracking Object Tracking in 3D Event/Behavior analysis Demonstration on DURIP Testbed Top Level POA&M (Summary Task) Field deployment Final demonstration 2012 Thrust Area Technical Review

4. Project Work Breakdown Structure 2012 Thrust Area Technical Review

5. 5 Detailed Technical Approach Objective –Development of a rapidly deployable network of smart camera sensors to gather mission-specific information for surveillance and human activity analysis Challenges –Minimize human interaction –Wireless communication constraints: limited bandwidth, unpredictable latency, dropped packets, and time synchronization –Few examples to train the system

6. 2012 Thrust Area Technical Review Detailed Technical Approach 6 Events in FY 10 Setup of Outdoor Camera Network Quick Chessboard Calibration Preliminary network topology discovery Events in FY11 Network topology discovery Automatic calibration without patterns Beyond 2D tracking Continued multi-camera tracking with overlapping and non-overlapping views. Preliminary distributed processing with existing work Events in FY12 Geocalibration with mobilesensors Multi-camera multi-object tracking in a wide-area network Browsing and search in a distributed network Project Roadmap

7. 2012 Thrust Area Technical Review Detailed Technical Approach 7 Project Introduction Overview of the UCSB Campus Camera Network –Infrastructure funded by a ONR DURIP award –Implementation by students on this project Research Accomplishments –Distributed Geocalibration –Distributed tracking and fusion –Topology discovery and network data summarization –Browsing and search in a distributed network Summary and Conclusions

8. 2012 Thrust Area Technical Review Detailed Technical Approach 1.Camera Placement: Where to place the cameras? –Using prior information, such as GPS tracks, to optimally place cameras –Novel approach to camera placement based upon “optimal” reconstruction of tracks and activities with selected observations 2.Calibration and Topology: How are the cameras related? –Necessary for most multi-camera applications to understand how the cameras are related in a physical space (calibration) or relatively according to their activities (topology) –Novel methods for calibrating additional cameras in a large calibrated network –Novel methods for calibrating cameras in a network individually, but in a global coordinate system without entering the scene –Examined topology methods on our network 8

9. 2012 Thrust Area Technical Review Detailed Technical Approach 3.Object Tracking: How to track or search objects? –Tracking or searching objects in a multi-camera network requires taking advantage of all overlapping and relevant information –Novel method for distributed, online multi-camera tracking using multiple instance learning and particle filtering with appearance and geometric fusion. –Novel trajectory searching with graph modeling 4.Event/Behavior Analysis: How to perform activity analysis over a distributed camera network? –The ultimate goal of a wide-area network is to make high level decisions and recognitions across many cameras –Novel camera network summarization algorithm –Addressing the relatively unexplored problem of activities analysis in a large, wide-area network with many non-overlapping views 9

10. 2012 Thrust Area Technical Review Topology discovery Camera placement N+1 camera calibration Geo-calibration Distributed object tracking Object searching Network summarization Wide-area activity analysis Topic relationships

11. 2012 Thrust Area Technical Review 11 UCSB Camera Network Examined challenges through manual network setup Developing optimal camera placement algorithm based on GPS trajectories

12. 2012 Thrust Area Technical Review Project Technical Assesment New Geo-calibration formulation with mobile sensors Distributed Object Tracking on a Camera Network Video Summarization in a Multi-camera setting Distributed browsing and search 12 Recent Progress

13. 2012 Thrust Area Technical Review Project Technical Assesment - 1 Project Introduction Overview of the UCSB Campus Camera Network –Infrastructure funded by a ONR DURIP award –Implementation by students on this project Research Accomplishments –Distributed Geocalibration –Distributed tracking and fusion –Topology discovery and network data summarization –Browsing and search in a distributed network Summary and Conclusions 13

14. 2012 Thrust Area Technical Review 1. Geocalibration using mobile devices

15. 2012 Thrust Area Technical Review Geo-Calibration How do you calibrate cameras with non-overlapping views? 15

16. 2012 Thrust Area Technical Review Geo-Calibration Geo-Calibration is the process of finding the projective geometry of a camera in a global coordinate system. Traditional, image-based methods are in a local coordinate system and cannot be shared across different calibrations 1 Existing methods for integrating global coordinates involve GPS tagged objects in the scene or aligning with a map 2 16 1. Zhang, “A flexible new technique for camera calibration,” PAMI 2000 2. Kaminsky et al. “Alignment of 3D Point Clouds to Overhead Images.” W on Internet Vision 2009.

17. 2012 Thrust Area Technical Review Geo-Calibration Method Take pictures of the camera’s scene using smartphones and collect location and orientation metadata –Location sensor: GPS –Orientation sensor: compass + accelerometer or gyroscope No objects need to be placed in the scene Data can be collected rapidly 17

18. 2012 Thrust Area Technical Review Geo-Calibration Method Geo-calibration: Given image(s) from a camera 1.Collect, using smartphone, calibration images with GPS and orientation metadata 2.Calibrate cameras in a local coordinate system using images 3.Transform from local to global GPS coordinates 4.Refine position and orientation estimates Two approaches: centralized and distributed 18

19. 2012 Thrust Area Technical Review Distributed Geo-Calibration Assume GPS positions accurate, then average across all stereo/pairwise estimates Rotation average using Karcher Mean Position average using least squares 19 GPS Points Two-view estimate Least squares intersection d is the geodesic distance in rotational space SO(3)

20. 2012 Thrust Area Technical Review Simulation Results 2 quarter circle rows of cameras every 5 degrees Middle camera used as fixed camera Random set of points around origin

21. 2012 Thrust Area Technical Review Simulation Results GPS NoiseFeature Point Noise Location Error from Orientation Noise Orientation Error from Orientation Noise

22. 2012 Thrust Area Technical Review Real World Results Smartphone: Data collected using HTC Evo smartphone Accurate Sensors: Data collected using CyberQuad UAV sensors and camcorder Error Distance compared against GPS/Sensor from data DatasetError DistanceBatchDistributed SmartphoneLocation2.83 m4.21 m Orientation0.23990.0017 Accurate SensorsLocation1.80 m2.001 m Orientation5.14900.0002

23. 2012 Thrust Area Technical Review Smartphone Results 23

24. 2012 Thrust Area Technical Review Accurate Sensor Results

25. 2012 Thrust Area Technical Review Geo-calibration MLE New formulation as a maximum likelihood Maximize the orientation and position of the fixed camera given: –Smartphone orientation position and location –Pairwise relationship based on the image measurements 25

26. 2012 Thrust Area Technical Review Smartphone Measurements Smartphone orientation calculated from compass and gyroscope/accelerometer Smartphone position calculated from GPS with GPS error/accuracy 26

27. 2012 Thrust Area Technical Review Pairwise Measurements Find and match features points (X i, X q ) to find a relative rotation and translation 27 Relative rotation Relative position and unknown depth

28. 2012 Thrust Area Technical Review Geo-calibration Maximum likelihood of probability of fixed cameras orientation and position (R q, t q ) Optimal Estimates 28

29. 2012 Thrust Area Technical Review Consensus Framework This estimate can be solved using a consensus algorithm In a consensus algorithm, each smartphone has a state of the fixed camera’s orientation and location. The states are iteratively updated to a consensus state. 29

30. 2012 Thrust Area Technical Review Geo-calibration Experiments Mount a camera over GPS Benchmarks Collect images and sensor metadata using a smartphone 30

31. 2012 Thrust Area Technical Review Geo-Calibration Summary Technical Report available for initial formulation, Kuo et al. “Map Me: Camera Geo-calibration using mobile devices” Next: Experimentation on New Formulation

32. 2012 Thrust Area Technical Review Project Technical Assesment- 2 Project Introduction Overview of the UCSB Campus Camera Network –Infrastructure funded by a ONR DURIP award –Implementation by students on this project Research Accomplishments –Distributed Geocalibration –Distributed tracking and fusion –Topology discovery and network data summarization –Browsing and search in a distributed network Summary and Conclusions 32

33. 2012 Thrust Area Technical Review Topology discovery Camera placement N+1 camera calibration Geo-calibration Distributed object tracking Object searching Network summarization Wide-area activity analysis Recap: Topic relationships

34. 2012 Thrust Area Technical Review Object Tracking and Search 34 In a smart camera network setup, Distributed multi-camera object tracking –Given a target, how do the network of cameras with overlapping views robustly track the target with collaboration despite limited network bandwidth? Efficient object searching in a large network –How to search for objects of interest efficiently without significant visual processing at search time?

35. 2012 Thrust Area Technical Review Distributed browsing & search Project Technical Assesment

36. 2012 Thrust Area Technical Review Problem: A camera network deployed over a large area No live streaming of any video Local camera nodes have storage to archive video and l imited processing power for simple video analysis How a human image analyst at a distance central node interacts with the remote cameras? 36 ? Browsing and Searching

37. 2012 Thrust Area Technical Review Envision the following application scenarios: –A user instantiates the interaction with the network by specifying regions on the image plane (cameras, time intervals) of interest. E.g., “FIND object instances related to region A FROM camera 1 OR region B FROM camera 4 between time 9:30am and 9:35am” –With the results from the previous scenario, the user could then identify one specific object of interest to initiate further searching for the same or related objects. E.g., “FIND all objects related to the object instance at region C FROM camera 1 at time 9:32:41.3am”. 37 Browsing and Searching

38. 2012 Thrust Area Technical Review Try to provide human users with an high-level interface, such as dynamic global scene visualization and aim to detect/track all observed objects across the entire camera network To deal with appearance variations across views, much prior work focused on finding the best matching criterion (Javed ICCV’03,Javed CVPR’05, Farenzena CVPR’10, Zhen CVPR’11, Rios-Cabrera CVPR’11.) What if we cannot have reliable object detection and tracking? Any possible strategy without pair-wise matching or global trajectory finding? 38 Conventional Approaches

39. 2012 Thrust Area Technical Review Instead of trying to find global trajectories for every object visible in the network, model camera observation directly with a graph model Act as an intermediate agent between distributed cameras and human image analysts and provide recommendations to the user with a concise and representative set of video snapshots captured by the camera network Help the image analysts to browse, search and identify objects of interest by giving a canonical overview of the entire set of visual observations in the network 39 Proposed System

40. 2012 Thrust Area Technical Review 40 Proposed System

41. 2012 Thrust Area Technical Review Real time object detection with background modeling and tracking with mean-shift algorithm An observation record is generated for each detected and tracked object by the camera and sent to the central node over the network 41 Real-time detection & tracking

42. 2012 Thrust Area Technical Review Given a user query, we need to find video frames with following properties: –Centrality, representative ones which are closely related to the query and many other observations and hence considered important. –Diversity, covering as many distinct groups as possible A graph G(V, W) to model relationship among camera observations and perform unified graph ranking for different queries –Individual camera observations (i.e., frames with detected objects) form the vertices set V –Weight matrix W defines the strength of connectivity between camera observations –G(V, W) built at the central server incrementally as the new records are received in real time from the cameras. 42 Modeling camera observations

43. 2012 Thrust Area Technical Review Divide image plane into 8x6 blocks Model the time delay for an object to travel between any two blocks across cameras with a Gaussian model with known mean and variance 43 Spatial Temporal Topology

44. 2012 Thrust Area Technical Review 11 camera nodes (Cisco wireless-G WVC2300) observing bike path Each camera streams video to a dedicated computer to simulate a smart camera node Approx. 600 meters in width/length 44 Demonstration – Test bed

45. 2012 Thrust Area Technical Review 45 A browsing example with regions of interest indicated by the rectangles in camera C8 and C9 Browsing Demonstration

46. 2012 Thrust Area Technical Review 46 Browsing Demonstration Top 10 ranked frames (Decreasing order: left to right, top to down) A total of 10 distinct objects satisfying the criterion. All of them have been identified (labeled in yellow). The 8th ranked frame is a “false positive” (it has not passed the queried regions within the specified time interval).

47. 2012 Thrust Area Technical Review Searching Demonstration Results when searching for object P6 (2 nd -11 th ranked frame)

48. 2012 Thrust Area Technical Review Topology discovery Camera placement N+1 camera calibration Geo-calibration Distributed object tracking Object searching Network summarization Wide-area activity analysis Recap-Topic relationships

49. 2012 Thrust Area Technical Review 49 Forecasted Key Events UAV: Provides intelligence from overhead Rapidly Deployed Smart Camera: Provides ground- based views and distributed image analysis Events in FY 10 Setup of Outdoor Camera Network Quick Chessboard Calibration Preliminary network topology discovery Events in FY11 Network topology discovery Automatic calibration without patterns Beyond 2D tracking Continued multi-camera tracking with overlapping and non-overlapping views. Preliminary distributed processing with existing work Events in FY12 Geo-calibration with mobile sensors Multi-camera multi-object tracking in a wide-area network Browsing and search in a distributed network Soldier image courtesy of US Marines. Map image courtesy of Google Earth.

50. 2012 Thrust Area Technical Review Project Technical Risk Assessment No.Risk Area and DescriptionRisk RatingRisk Reduction ActionsDue Date (Date Completed) (L,M,H) General 1Challenging datasets due to wireless communication constraints L1) Improve antenna and camera placements 2) Robust algorithms 1) 12/2010 2) 09/2012 Calibration and Topology 2Lack of distinctive environmental features for autocalibration M1) Weak scene calibration 2) Explore mobile cameras 1)05/2011 2)05/2011 3Current topology approach may not scale to large separations M1)Explore tracking-based approach 2)Explore mobile sensors 1)11/2010 2)6/2012 Object Detection/Tracking 4Algorithm fails in severe occlusionMExplicit occlusion handling09/2011 Event/Behavior Analysis 5Complexity of the algorithmsMDistribute the algorithm to multiple computing nodes 03/2012 6UAV data exploration: FAA and university regulations HCollect data in remote areas (non- urban) 6/2012 50

51. 2012 Thrust Area Technical Review 51 Summary On target with proposed schedule Setup of campus-wide camera network Quick chessboard calibration Camera network topology discovery Wide-area human mobility patterns Single camera activity discovery Ongoing Geo-Calibration: automatic camera calibration and camera placement Online learning and multi-camera fusion for distributed tracking Multi-camera activity discovery

52. 2012 Thrust Area Technical Review 52 Students/Acknowledgement Zefeng Ni (graduated Fall 2011) –Distributed Tracking, activity analysis Thomas Kuo (expected to graduate in 2012) –Calibration/geo calibration on Carter De Leo JieJun Xu (expected to graduate in 2012) Thanks to ONR/Dr. Martin Kruger for the support and encouragement!

53. 2012 Thrust Area Technical Review 53 BACK-UP SLIDES

54. 2012 Thrust Area Technical Review UAV Data Collection Extending to aerial imagery –Getting aerial data is challenging. –Not easy to combine with ground based network on campus due to university insurance policies (expensive) and unknown FAA regulations concerning UAV research We are working on setting up a network in a nature preserve area where there is no human traffic –Should be less constrained in obtaining UAV data in such areas –Cons: no human activity in such areas. Collect data in remote areas with human actors –Plan to accomplish this before the end of Spring quarter 2012. –Preliminary data already collected. 54