Reading Notes: Special Issue on Distributed Smart Cameras, Proceedings of the IEEE Mahmut Karakaya Graduate Student Electrical Engineering and Computer Science University of Tennessee, Knoxville

Outline Introduction to Visual Sensor Networks (VSNs) Limitations of Visual Sensor Platforms Applications of VSN Challenges in Visual Sensors Trends in VSN Basic Concepts for Target Detection, Tracking and Recognition Camera Calibration with Belief Propagation Overview of papers from Proceedings of the IEEE, 96(10), October 2008 “An Introduction to Distributed Smart Cameras” by Rinner and Wolf. “Object Detection, Tracking and Recognition for MSCs” by Sankaranarayanan, Veeraraghavan, Chellappa. “Calibrating Distributed Camera Networks” by Devarajan, Cheng, Radke

Introduction to Visual Sensor Networks What is MSP?  Visual sensing  Computing  Wireless comm.  Collaborative visual computing  Centralized  Distributed What is VSN?  Collaboration in VSN:  To compensate for the limitations of each sensor node,  To improve the accuracy and robustness of the sensor network. * Figure Courtesy of [1]

Limitations of Visual Sensor Platforms SMALL SIZE, LOW POWER, LOW-COST = Low-processing speed Microprocessors and FPGA Small memory space Huge data volume i.e. image/video Bottleneck of system performance Low-bandwidth communication, Low level comm. protocols, ZigBee. $(Communication) > 1000x$(Computation) Scarce energy sources. Powered by battery.

Applications  Intelligent Video Surveillance Systems: Detect abnormal behaviors in the observed scene.  Intelligent Transportation Systems: Traffic monitoring, intelligent automobiles (inside and outside of the vehicle).  Medicine: Monitor patients, to see live retinal images during surgery.  Entertainment and Smart Environments: Gesture recognition.

Limited Field of View (FOV) Directional sensing Angle of view of normal lens 25 o -50 o. Visual Occlusion: Static and Dynamic Capture a target only when It stands in the field of view, No other occluding targets Challenges in Visual Sensors * Figure Courtesy of [1]

Trends in VSN From Static to Dynamic and Adaptive: To execute adaptive algorithms in order to better account for changes in the observed scene, To exploit static and mobile cameras such as PTZ cameras, To change camera functionality to provide increased autonomy. From Small to Very Large Camera Sets: Consider new types of information that to extract from the cameras. From Vision-Only to Multi-sensor Systems: Integrate different sensors i.e. audio, seismic, thermal Exploits the distinct characteristics of the individual sensors resulting in an enhanced overall output.

Distributed Computing & Distributed Camera Networks Distributed computing: a network of processors, not have direct knowledge of the state of other nodes. Distributed algorithms are designed to minimize the number of messages required to complete the algorithm. Processing all the data centrally poses several problems because of the data volume so process it locally before transmission. Data only go to the necessary nodes. Partition the network, so that use bandwidth efficiently. Real-time considerations: A distributed system can make sure only relevant nodes are involved in a given decision.

Basic Concepts for Target Detection and Tracking in VSN Central projection: Map the targets from real-world to image plane by using the projection matrix. Epipolar Geometry: Epipolar lines between focal point of two cameras and a pixel-point object are used to obtain correspondence across multiple views. Triangulation: The correspondences between multiple views is be utilized to localize targets. * Figure Courtesy of [2]

Features Extraction for Target Recognition in VSN Global properties: i.e. shape, color, texture Very sensitive to external conditions such as lighting, pose, viewpoint etc. Local features: i.e. discriminative points Discriminative points are chosen by using feature detector methods i.e. Harris corner detector and scale-invariant feature transform (SIFT). The object is defined by using these local descriptors which are very robust to viewpoint and illumination.

Geometry of the Camera & Parameters Typically, a camera is described by two sets of parameters:  The Internal parameters  The focal length  Position of principal points  The skew.  The external parameters describe the placement of the camera in a world coordinate system using  The rotation matrix and  The translation vector. * Figure Courtesy of Satya Prakash Mallick

Neighborhood Clustering Figure: (a) A snapshot of the instantaneous state of a camera network, indicating the fields of view of ten cameras. (b) The associated communication graph. (c) The associated vision graph Vision Graph: Detect the feature points in the image, matching the detected features through the neighbor nodes in vision graph. * Figure Courtesy of [3]

Feature Point Detection Figure: (a) Original image, with detected feature points overlaid. (b) Top view of the reconstructed 3-D scene and camera configuration for the experiment. Scale-invariant feature transform (SIFT): determines the “blobs” by processing the image at multiple scales with difference of Gaussian (DoG) filter and returning the scale-space extreme points of the result.

Belief Propagation Each camera node estimates the calibration parameters of neighboring cameras and updates them in a distributed algorithm to reach the accurate calibration. Summary of BP For each node Forms a neighborhood cluster, independently Performs the local calibration, based on common scene points. Calibrated nodes and scene points are incrementally merged into a common coordinate frame.

Questions THANK YOU…..

References [1] Rinner, B.; Wolf, W. “An Introduction to Distributed Smart Cameras” Proceedings of the IEEE, 96(10), p October 2008 [2] Sankaranarayanan, A.C.; Veeraraghavan, A.; Chellappa, R. “Object Detection, Tracking and Recognition for Multiple Smart Cameras” Proceedings of the IEEE, 96(10), p , October 2008 [3] Devarajan, D.; Zhaolin Cheng; Radke, R.J. “Calibrating Distributed Camera Networks” Proceedings of the IEEE, 96(10), p , October 2008