Presentation is loading. Please wait.

Presentation is loading. Please wait.

DDDAMS-based Surveillance and Crowd Control via UAVs and UGVs

Similar presentations

Presentation on theme: "DDDAMS-based Surveillance and Crowd Control via UAVs and UGVs"— Presentation transcript:

1 DDDAMS-based Surveillance and Crowd Control via UAVs and UGVs
Sponsor: Air Force Office of Scientific Research under FA Program Manager: Dr. Frederica Darema PIs: Young-Jun Son1, Jian Liu1, Jyh-Ming Lien2 Students: A. Khaleghi1, D. Xu1, Z. Wang1, M. Li1, and C. Vo2 1Systems and Industrial Engineering, University of Arizona 2Computer Science, George Mason University PI Contact: ICCS 2013, June 7, 2013

2 Motivating Problem: TUCSON-1 (Border Patrol)
23-mile long area surrounding the Sasabe port of entry in AZ Major components: sensor towers, communication towers, ground sensors, UAVs (Arducopter; Parrot AR.Drone), UGVs (sonar, laser), crowds, terrain (NASA) Eagle Vision Ev3000-D-IR Capabilities Up to: IR Laser Illuminator 3 km Wireless Link 50 km IR Thermal Camera Range 20 km Uncooled Thermal Camera Range 4 km Trident Sentry Node Capabilities Life Span 30-45 days Radio Frequency Range 300 m Personal Detection and Tacking Range 30-50 m

3 Surveillance and Crowd Control Vehicle motion planning
Overview of Project Problem: A highly complex, uncertain, dynamically changing border patrol environment Goal: To create robust, multi-scale, and effective urban surveillance and crowd control strategies using UAV/UGVs Various modules: detection, tracking, motion planning Approach: Comprehensive planning and control framework based on DDDAMS Involving integrated simulation environment for hardware, software, human components Key: What data to use; when to use; how to use[1] Surveillance and Crowd Control Formation control, swarm coordination Target Tracking Vehicle Assignment and Searching Target Detection Surveillance region selection Vehicle motion planning DDDAMS

4 Proposed DDDAMS Framework

5 Fidelity Representation via Crowd Dynamics
Crowd view by UAV Coarser visibility cell View of dense crowd regions Representation of crowd individuals Low fidelity Low density region invisible by UAV High density region visible by UAV Crowd view by UGV Finer visibility cell View of individuals Next waypoints of UGV Control command generation for UAV/UGV motion planning Hybrid fidelity High fidelity Next waypoints of UAVs

6 Framework of Proposed Methodology

7 UAV UGV for Crowd Detection
Detection algorithms (using open source computer vision libraries - OpenCV) Human detection (upright humans) Histogram of Oriented Gradient (HOG)[5] Motion detection Background subtraction algorithms[6] Sensors in Parrot AR. Drone 2.0 Front Camera: CMOS sensor with a degrees angle lens Ultrasound sensor Emission frequency: 40kHz Range: 6 meters

8 Framework of Proposed Methodology

9 Negligible observation error
Crowd Tracking Module Crowd Dynamics Identification Crowd Motion Prediction UGV Negligible observation error With observation error Low resolution UGV autoregressive model -> forecasting UAV aggregation model -> prediction UGV state space model -> filtering High Resolution UGV autoregressive model -> forecasting UAV state space model -> filtering UAV

10 UAV with Low Resolution
Ci,j(t): unit cell at the ith row and the jth column at time t. Visibility state: 0 (invisible), 1 (visible). Si,j(t)=[Ci-1,j-1(t), Ci-1,j(t), Ci-1,j+1(t), Ci,j-1(t), Ci,j(t), Ci,j+1(t), Ci+1,j-1(t), Ci+1,j(t), Ci+1,j+1(t)]T. C2,3(t) C2,4(t) C2,5(t) C3,3(t) C3,4(t) C3,5(t) C4,3(t) C4,4(t) C4,5(t) C2,4(t)=1, visible C3,4(t)=0, invisible S3,4(t)=[0,1,1,0,0,0,0,0,0]T Crowd view by UAV at time t

11 UAV with Low Resolution – Visibility Prediction
Let ps(i,j)=Pr(Ci,j(t)=1|Si,j(t-1)=s), s ranging from [0,0,0,0,0,0,0,0,0]T to [1,1,1,1,1,1,1,1,1]T. Predict Ci,j(t+1)’s visibility Estimate ps(i,j) C3,4(t+1)=0 If ps(3,4)<0.5 ? C3,4(t+1)=1 If ps(3,4)>0.5 Observation at time t Prediction at time t+1 ps(3,4)=Pr(C3,4(t+1)=1|S3,4(t)=s),where s=[0,1,1,0,0,0,0,0,0]T

12 Crowd Prediction by UAV
Bayesian estimation of ps(i,j) Example: ps(3,4), s=[0,1,1,0,0,0,0,0,0]T t t+1 Beta prior: Prior knowledge of visibility Data: Total # of observations t1 t1+1 ty ty+1 t’1 t’1+1 t’N-y t’N-y+1 counts counts

13 Crowd Prediction by UAV (Cont’d)
Posterior: updated belief of cell (3, 4) being visible. where Based on UAV’s information t+1 UAV’s prior UAV’s data UAV’s data UAV’s prior Prediction at t+2 Bayesian estimator: expected probability for cell (3, 4) to be visible.

14 Crowd Prediction by UAV with UGVs’ Information Aggregation
Individual dynamics captured by UGVs UGVs’ prediction at t+2 t+1 UAV’s prediction at t+2 Combined prediction at t+2 t+1

15 Crowd Prediction by UAV with UGVs’ Information Aggregation (Cont’d)
With aggregated information: where UGV’s information UAV’s information UGV’s information UAV’s information Balance weight: Close-to-one weight assigned if UGV’s prediction is accurate Close-to-zero weight assigned if UGV’s prediction is inaccurate UGV’s information = predicted individual dynamics by UGVs Crowd Dynamics Identification

16 Crowd Dynamics Identification - Model
State space model for linear dynamic systems : stack up of NI state vector, : state noise : measurement noise location speed : state transition matrix (interaction among individuals) : stack up of NI observation vector Gaussian random variable

17 Crowd Dynamics Identification – Model (Cont’d)
State space model for linear dynamic systems : observation matrix (link state with observation) : environmental effect; : environmental factors : effects of UAV/UGVs; : state of UAV/UGVs Functionality of With UAV/UGV effect Without UGV/UAV effect

18 Tracking Algorithms Objective State-of-the-art methods
Identify the crowd motion dynamics and predict crowd location State-of-the-art methods Algorithm Feature Advantage Limitation Kalman Filter[1] Linear dynamic systems Gaussian noise Continuous state space Exact solution Normality and linearity assumptions Extended Kalman filter[2] First-order Taylor series Linearization For nonlinear system Computational complexity Unscented Kalman filter[3] Sampling-based linearization Uni-modal assumption

19 Tracking Algorithms (Cont’d)
State-of-the-art methods Algorithm Feature Advantage Limitation Switching Kalman filter[] Dynamic switching For nonlinear systems Computationally intensive, approximate solutions Grid-based filter[4] Discrete state space No linear/normality assumption Computationally intensive Particle filter[5] Monte Carlo estimation

20 Crowd Dynamics Identification for UGV
Assumption: negligible measurement noise in state space model Equivalence to autoregressive model with exogenous input (ARX model[7]) ARX model

21 Crowd Dynamics Identification for UGV (Cont’d)
ARX Model fitting Data: most recent observations for each detected individual Estimation method: ordinary least square When certain cell becomes dense according to prediction Benefit: effective, lower computation cost in identification observed location at time t observations up to time t predicted location at time t+1 Update UAV prior

22 Framework of Proposed Methodology

23 Motion Planning Module - Algorithm
Find the optimal path, given a map, start location & destination A grid is constructed based on physical obstacles in the environment for vehicle movements 0 for empty grids 1 for occupied grids A* algorithm with 8- point connectivity used, considering two objectives for UAV ( 𝑓 1 𝐴 , 𝑓 2 𝐴 ) and UGV ( 𝑓 1 𝐺 , 𝑓 2 𝐺 ) Graph Search Algorithms Method Complexity Features Dijkstra Exact cost from start point to any vertex n high Not fast enough Best-First Search Estimated cost from vertex n to the end point Low Not optimal A* Total cost from start point to end point low Guarantees the shortest path in a reasonable time

24 Motion Planning Module - Objectives
Objective 1: Minimize Euclidian traveling distance of UAVs and UGVs Objective 2: Minimize energy consumption considering UAV’s above ground level (AGL) change and UGV’s downhill/uphill movement where, 𝑓 1 𝐴 = 𝑡= 𝑡 0 𝑇−1 [ 𝑥 𝐴 𝑡+1 − 𝑥 𝐴 𝑡 𝑦 𝐴 𝑡+1 − 𝑦 𝐴 𝑡 𝑧 𝐴 𝑡+1 − 𝑧 𝐴 𝑡 𝑓 1 𝐺 = 𝑡= 𝑡 0 𝑇−1 [ 𝑥 𝐺 𝑡+1 − 𝑥 𝐺 𝑡 𝑦 𝐺 𝑡+1 − 𝑦 𝐺 𝑡 𝑒𝑙𝑒𝑣( 𝑥 𝐺 𝑡+1 , 𝑦 𝐺 𝑡+1 )−𝑒𝑙𝑒𝑣( 𝑥 𝐺 𝑡 , 𝑦 𝐺 𝑡 ) 2 } 0.5 𝑓 2 𝐴 = 𝑡= 𝑡 0 𝑇−1 𝑒𝑙𝑒𝑣𝑃𝑒𝑛𝑎𝑙𝑡𝑦 𝑎𝑛𝑔𝑙𝑒 ∗ [ 𝑥 𝐴 𝑡+1 − 𝑥 𝐴 𝑡 𝑦 𝐴 𝑡+1 − 𝑦 𝐴 𝑡 𝑧 𝐴 𝑡+1 − 𝑧 𝐴 𝑡 𝑓 2 𝐺 = 𝑡= 𝑡 0 𝑇−1 𝑒𝑙𝑒𝑣𝑃𝑒𝑛𝑎𝑙𝑡𝑦 𝑎𝑛𝑔𝑙𝑒 ∗ [ 𝑥 𝐺 𝑡+1 − 𝑥 𝐺 𝑡 𝑦 𝐺 𝑡+1 − 𝑦 𝐺 𝑡 elev( 𝑥 𝐺 t+1 , 𝑦 𝐺 t+1 )−elev( 𝑥 𝐺 t , 𝑦 𝐺 t ) 2 } 0.5 elevPenalty Angle (degree) UGV UAV [ 𝑥 𝐴 t , 𝑦 𝐴 t , 𝑧 𝐴 t ]: Cartesian coordinates of UAV at time t [ 𝑥 𝐺 t , 𝑦 𝐺 t ]: Cartesian coordinates of UGV at time t elev[ 𝑥 𝐺 t , 𝑦 𝐺 t ]: terrain elevation at time t angle= arctan 𝑒𝑙𝑒𝑣 𝑡+1 −𝑒𝑙𝑒𝑣(𝑡) 𝑥 𝐺 𝑡+1 − 𝑥 𝐺 𝑡 𝑦 𝐺 𝑡+1 − 𝑦 𝐺 𝑡

25 Control Strategies minimizing a linear combination of both objectives ( ) minimizing travel time ( ) minimizing energy consumption ( ) min 𝛼 1 𝑓 1 + 𝛼 2 𝑓 2 Multi-objective optimization for the evaluation function of various paths based on control strategies Weighting method for multi-objective motion planning problem with Normalize each objective 𝑓 𝑘 (𝑥) as 𝑓 𝑘 (𝑥) Compute the convex combination of all objectives using weights 𝛼 𝑘 for 𝑓 𝑘 (𝑥) inf 𝑓 𝑘 (𝑥) = 𝑚 𝑘 sup 𝑓 𝑘 (𝑥) = 𝑀 𝑘

26 Assembled Arduino-based UAV (Arducopter)
Specifications Additional payload: up to 1200g Flying time: up to 30 minutes (with one 5000 mAh 11.1V lithium polymer battery) Radio range: approx. 1 mile (more range with amplifier) Air data rates: up to 250kbps Radio Frequency: US915 Mhz, Europe433 Mhz Radio Control Training Simulator(AeroSIM®) used for practice to minimize risk of damaging real hardware Flight test path on Google map

27 Agent-based Hardware-in-the-loop Simulation
Another simulator

28 Finite State Automata based UAV/UGV Execution

29 Without observation error With Low resolution High
UGV Without observation error With Low resolution UGV autoregressive model -> forecasting UAV aggregation model -> prediction UGV state space model -> filtering High UGV autoregressive model -> forecasting UAV state space model -> filtering UAV

30 Performance Improved significantly
Experiment – (1) Investigate the effect of detection range on crowd coverage performance Setting A crowd of 40 individuals 1 UAV, 1 UGV; same detection range Fixed value for tracking horizon and motion planning frequency Result Explanation If detection range increases, crowd coverage percentage increases and gets smoother over time Performance Improved significantly

31 Performance Improved significantly
Experiment – (2) Investigate the effect of number of grids on crowd coverage performance (e.g. GPS resolution) Setting 1 UAV, 1 UGV; Detection range is 15m for both UAV and UGV A crowd of 40 individuals; Result Explanation Performance fluctuates a lot at number of grids 20*20 Performance improves and fluctuation diminishes over time as number of grids increases. Performance Improved significantly

32 Experiment – (3) Investigate different fidelity levels on crowd coverage performance and computational resource usage (e.g. CPU usage) Setting 2 UAVs, 2 UGVs A crowd of 80 individuals separated as two different groups Result Explanation High simulation fidelity improve the system performance, but consuming more computational power Low fidelity performance reduced

33 Without observation error With Low resolution High
UGV Without observation error With Low resolution UGV autoregressive model -> forecasting UAV aggregation model -> prediction UGV state space model -> filtering High UGV autoregressive model -> forecasting UAV state space model -> filtering UAV

34 Numerical Simulation Study
Simulation setting 50 individuals with changing dynamics (e.g. changing speed) One UAV (monitor whole crowd) and two UGVs (track two subgroups of individuals) UGV tracking: AR model prediction UAV tracking: proposed Bayesian updating approach Crowd dynamics UGV view range Time 1 to 4 Time 5 to 6 Time 7 to 15 UAV view range

35 Crowd Tracking Outcome

36 Crowd Tracking Outcome

37 Summary and Ongoing Works
DDDAMS-based planning and control framework for surveillance and crowd control via UAVs and UGVs A series of algorithms for crowd tracking Motion planning algorithms Integrated agent-based simulation platform with UAVs/UGVs Ongoing works Address different control architectures (hierarchical, heterarchical, hybrid) Coordinators: ground controller, UAV (team leader) Further develop vision-based crowd detection Enhancement of crowd motion dynamics by extended-BDI Enhance crowd motion dynamics identification + crowd motion prediction (e.g. multiple visibility level) Expand system automation test

38 QUESTIONS Young-Jun Son;

39 References – (1) [1] Darema, F. "Dynamic Data Driven Applications Systems: A New Paradigm for Application Simulations and Measurements.“ International Conference on Computational Science. 2004, pp. 662–669. [2] Hays, R., Singer, M. Simulation fidelity in training system design: Bridging the gap between reality and training. Springer-Verlag, [3] Celik, N., Lee, S., Vasudevan, K., Son, Y., "DDDAS-based Multi-fidelity Simulation Framework for Supply Chain Systems," IIE Transactions, 2010, 42(5), [4] Celik, N., Son, Y.-J., “Sequential Monte Carlo-based Fidelity Selection in Dynamic-data-driven Adaptive Multi-scale Simulations,” International Journal of Production Research, 2012, 50, [5] Dalal N., Triggs, B., “Histograms of Oriented Gradients for Human Detection,” Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2005, [6] Sheikh, Y., Javed, O., Kanade, T., “Background Subtraction for Freely Moving Cameras,” Proceedings of IEEE 12th International Conference on Computer Vision, 2009, [7] Welch, G. and G. Bishop, An introduction to the Kalman filter [8] Einicke, G.A.; White, L.B. (September 1999). "Robust Extended Kalman Filtering". IEEE Trans. Signal Processing 47 (9): 2596–2599.

40 References – (2) [9] Julier, S.J. and J.K. Uhlmann, Unscented filtering and nonlinear estimation. Proceedings of the IEEE, (3): p [10] Pavlovic, V., J.M. Rehg, and J. MacCormick, Learning switching linear models of human motion. Advances in Neural Information Processing Systems, 2001: p [11] Silbert, M., T. Mazzuchi, and S. Sarkani. Comparison of a grid-based filter to a Kalman filter for the state estimation of a maneuvering target. in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series [12] Ristic, B., S. Arulampalam, and N. Gordon, Beyond the Kalman filter: Particle filters for tracking applications. 2004: Artech House Publishers. [13] Lütkepohl,H. New Introduction to Multiple Time Series Analysis. Berlin: Springer, [14] Patel, A "Amit's Thoughts on Path-finding- Introduction to A*" Accessed Jun 2, 2013, [15] Sinnott, R. W. “Virtues of the Haversine.” Sky telescope, 1984, 68, 159. [16] Moussaïd, M., D. Helbing, and G. Theraulaz. “How Simple Rules Determine Pedestrian Behavior and Crowd Disasters.” In Proceedings of the National Academy of Sciences 2011, 108, No. 17:

Download ppt "DDDAMS-based Surveillance and Crowd Control via UAVs and UGVs"

Similar presentations

Ads by Google