Presentation is loading. Please wait.

Presentation is loading. Please wait.

DDDAMS-based Surveillance and Crowd Control via UAVs and UGVs Sponsor: Air Force Office of Scientific Research under FA9550-12-1-0238 Program Manager:

Similar presentations


Presentation on theme: "DDDAMS-based Surveillance and Crowd Control via UAVs and UGVs Sponsor: Air Force Office of Scientific Research under FA9550-12-1-0238 Program Manager:"— Presentation transcript:

1 DDDAMS-based Surveillance and Crowd Control via UAVs and UGVs Sponsor: Air Force Office of Scientific Research under FA9550-12-1-0238 Program Manager: Dr. Frederica Darema PIs: Young-Jun Son 1, Jian Liu 1, Jyh-Ming Lien 2 Students: A. Khaleghi 1, D. Xu 1, Z. Wang 1, M. Li 1, and C. Vo 2 1 Systems and Industrial Engineering, University of Arizona 2 Computer Science, George Mason University PI Contact: son@sie.arizona.edu; 1-520-9530son@sie.arizona.edu ICCS 2013, June 7, 2013

2 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Motivating Problem: TUCSON-1 (Border Patrol) 23-mile long area surrounding the Sasabe port of entry in AZ Major components: sensor towers, communication towers, ground sensors, UAVs (Arducopter; Parrot AR.Drone), UGVs (sonar, laser), crowds, terrain (NASA) Eagle Vision Ev3000-D-IR CapabilitiesUp to: IR Laser Illuminator3 km Wireless Link50 km IR Thermal Camera Range20 km Uncooled Thermal Camera Range4 km Trident Sentry Node Capabilities Life Span30-45 days Radio Frequency Range300 m Personal Detection and Tacking Range30-50 m

3 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Overview of Project Problem: A highly complex, uncertain, dynamically changing border patrol environment Goal: To create robust, multi-scale, and effective urban surveillance and crowd control strategies using UAV/UGVs –Various modules: detection, tracking, motion planning Approach: Comprehensive planning and control framework based on DDDAMS –Involving integrated simulation environment for hardware, software, human components –Key: What data to use; when to use; how to use [1] Surveillance and Crowd Control Formation control, swarm coordination Target Tracking Vehicle Assignment and Searching Target Detection Surveillance region selection Vehicle motion planning DDDAMS

4 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Proposed DDDAMS Framework

5 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Fidelity Representation via Crowd Dynamics Next waypoints of UGV Low density region invisible by UAV Representation of crowd individuals Crowd view by UAV Crowd view by UGV Coarser visibility cell View of dense crowd regions Finer visibility cell View of individuals Next waypoints of UAVs High density region visible by UAV High fidelity Low fidelity Hybrid fidelity Control command generation for UAV/UGV motion planning

6 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Framework of Proposed Methodology

7 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Detection algorithms (using open source computer vision libraries - OpenCV) –Human detection (upright humans) Histogram of Oriented Gradient (HOG) [5] –Motion detection Background subtraction algorithms [6] Sensors in Parrot AR. Drone 2.0 – Front Camera: CMOS sensor with a 90 degrees angle lens – Ultrasound sensor Emission frequency: 40kHz Range: 6 meters UAV UGV for Crowd Detection

8 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Framework of Proposed Methodology

9 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Crowd Tracking Module UGV Negligible observation error With observation error Low resolution UGV autoregressive model -> forecasting UAV aggregation model -> prediction UGV state space model -> filtering UAV aggregation model -> prediction High Resolution UGV autoregressive model -> forecasting UAV state space model -> filtering UGV state space model -> filtering UAV state space model -> filtering Crowd Dynamics Identification Crowd Motion Prediction UAV

10 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson UAV with Low Resolution C i,j (t): unit cell at the i th row and the j th column at time t. Visibility state: 0 (invisible), 1 (visible). S i,j (t)=[C i-1,j-1 (t), C i-1,j (t), C i-1,j+1 (t), C i,j-1 (t), C i,j (t), C i,j+1 (t), C i+1,j-1 (t), C i+1,j (t), C i+1,j+1 (t)] T. Crowd view by UAV at time t C 3,4 (t)=0, invisible C 2,4 (t)=1, visible S 3,4 (t)=[0,1,1,0,0,0,0,0,0] T C 2,3 (t) C 2,4 (t)C 2,5 (t) C 3,3 (t) C 3,4 (t)C 3,5 (t) C 4,3 (t)C 4,4 (t)C 4,5 (t)

11 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson UAV with Low Resolution – Visibility Prediction Let p s (i,j)=Pr(C i,j (t)=1|S i,j (t-1)=s), s ranging from [0,0,0,0,0,0,0,0,0] T to [1,1,1,1,1,1,1,1,1] T. Predict C i,j (t+1)’s visibility Observation at time t Prediction at time t+1 C 3,4 (t+1)=0 C 3,4 (t+1)=1 If p s (3,4)>0.5 If p s (3,4)<0.5 p s (3,4)=Pr(C 3,4 (t+1)=1|S 3,4 (t)=s),where s=[0,1,1,0,0,0,0,0,0] T Estimate p s (i,j)

12 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Crowd Prediction by UAV Bayesian estimation of p s (i,j) Beta prior: Example: p s (3,4), s=[0,1,1,0,0,0,0,0,0] T Prior knowledge of visibility t t+1 … Total # of observations counts t1t1 t 1 +1 t’ 1 t’ 1 +1 tyty t y +1 t’ N-y t’ N-y +1 … counts Data:

13 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Crowd Prediction by UAV (Cont’d) t+1 Posterior: updated belief of cell (3, 4) being visible. where Based on UAV’s information Prediction at t+2 UAV’s prior UAV’s data Bayesian estimator: expected probability for cell (3, 4) to be visible.

14 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Crowd Prediction by UAV with UGVs’ Information Aggregation Individual dynamics captured by UGVs UGVs’ prediction at t+2 t+1 Combined prediction at t+2 UAV’s prediction at t+2

15 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Crowd Prediction by UAV with UGVs’ Information Aggregation (Cont’d) With aggregated information: where UAV’s information UGV’s information UAV’s informationUGV’s information Balance weight: Close-to-one weight assigned if UGV’s prediction is accurate Close-to-zero weight assigned if UGV’s prediction is inaccurate UGV’s information = predicted individual dynamics by UGVs Crowd Dynamics Identification

16 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Crowd Dynamics Identification - Model State space model for linear dynamic systems : stack up of N I state vector, : state noise : measurement noise locationspeed Gaussian random variable : state transition matrix (interaction among individuals) : stack up of N I observation vector

17 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Crowd Dynamics Identification – Model (Cont’d) State space model for linear dynamic systems : observation matrix (link state with observation) : environmental effect; : environmental factors : effects of UAV/UGVs; : state of UAV/UGVs With UAV/UGV effect Without UGV/UAV effect Functionality of

18 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Tracking Algorithms Objective Identify the crowd motion dynamics and predict crowd location State-of-the-art methods AlgorithmFeatureAdvantageLimitation Kalman Filter [1] Linear dynamic systems Gaussian noise Continuous state space Exact solution Normality and linearity assumptions Extended Kalman filter [2] First-order Taylor series Linearization For nonlinear system Computational complexity Unscented Kalman filter [3] Sampling-based linearization For nonlinear system Uni-modal assumption

19 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Tracking Algorithms (Cont’d) State-of-the-art methods AlgorithmFeatureAdvantageLimitation Switching Kalman filter [] Dynamic switching For nonlinear systems Computationally intensive, approximate solutions Grid-based filter [4] Discrete state space No linear/normality assumption Computationally intensive Particle filter [5] Monte Carlo estimation No linear/normality assumption Computationally intensive

20 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Crowd Dynamics Identification for UGV Assumption: negligible measurement noise in state space model Equivalence to autoregressive model with exogenous input (ARX model [7] ) ARX model

21 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Crowd Dynamics Identification for UGV (Cont’d) ARX Model fitting –Data: most recent observations for each detected individual –Estimation method: ordinary least square When certain cell becomes dense according to prediction Benefit: effective, lower computation cost in identification predicted location at time t+1 Update UAV prior observed location at time t observations up to time t

22 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Framework of Proposed Methodology

23 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Motion Planning Module - Algorithm Graph Search AlgorithmsMethodComplexityFeatures Dijkstra Exact cost from start point to any vertex nhighNot fast enough Best-First Search Estimated cost from vertex n to the end pointLowNot optimal A* Total cost from start point to end pointlow Guarantees the shortest path in a reasonable time

24 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Motion Planning Module - Objectives  Objective 1: Minimize Euclidian traveling distance of UAVs and UGVs  Objective 2: Minimize energy consumption considering UAV’s above ground level (AGL) change and UGV’s downhill/uphill movement where, elevPenalty Angle ( degree ) UGV UAV

25 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Control Strategies minimizing travel time ( ) minimizing energy consumption ( ) minimizing a linear combination of both objectives ( )

26 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Assembled Arduino-based UAV (Arducopter) Specifications Additional payload: up to 1200g Flying time: up to 30 minutes (with one 5000 mAh 11.1V lithium polymer battery) Radio range: approx. 1 mile (more range with amplifier) Air data rates: up to 250kbps Radio Frequency: US  915 Mhz, Europe  433 Mhz Flight test path on Google map Radio Control Training Simulator(AeroSIM ® ) used for practice to minimize risk of damaging real hardware

27 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Agent-based Hardware-in-the-loop Simulation Another simulator

28 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Finite State Automata based UAV/UGV Execution

29 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson UGV Without observation error With observation error Low resolution UGV autoregressive model -> forecasting UAV aggregation model -> prediction UGV state space model -> filtering UAV aggregation model -> prediction High resolution UGV autoregressive model -> forecasting UAV state space model -> filtering UGV state space model -> filtering UAV state space model -> filtering UAV

30 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Experiment – (1) Investigate the effect of detection range on crowd coverage performance Setting –A crowd of 40 individuals –1 UAV, 1 UGV; same detection range –Fixed value for tracking horizon and motion planning frequency Result Explanation –If detection range increases, crowd coverage percentage increases and gets smoother over time Performance Improved significantly

31 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Experiment – (2) Investigate the effect of number of grids on crowd coverage performance (e.g. GPS resolution) Setting –1 UAV, 1 UGV; Detection range is 15m for both UAV and UGV –A crowd of 40 individuals; Result Explanation –Performance fluctuates a lot at number of grids 20*20 –Performance improves and fluctuation diminishes over time as number of grids increases. Performance Improved significantly

32 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Experiment – (3) Investigate different fidelity levels on crowd coverage performance and computational resource usage (e.g. CPU usage) Setting –2 UAVs, 2 UGVs –A crowd of 80 individuals separated as two different groups Result Explanation –High simulation fidelity improve the system performance, but consuming more computational power Low fidelity performance reduced

33 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson UGV Without observation error With observation error Low resolution UGV autoregressive model -> forecasting UAV aggregation model -> prediction UGV state space model -> filtering UAV aggregation model -> prediction High resolution UGV autoregressive model -> forecasting UAV state space model -> filtering UGV state space model -> filtering UAV state space model -> filtering UAV

34 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Numerical Simulation Study Simulation setting –50 individuals with changing dynamics (e.g. changing speed) –One UAV (monitor whole crowd) and two UGVs (track two subgroups of individuals) –UGV tracking: AR model prediction –UAV tracking: proposed Bayesian updating approach –Crowd dynamics Time 1 to 4 Time 5 to 6Time 7 to 15 UGV view range UAV view range

35 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Crowd Tracking Outcome

36 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Crowd Tracking Outcome

37 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson Summary and Ongoing Works DDDAMS-based planning and control framework for surveillance and crowd control via UAVs and UGVs –A series of algorithms for crowd tracking –Motion planning algorithms –Integrated agent-based simulation platform with UAVs/UGVs Ongoing works –Address different control architectures (hierarchical, heterarchical, hybrid) Coordinators: ground controller, UAV (team leader) –Further develop vision-based crowd detection –Enhancement of crowd motion dynamics by extended-BDI –Enhance crowd motion dynamics identification + crowd motion prediction (e.g. multiple visibility level) –Expand system automation test

38 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson QUESTIONS Young-Jun Son; son@sie.arizona.eduson@sie.arizona.edu http://www.sie.arizona.edu/faculty/son/index.html 1-520-626-9530

39 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson References – (1) [1] Darema, F. "Dynamic Data Driven Applications Systems: A New Paradigm for Application Simulations and Measurements.“ International Conference on Computational Science. 2004, pp. 662–669. [2] Hays, R., Singer, M. Simulation fidelity in training system design: Bridging the gap between reality and training. Springer-Verlag, 1989. [3] Celik, N., Lee, S., Vasudevan, K., Son, Y., "DDDAS-based Multi-fidelity Simulation Framework for Supply Chain Systems," IIE Transactions, 2010, 42(5), 325-341. [4] Celik, N., Son, Y.-J., “Sequential Monte Carlo-based Fidelity Selection in Dynamic-data-driven Adaptive Multi-scale Simulations,” International Journal of Production Research, 2012, 50, 843-865. [5] Dalal N., Triggs, B., “Histograms of Oriented Gradients for Human Detection,” Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2005, 886-893. [6] Sheikh, Y., Javed, O., Kanade, T., “Background Subtraction for Freely Moving Cameras,” Proceedings of IEEE 12 th International Conference on Computer Vision, 2009, 1219-1225. [7] Welch, G. and G. Bishop, An introduction to the Kalman filter. 1995. [8] Einicke, G.A.; White, L.B. (September 1999). "Robust Extended Kalman Filtering". IEEE Trans. Signal Processing 47 (9): 2596–2599.

40 Computer Integrated Manufacturing & Simulation Lab Department of Systems and Industrial Engineering, The University of Arizona, Tucson References – (2) [9] Julier, S.J. and J.K. Uhlmann, Unscented filtering and nonlinear estimation. Proceedings of the IEEE, 2004. 92(3): p. 401-422. [10] Pavlovic, V., J.M. Rehg, and J. MacCormick, Learning switching linear models of human motion. Advances in Neural Information Processing Systems, 2001: p. 981-987. [11] Silbert, M., T. Mazzuchi, and S. Sarkani. Comparison of a grid-based filter to a Kalman filter for the state estimation of a maneuvering target. in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series. 2011. [12] Ristic, B., S. Arulampalam, and N. Gordon, Beyond the Kalman filter: Particle filters for tracking applications. 2004: Artech House Publishers. [13] Lütkepohl,H. New Introduction to Multiple Time Series Analysis. Berlin: Springer, 2006. [14] Patel, A. 2003. "Amit's Thoughts on Path-finding- Introduction to A*" Accessed Jun 2, 2013, http://theory.stanford.edu/~amitp/GameProgramming/AStarComparison.html. http://theory.stanford.edu/~amitp/GameProgramming/AStarComparison.html [15] Sinnott, R. W. “Virtues of the Haversine.” Sky telescope, 1984, 68, 159. [16] Moussaïd, M., D. Helbing, and G. Theraulaz. “How Simple Rules Determine Pedestrian Behavior and Crowd Disasters.” In Proceedings of the National Academy of Sciences 2011, 108, No. 17:6884-6888.


Download ppt "DDDAMS-based Surveillance and Crowd Control via UAVs and UGVs Sponsor: Air Force Office of Scientific Research under FA9550-12-1-0238 Program Manager:"

Similar presentations


Ads by Google