Presentation is loading. Please wait.

Presentation is loading. Please wait.

Safe Testing of Autonomy in Complex, Interactive Environments (TACE) Flight Software Workshop December 16 – 18 2014 David Scheidt

Similar presentations

Presentation on theme: "Safe Testing of Autonomy in Complex, Interactive Environments (TACE) Flight Software Workshop December 16 – 18 2014 David Scheidt"— Presentation transcript:

1 Safe Testing of Autonomy in Complex, Interactive Environments (TACE) Flight Software Workshop December 16 – 18 2014 David Scheidt Robert Lutz William D’Amico Subodh Harmalkar

2 2 Outline  Test & Evaluation Need and Challenges  Background and Definitions  Current Effort under TRMC’s T&E/S&T program  Path to a Testing Capability  Summary

3 3 From Tele to Auto to Autonomous Operations  Unmanned systems usually require the operator to be a “user-in-the-loop”  APL has demonstrated the autonomous operation of multiple and cooperative UXVs where the operator is a “user-on-the-loop”  There have been many demonstrations of APL’s Mission Level Autonomy (MLA) under JHU/APL internal research and at Camp Roberts (CA)  MLA has been shown to be “platform” and “vendor” agnostic riding above proprietary control/guidance algorithms  MLA research led to insights on what would be needed for rigorous test and evaluation UXV SensorControl Autonomous Operations Operational objectives Measure effect Measure effect Operating environmental decisions A measure of effectiveness of an autonomous system is directly related to “operational objectives” thus making the test methodology and test results applicable to both developmental and operational test environments – true joint DT/OT is needed

4 4 Systems with Autonomous Behaviors Are Here ACTUV: ANTI-SUBMARINE WARFARE CONTINUOUS TRAIL UNMANNED VESSEL AMAS: Autonomous Mobility Appliqué System AACUS: Autonomous Aerial Cargo/Utility System LDUUV: Large Diameter UUV

5 5 Autonomy Test & Evaluation Challenge Autonomous systems respond to unpredictable change by devising a course of action. Before we deploy such systems how can we be sure that autonomous decisions that will be produced will always: (1) achieve objectives set by human supervisors and (2) not produce unacceptable unintended consequences. Testing autonomous systems is particularly challenging since we cannot possibly test all interactions between the autonomous system and the natural world.

6 6 197019902000 Air-Exjam (later EXDRONE and BMQ-147 Dragon) Scheer “Beast” Autonomy S&T Vision Program Predator/SSN Interoperability Demo NEAR burn failureDARPA UUV Program History of Autonomous Unmanned Systems at APL 196520102005 4 UGVs @ APG 2 UAVs & 4 UGVs @ APG 1 UAV 1 UGS @ APG Tactical Sensing 1 UAV & Multiple UGS Unmanned Surface Vehicles 3 UAVs, 2 UGS, 3 Mobile Users @ Dugway & DOE Range TNT Experiments 6 UAVs, 3 UGS @ Camp Roberts 1 UUV 2015 OPISR – 4 UAVs, 1 UGV, 2 USV, 1 UUV, 3 UGS @ Webster Small Ocean- going USVs In Atlantic, Pacific & Gulf of Mexico JHU/APL has been developing autonomous unmanned vehicles since 1962. Since 2002 JHU/APL has conducted dozens of autonomous vehicle flight programs and flown/launched hundreds of sorties. Currently JHU/APL has 24 active autonomy programs. JHU/APL Proprietary New Horizons TACE DADFS Agile UxV Autonomy

7 7 Combined Autonomous Air/Ground Missions 2004 Aberdeen Test Center OSD NII Swarming Unmanned Vehicle Experiment – Cooperative Search, Patrol & Track (4 UGVs and 2 UAV) Combinations of autonomous air and ground vehicles (an AACUS/AMAS collaboration) are clearly possible and probably required for many military missions

8 8 Challenges for Autonomous System Test Test & Evaluation (T&E) Need -Build an infrastructure for SAFE/LIVE testing of autonomous unmanned vehicles (AUVs), especially for tactical UAVs that have significant airspace and platform- related range safety issues -Demonstrate live, virtual, and constructive (LVC) methods for SAFE/LIVE tests -LVC methods should be Test and Training Network Architecture (TENA) compliant -To our knowledge there are no AUV established T&E capabilities with these features T&E Challenge –Provide an infrastructure that supports safe testing when testing autonomous systems operating over the horizon or in denied environments. –Provide an infrastructure that supports safe testing when autonomous systems perform unpredictable actions –Provide an accurate, real-time, live-virtual-constructive environment that interacts with the system under test’s autonomy in unpredictable ways. –Reliable command and control (C2) of the systems under test (SUT) and robust 2-way data links are needed to stimulate the SUT, to record behaviors, and to maintain SAFE/LIVE test control

9 9 Safe Testing of Autonomy in Complex, Interactive Environments (TACE) – Key Elements  Capability  TACE is a 3-phase/36-month program (started in April 2013) where the open/general architecture will be at TRL6 with initial capabilities for the testing (black and white box) of autonomous unmanned vehicles (AUVs) of all types. TACE is “portable” but will rely upon a “thin client” interface to the SUT and the inclusion on the SUT of a minimal “TACE test applique.” There are 3 levels of control/override within TACE: SUT autonomy, TACE automatic experiment controls, test range safety officer control  Relevance to T&E Scenarios  The TACE architecture will use rigorous techniques to prevent unsafe AUV operations or actions while stimulating, measuring, monitoring, and recording the AUV’s autonomous response/performance in complex environments. TACE assurance algorithms provide mathematically rigorous, on-board, real-time, safety guarantees, while TACE dual simulation system provides a complex live-virtual constructive test environment.  Cost Benefits  A critical element, as always, is selecting the critical tests. TACE is not a planning tool. A recent award was made to JHU/APL under the Unmanned and Autonomous System Test (UAST) program for an autonomy planning tool, Rapid Adversarial Planning Tool (RAPT). The “cost” of conducting tests where the “Achilles heel” of the embedded autonomous behavior is unknown would involve very high risk to the loss of the SUT and the delay of system fielding. The combination of tools such as RAPT and TACE should prevent unnecessary and expensive testing

10 10 TACE System Architecture

11 11 TACE System Under Test (SUT )Payload TACE’s payload for the Boeing ScanEagle is a repurposing of JHU/APL’s Autonomy Toolkit (ATK) software, which has been used on more than a dozen types of UAVs in Boeing, Lockheed, Aerovironment manufactured vehicles, and hardware previously developed jointly under internal research funds by Boeing and JHU/APL TACE SUT payload installed in a Boeing ScanEagle payload bay ATK payload with Persistent Systems Wave Relay Wireless Local Area Network Card

12 12 TACE Flight Tests at Aberdeen Test Center APL Test Team on the tarmac at Phillips Army Airfield (PAAF) Aberdeen Test Center (ATC) Hand launch of the Procerus research AUV controlled by JHU/APL’s Autonomy Tool Kit (ATK) Five Test Events with Multiple Sorties Were Executed during January/February 2014

13 13 Sample Results from TACE Testing at ATC Constraint Violation - Range Safety Executive User Interface SUT Moving NoGo Virtual Target Safe Loiter Point Fixed NoGo Area

14 14 Path to a Testing Capability – Phase 2 OV1 Test Director Test Ground Station Intruder C2 Restricted Airspace(s) Unicorn UAV Range Safety ScanEagle SUT-System Under Test Native UHF Link Wave Relay 2400MHz Live target track to SUT Intruder GS UHF Safety Link Land Lines Onboard Test Payload Autonomous Behaviors Search, Track, Avoid Watchdog Behaviors 3D Range Boundaries Platform Safety Limits Virtual JIMM Entities 1 to N Virtual Translated Air Traffic Live Unicorn UAV Defense Research & Engineering Network Connection Tactical EO/IR UHF PNT Downlink Test Range Topology ScanEagle as the SUT at Yuma Test Center

15 15 10 Miles 1.0 Mile 2.5 Miles 7.5 Miles Geographic Constraint Boundaries 1.0 Mile 1.0 Mile 2.0 Miles Notional Playbox at a Large Test Range Flight Test Operations Base

16 16 Proximity Violation (Virtual Entity) Track Established Violation Live Ground Entity Virtual Aircraft Note: paths shown are not actual paths)

17 17 Summary for Autonomy and T&E Perspectives  TACE Phase 3 will provide some basic operational functions at TRL 6 – this is an “initial architecture” but not a “range capability”  To our knowledge, there are no “active” programs to extend beyond TRL6 – highly understandable since there are no PORs with autonomous behaviors  At the recent NDIA Annual T&E conference – SHIFT LEFT is coming  RFPs will include CONOPS and initial TEMPs - will “autonomy” be a “shift left” –  Insights from Dr. Brown, Director TRMC - cited “cyber, big data, hypersonic, and autonomy” as the T&E challenge areas for the near future  The NATO Science and Technology Organization (STO) promotes and... Validation and Verification of Autonomous Systems (SCI-274)

18 18Acknowledgements JHU/APL TACE Team: Robert Chalmers, Robert Bamberger, Kristine Ramachandran, Dean Kleissas, Brendan John, Subodh Harmalkar, William Van Besien, Michael Biggins TACE Subcontractors to JHU/APL : – Trideum: Michael O’Connor – Boeing Corporation: Gabriel Santander – InSitu Inc.: Jerry McWithey Government Flight Test Support: – Aberdeen Test Center: John Wiley – Yuma Test Center: Mary Beth Weaver and Bob Vondell Unmanned and Autonomous Systems Test Test Technology Area – Executive Agent: Vernon Panei – Deputy Executive Agent: Stephanie Riddle – Subject Matter Experts: William Hamel, Kirk Bonnevier


Download ppt "Safe Testing of Autonomy in Complex, Interactive Environments (TACE) Flight Software Workshop December 16 – 18 2014 David Scheidt"

Similar presentations

Ads by Google