1. North Carolina Agricultural and Technical State University Explore. Discover. Become. Ali Karimoddini, PhD Autonomous Cooperative Control of Emergent Systems of Systems (ACCESS) Lab, Director TECHLAV Center, Deputy Director and leader of Research Thrust 2 Department of Electrical and Computer Engineering North Carolina A&T State University 1601 E. Market Street/524 McNair Hall Greensboro, NC 27411 Email: akarimod@ncat.eduakarimod@ncat.edu Website: http://eceserver.ncat.edu/akarimod/http://eceserver.ncat.edu/akarimod/ Office: 336-285-3847 Fax: 336-334-7716 DoD 2015 “Taking the Pentagon to the People” HBCU/MI Technical Assistance Training Greensboro, NC 8 June 2015 Developing a data-driven Perception Inference Engine (PIE) for Test & Evaluation of autonomous systems

2. North Carolina Agricultural and Technical State University 2 Remark: The views and conclusions being discussed here are those of the panelist and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DoD, TRMC, or the U.S. Government. Focused on the mission of the Test Resource Management Center (TRMC) to address T&E needs of Department of Defense (DoD), ACIT Institute has developed a novel data- driven technique for test and evaluation of autonomous systems using an advance fuzzy expert system. A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

3. North Carolina Agricultural and Technical State University A sad moment … 3 On October 28, 2014, the first stage of an Antares rocket on an unmanned resupply mission carrying Cygnus CRS Orb-3 failed catastrophically six seconds after liftoff from Mid-Atlantic Regional Spaceport at Wallops Flight Facility, Virginia. The flight termination system was activated just before the rocket hit the ground, but an explosion and fire destroyed the vehicle and cargo. There were no casualties, and Launch Pad 0A escaped significant damage. A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

4. North Carolina Agricultural and Technical State University Motivations Testing of Unmanned Systems is required for the Military Departments to be able to certify compliance with regulations and demonstrate safe operations. Unmanned Systems must meet the same requirements of a manned systems that is intended to be put into service. Challenge: Testing unmanned systems in general is a significant challenge and can be very costly. 4 A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

5. North Carolina Agricultural and Technical State University What is a right key? Problem Solution Testing simple systems Testing Software Testing complex systems Set of experiments Software model checking Formal verifcation Data driven techniques Pros SimplicityPowerful for testing software Guarantee the performance Capture complexity and unmodelled behaviors Cons Not scalable and not expandable for complex systems Specific to software and difficult to be used for hardware testing Not applicable to complex systems with unmodelled behaviors Only valid for the trained range Model based algorithmic testing 5 A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

6. North Carolina Agricultural and Technical State University Sources of complexity: Cyber-Physical nature Cyber-physical systems (CPS) are engineered systems with tight combination of (large number of) interacting computational systems and physical processes. Control Communication Computation 6 A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

7. North Carolina Agricultural and Technical State University Project goals: Project goals: Developing a Data-driven Perception Inference Engine (PIE) tool to 3- Predict behavior and evaluate the performance of increasingly intelligent systems 4- Capture the dynamic, non-deterministic, uncertain behavior of intelligent, autonomous systems 2- Evaluate intelligent systems from a cognitive perspective 1- Infer the internal states of the system from external observations only 7 A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

8. North Carolina Agricultural and Technical State University Assumptions / Challenges Assumptions: Testers may have only limited knowledge of the internal states of the system under test, but can externally observe the behavior of the system Challenges: How to infer the internal states and dynamics of the system from only external observations and how to use this information to evaluate the performance of the system. 8 A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

9. North Carolina Agricultural and Technical State University Our approach Approach: Creating a tool to enable users to predict the system’s perception, decision-making, and behaviors by integrating Type 2 Fuzzy Logic System (FLS) Learning Classifier Systems (LCS) We use Type 2 FLS due to its unique capabilities in handling uncertainty and capturing unmodelled emerging behaviors of the system and environment. We use LCS as a capable machine learning technique to synthesize the data base to form the knwoledge base. 9 A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

10. North Carolina Agricultural and Technical State University General Structure of the Fuzzy Type-2 Based PIE Teleop Unit Sensing unit Decision making and planning Computer Actuators Command Center Adjustment (Knowledge base) Rule Base Inference System Output Process Type Reducer Defuzzifier Fuzzifier Matrix Translation of Fuzzy Rules Rule Generator Fuzzifier T2FLSLCS PIE 10 A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

11. North Carolina Agricultural and Technical State University Our T&E Team at ACIT Institute 11 Dr. A. Homaifar Dr. A. Karimoddini Principle Investigators: Research Associates: Graduate Students Undergraduate Students Daniel Opoku Nnamdi J. Enyinna Alejandro White Muhammad Sohail Evan Olney Emmanuel ArzateBilly Whitehead Nicholas Donald Michael Lowe Vin K A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

12. North Carolina Agricultural and Technical State University Acknowledgment 12 Test Resources Management Center (TRMC) Scientific Research Corporation (SRC) Thanks to for supporting the NC A&T project on developing a T&E tool for testing and evaluation of unmanned systems. A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

13. North Carolina Agricultural and Technical State University 13 Q & A A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems