1. SENSOR FUSION LABORATORY Director: Thad Roppel, Associate Professor AU Electrical and Computer Engineering Dept. troppel@eng.auburn.edu EXAMPLES Infrared / Millimeter wave radar for vehicle detection and identification Chemical sensor arrays – “artificial nose” Biomimetics – imitating animal sensorimotor behaviors Biomedical – using electrical and optical probes to study cardiac arrhythmias MISSION: Study the benefits of using simultaneous information from multiple sensors to probe the environment.

2. SENSOR FUSION LABORATORY Problem Complexity: Human vs. Machine HUMAN MACHINE EASY HARD EASY HARD Maximum Potential Benefit Object recognition Linguistics Extraction of Relevant Features from Sensor Arrays Arithmetic Logic Thresholding Tallying Judging

3. Personnel and Publications PERSONNEL Ting-To Lo (PhD): Molecular Switching in Biosensors Rama Narendran (PhD): Biomimetic Simulations of Organized Machine Behavior Jun Pan (PhD): Wireless Protocol for Electrical and Optical Cardiac Microprobes Aroldo Couto (MS): Flight Stabilization Using Adaptive Artificial Neural Networks Brian Wingfield (MS): Silicon Processing for Lateral Emission Fiber-Optic Sensors REPRESENTATIVE RECENT PUBLICATIONS D. M. Wilson, T. Roppel, and R. Kalim, "Aggregation of Sensory Input for Robust Performance in Chemical Sensing Microsystems," Sensors and Actuators B, 64(1–3), 107-117, June 2000. T. Roppel and D. M. Wilson, "Biologically-Inspired Pattern Recognition for Odor Detection," Pattern Recognition Letters, 21(3), 213–219, March 2000. D. M. Wilson, K. Dunman, T. Roppel, and R. Kalim, "Rank Extraction in Tin-Oxide Sensor Arrays," Sensors and Actuators B, 62(3), 199-210, April 2000. T. Roppel, R. Kalim, and D. Wilson, "Sensory Plane Analog-VLSI for Interfacing Sensor Arrays to Neural Networks, " Virtual Intelligence and Dynamic Neural Networks VI-DYNN '98, Stockholm, Sweden, June 22-26, 1998.

4. IR / MMW DATA FUSION Support: AFOSR 1992-93 Project Goal: Improved identification of military vehicles from aerial scenes. LANCE Missile Launcher T-62 Tank M-113 Armored Personnel Carrier (APC)

5. IR / MMW Fusion, cont’d APPROACH: IR SCENE PIXELS MMW RADAR DATA NEURAL NETWORK APC TANK LAUNCHER PERFORMANCE ASSESSMENT: ATL A+-- T-+- L--+ Multiple permutations Confusion matrix Average result OVERALL RESULT: 14 % improvement with sensor fusion

6. Chemical Sensor Arrays Support: DARPA 1997-99 PROJECT GOAL: Improved identification and detection of chemical plumes in non-laboratory conditions. VEHICLE SENSORS PLUME COMMAND STATION RF LINK ROAD WIND

7. Chemical Sensor Arrays, cont’d Odor Sensor Array 0100200300400500 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Timestep Sensor Voltage Sensor Outputs Sensor Array Dynamic Response

8. Chemical Sensor Arrays, cont’d Preprocessing

9. Chemical Sensor Arrays, cont’d ace Sample 1Sample 2 1 20 Sample 3 1 20 amm dal g87 g89 g93 oil pth Sensor # xyl 51015 Sensor # 51015 Sensor # 51015

10. Chemical Sensor Arrays, cont’d

12. Chemical Sensor Arrays - Summary A recurrent neural network was trained to recognize 9 odors presented in an arbitrary time sequence. Response time is reduced by an order of magnitude by threshold preprocessing. Well-suited for use as a front-end for a hierarchical suite of NN’s in a portable, near-real time odor classification device.

13. BIOMIMETICS PROJECT GOAL: Learn about optic flow-based sensor fusion from animals. Apply this to flying a drone to target using onboard video. Flies land accurately Bees find flowers Bats catch evading insects in flight

14. BIOMIMETICS, cont’d These are natural examples of optic flow. Represent sensory image field by motion vector field. Image SequenceOptic Flow Field

15. BIOMIMETICS, cont’d EXAMPLES A fly can land by maintaining constant optic flow. A dog can track by maintaining constant sensory flow across olfactory epithelium and following the gradient (using sniffing as a form of “chopper amplifier.” Can we navigate a drone or guide a missile to target with a similar approach?

16. Neuro-fuzzy Control Systems FUZZY LOGIC Multivalued logic based on established, rigorous mathematical theory Allows intermediate values to be defined between extremes (yes/no, fast/slow) Permits valid computational decisions to be made using syntactic input Yields fast, rule-based computations which can be validated systematically NEURAL NETWORK Nonlinear dynamical system “black box” Trained using field data Hard to validate

17. Neuro-fuzzy Application Example: Optic Flow for 2D Translation and Rotation Specifying the initial membership functions for the fuzzy sets

18. Membership Functions

19. Fuzzy Rules Two example rules (implement multi-camera fusion)... If angle is zero and angular velocity is zero and speed is zero, then translation is zero and rotation is zero. If angle is zero and angular velocity is pos. low and speed is pos. low, then translation is pos. low and rotation is CW low. Example meta-rule... If GPS is unreliable then do not use. Italics: predicate Bold: consequence Underlined: fuzzy value

20. Neuro-fuzzy Training Membership functions and rule weights are trained on field data using a neural network. Training can be off-line or on-line (adaptive). Any suitable NN can be used. After training, validation can be achieved to any desired degree by systematically firing rules and observing system behavior. The system dynamics is embedded in the fuzzy rules, not the neural network.

21. De-fuzzification Ref.: “A brief course in Fuzzy Logic and Fuzzy Control” http://www.flll.uni-linz.ac.at/pdw/fuzzy/fuzzy.html

22. Neuro-fuzzy Navigator OPTIC FLOW PROCESSOR SENSOR FUSION & FEATURE EXTRACTION NEURAL NET NAVIGATOR Channel 1 GPS IMU GPS/INS NAVIGATION Channel 2 Magnetometer NEURAL NETWORK TRAINING - - + + Measured Velocity & Position Velocity & Position Estimate Attitude STATE ESTIMATOR OPTIC FLOW PROCESSOR SENSOR FUSION & FEATURE EXTRACTION NEURAL NET NAVIGATOR Channel 1 GPS IMU GPS/INS NAVIGATION GPS IMU GPS/INS NAVIGATION Channel 2 Magnetometer NEURAL NETWORK TRAINING NEURAL NETWORK TRAINING - - + + Measured Velocity & Position Velocity & Position Estimate Attitude STATE ESTIMATOR Neuro-fuzzy Processing