Developing a Computational Model of the Pilot’s Best Possible Expectation of Aircraft State Given Vestibular and Visual Cues Can Onur Master’s Thesis Defence Committee Members Dr. Amy Pritchett (Chair) Dr. Eric Johnson Dr. Santosh Mathan This work is funded by the NASA Aviation Safety Program

Introduction and Motivation Objectives 1 – Developing the Model Overview Introduction and Motivation Objectives 1 – Developing the Model 2 – Verify & Validate the Model Conclusion

Spatial Disorientation in Aviation Spatial Disorientation (SD): occurs when a pilot fails to properly sense the aircraft’s motion, position or attitude “The chance of a pilot experiencing SD during their career is in the order of 90 to 100 per cent.” Australian Transport Safety Bureau, 2007 I would like first define some terminologies to understand the motivation of this thesis better. SD, in aviation, is an important phenomenon. And it occurs… According to ATSB, nearly every pilot experiences SD at least once in their careers. If the SD is not recognized immediately, it leads to LOC leads Spatial Disorientation (SD) Loss of Control (LOC) SKYbrary Flight Safety Foundation, 1992

Loss of Control Accidents Fatal Accidents – Worldwide Commercial Jet Fleet – 2003 Through 2012 ( ) The Loss of Control (LOC) accidents are still a major threat for air transport and general aviation and have the highest risk for fatalities. (It is estimated that 0.3 fatal accidents occur per million departures.) And SD makes up 32 percent of these LOC accidents. Boeing, 2013 Bateman, et al., 2011

Aircraft State Perception and Susceptibility to SD Contributor #1: The Vestibular System Vestibular System Semi-Circular Canals Otolith Human vestibular system evolved in a 1-g environment (walking, running, sitting) So, to understand SD better, let’s see how we perceive orientation and what makes us susceptible to SD in aviation. There are 3 major contributors to it; Vestibular system located in the inner ear, and helps us keep our balance and posture. However, since this system evolved in a 1-g environment, it can deceive and can lead to illusions during flight, Especially when the visual information is lacking. Limitations: Threshold Values (No sensation in case of a sub-t maneuver) Sensor Dynamics (Signals exponentially decay during a sustained stimulus)

Vestibular Illusions in Aviation Limitations are causing illusions (especially when visual cues are lacking) Somatogyral illusion occurs due to the vestibular thresholds for sensing. For example, in case of a sub-threshold roll maneuver, pilot cannot perceive the roll unless there is visual sensing involved. Somatogravic illusion occurs due to ambiguous nature of the otolith measurments. It’s basically a false pitch-up sensation during a forward linear acceleration. Somatogyral Illusions (mostly due to SCC) (e.g., dead-man’s spiral, leans) Somatogravic Illusions (mostly due to otolith) (e.g., false sensation of pitch)

Aircraft State Perception and Susceptibility to SD Contributor #2: The Visual System Flight desk displays are the most reliable source of information (If scanned) Contributor #3: Pilot Knowledge of the Aircraft Dynamics Pilot expertise through training and experience Ability to generate internal expectations of the aircraft state based on sensory cues Second contributor to orientation sensation is the visual sensors. Flight desk displays are the most reliable source of information in cockpit. All crucial aircraft states can be perceived through the flight instruments. The third major component is pilot’s knowledge of the aircraft dynamics. Pilots can generate internal expectations of aircraft states. These internal expectations form through training and experience.

Aircraft State Perception and Susceptibility to SD Contributor #2: The Visual System Flight desk displays are the most reliable source of information (If scanned) Problem 1: How does a pilot incorporate these sensory inputs and the expertise into their expectation of spatial orientation? Contributor #3: The Knowledge of the Aircraft Dynamics Pilot Expertise through training and experience Ability to generate internal expectations of the aircraft state based on sensory cues

Countermeasures to SD Training Alerting Systems Simulators “Believe your flight instruments” trainings Alerting Systems Auditory Tactile Visual Flight Deck Display Designs NextGen Flight Deck Displays Software/Hardware Enhancements Don’t say too much. Since SD is still has the highest risk for fatalities and pilots are prone to SD many researches have been done to find feasible countermeasures to SD. (related accidents/incidents) There are wide ranges of SD countermeasure propositions including training designs, auditory and tactile alerting systems, and flight instrument improvements/designs.

Countermeasures to SD Training Simulators “Believe your flight instruments” trainings Alerting Systems Auditory Tactile Visual Flight Deck Display Designs NextGen Flight Deck Displays Software/Hardware Enhancements Problem 2: How to identify the pilot’s information requirements? Problem 3: How to help analyze potential flight deck technology interventions? Problem 2: How to identify the pilot’s information requirements? What is needed to maintain the spatial orientation? Problem 3: How to help analyze potential flight deck technology interventions?

Introduction and Motivation Objectives 1 – Developing the Model Overview Introduction and Motivation Objectives 1 – Developing the Model 2 – Verify & Validate the Model Conclusion Now I am going to list the objectives of this thesis, which can help answer the research questions stated earlier.

Objectives Develop a computational model (Model-Based Observer) to predict the pilot’s best possible expectation of the aircraft state given vestibular and visual cues. Parameterize and verify & validate the model using: Preliminary scenarios (brief turns, banking maneuvers etc.) Empirical data from the literature

Introduction and Motivation Objectives 1 – Developing the Model Overview Introduction and Motivation Objectives 1 – Developing the Model 2 – Verify & Validate the Model Conclusion

The Model-Based Observer 1 Aircraft Dynamics ⌃ 2 Measurements of Aircraft State Best Possible 3 Pilot’s “Internal Simulation” of the Aircraft So, this model is looking at predicting the best possible pilot expectation by developing a Model Based Observer. Let us have a look at how we can construct such a model. We first need an aircraft… Here is a representation of aircraft dynamics. Here, is the pilot aspect of the MBO and this portion looks similar to the upper portion and represents an expert pilot that knows about the aircraft dynamics. Measurements represents the sensor outputs, say an airspeed indicator displaying the airspeed. I will talk about the implemented measurement systems and measurement errors in the next slides. As you can see, pilot has an estimation of this airspeed. … actual and estimated measurements are then compared and residuals are weight by a Gain to propagate the expectations. Since we are aiming for the best possible expectation, we should find the optimal gains. 4 Discrepancy between estimated and actual measurements

Discrete Visual Scanning Model Discrete Measurements … Measurement error (v) for visual scan Sensor noise Error due to display design (thick needle -> elevated error) Pilot perception error MBO stable for a range of error values The measurement error for visual scan includes several errors. It covers…. It is impossible to find a definite value for the visual measurement errors. That’s why a range that satisfies MBO’s stability has been determined by trial and error. timeline

Vestibular Model Continuous measurements of the aircraft states and state derivatives The SCC Dynamics (based on Merfeld’s model) The Otolith Dynamics (based on Grant & Best’s model) ySCC = yOTO = * * The vestibular system has already been extensively researched. For sensor dynamics of the vestibular components, we have used the previously developed models. The SCC takes the angular rates as inputs and outputs the afferent firing rate for angular rates. The otolith takes the specific forces and measures the specific force afferent firing rates. Just like the discrete visual model, the continuous vestibular model has measurement error. The error values for vestibular measurements can also be found from the previous studies and can be parameterize for the MBO Measurement error (v) for vestibular model Error values given in the previous work Merfeld 1990, Grant & Best 1986

MBO Structure: Hybrid Kalman Filter Continuous-time non-linear system dynamics (aircraft dynamics) P -> the error covariance matrix (a measure of the estimated accuracy of the state estimate) (Discrete RE) (Continuous RE) Here is how the MBO calculates the optimal gain values shown in the block diagram earlier. Riccati equations defines error covariance matrices both for continuous and for discrete cases. By minimizing the diagonal entries of this matrices, continuous and discrete optimal gain values can be found. They are called Kalman Gains and these gains help us propagate the best possible pilot expectation. (Discrete Kalman Gain ) (Continuous Kalman Gain) Kd & Kc are the optimal gains (Kalman Gains) to generate the best possible estimate

Introduction and Motivation Objectives 1 – Developing the Model Overview Introduction and Motivation Objectives 1 – Developing the Model 2 – Verify & Validate the Model Conclusion

Verification of the Model Components The Hybrid Kalman Filter The Semi-Circular Canal Model The Otolith Model The MBO components The Dryden Model Implemented in aircraft simulation to emulate turbulence Three MBO components can be listed as such. A fourth component I built is not included into the MBO but indirectly affects by emulating turbulence in the aircraft.

i) Kalman Filter Verification Gaussian Error is used in the Kalman Filter Estimation error is expected to be Gaussian Diagonal Entries = Predicted estimation error variances of each state Here is the 2-sigma bounds of the roll rate in a brief test case.. We can also do a normality test for the estimation error to make sure that it is Gaussian ->NEXT

i) Kalman Filter Verification Gaussian Error is used in the Kalman Filter Estimation error is expected to be Gaussian Diagonal Entries = Predicted estimation error variances of each state

i) Kalman Filter Verification – Measurements Impact Sub-Threshold (no visual – no SCC) Here is a simple distraction scenario. Pilot is not scanning the flight instruments from 3rd seconds to 15th seconds. You can see how the error variance grows when there is no visual measurements. You can also observe the effects of vestibular measurements in these graph. These are from a bank experiment. And in the sub-threshold case, there is neither visual nor SCC measurements. This leads to a more inflated variance bound compared to the above-t case. Above-Threshold (no visual)

ii) SCC Model Verification To verify the SCC, we compared the implemented SCC outputs to a previously developed models outputs. I built a simple test case that has a similar stimulus. And investigated the canal afferent responses of both. Time-crossing zero and peak values of both graphs are approximately the same. Here is the SCC implemented to the MBO. A similar aircraft state stimulus causes a similar SCC response. A previously developed model’s responses (Borah et. al, 1988)

ii) SCC Model Verification (sub-threshold behavior) The SCC does not provide accurate information in case of a sub-threshold maneuver Does it provide no measurement? (is it completely inactive) Does it provide measurements of zero?

ii) SCC Model Verification (sub-threshold behavior) (1) The SCC provides measurements of zero when maneuver is sub-threshold (2) The SCC provides no measurement when maneuver is sub-threshold … In the second case, sub-threshold regions cause an bigger expansion compared to the first configuration. This is because the second case does not provide any measurement when it’s sub-threshold. It is nothing more than ignoring the presence of the SCC in sub-threshold cases. Therefore, the MBO implements the first case, which is a SCC that provides measurements of zero when maneuver is sub-threshold. Sub-Threshold Above-T Sub-Threshold

iii) The Otolith Model Verification Forward Acceleration Experiment sf_x = –θ.g – ů Otoliths, in the literature, are not studied as much as the SCC. That’s why we didn’t have as much information for comparison and verification. However, we can at least check if the implemented model is satisfying the theoretical relationships. By looking at the otolith measurements during a no-pitch-forward-acceleration maneuver, we can see that it establishes the specific force relation.

iii) The Otolith Model Verification Pitch-up Experiment sf_x = –θ.g – ů Here is another test case. This is a pitch-up experiment. There is no commanded deceleration but aircraft decelerates due to the dynamics. And again the specific force relation is established. (While we didn’t command a deceleration the pitch up did cause it)

iv) The Dryden Model Verification Linear gust verification Angular gust verification I ran an FFT analysis for gusts from Matlab and form the implemented Dryden Models. Magnitude spectrums has been compared. Both linear and angular gusts from the implemented model show a good match with the Matlab’s validated Dryden Model

Validation of the Integrated MBO Ability of the model to predict known problems with pilot SD. Specifically, to reproduce the illusions that occur due to vestibular limitations when visual cues are lacking. Impact of visual scanning. Do the visual corrections help overcome the illusion?

Somatogyral Illusion Above-Threshold Banking Sub-Threshold Banking

Visual Corrections on Somatogyral Illusion Above-Threshold Banking Sub-Threshold Banking

Somatogravic Illusion Forward Acceleration Deceleration

Visual Corrections on Somatogravic Illusion Forward Acceleration

Introduction and Motivation Objectives 1 – Developing the Model Overview Introduction and Motivation Objectives 1 – Developing the Model 2 – Verify & Validate the Model Conclusion

Examples & Potential Design Interventions Concerns Model Representation Potential Design Intervention Pilot distraction No scan for some or all of the instruments Alerting for rare (and rarely-sampled) flight conditions. Inaccurate pilot perception of state from instruments Inaccurate or noisy measurement More accurate/higher resolution presentation of key aircraft states. Here, I would like to give an example on how this model can be used to represent certain critical SD concerns, and how can this model be used to give design insights to the NextGen flight instruments. So, one of the concerns is an inaccurate pilot perception of state from instruments. Say we have an airspeed indicator which is hard to read or interpret. This would correspond to an elevated measurement error for airspeed can can be represented in the model by having a bigger error value for it. Critical maneuvers can be run to see if this inaccurate measurement causes substantial deviation in pilot’s best possible expectation from the actual aircraft state. If so, this can give insight for the NextGen designers to aim for a higher resolution presentation of the airspeed indicator.

Summary & Contributions The MBO enables several analyses: Investigate the mechanism of spatial disorientation Predict the best possible pilot’s expectations of the aircraft state with a given visual scan pattern Identify the pilot’s information requirements (regarding the appropriate energy-state and attitude awareness) Analyze potential flight deck technology interventions and/or provide design insights for the NextGen flight deck display designs.

Thank You!