1. Developing a Computational Model of the Pilot’s Best Possible Expectation of Aircraft State Given Vestibular and Visual Cues
Can Onur
Master’s Thesis Defence
Committee Members
Dr. Amy Pritchett (Chair)
Dr. Eric Johnson
Dr. Santosh Mathan
This work is funded by the NASA Aviation Safety Program

2. Introduction and Motivation Objectives 1 – Developing the Model
Overview
Introduction and Motivation
Objectives
1 – Developing the Model
2 – Verify & Validate the Model
Conclusion

3. Spatial Disorientation in Aviation
Spatial Disorientation (SD): occurs when a pilot fails to properly sense the aircraft’s motion, position or attitude
“The chance of a pilot experiencing SD during their career is in the order of 90 to 100 per cent.”
Australian Transport Safety Bureau, 2007
I would like first define some terminologies to understand the motivation of this thesis better.
SD, in aviation, is an important phenomenon. And it occurs…
According to ATSB, nearly every pilot experiences SD at least once in their careers.
If the SD is not recognized immediately, it leads to LOC
leads
Spatial Disorientation (SD)
Loss of Control (LOC)
SKYbrary Flight Safety Foundation, 1992

4. Loss of Control Accidents
Fatal Accidents – Worldwide Commercial Jet Fleet – 2003 Through 2012
( )
The Loss of Control (LOC) accidents are still a major threat for air transport and general aviation and have the highest risk for fatalities. (It is estimated that 0.3 fatal accidents occur per million departures.)
And SD makes up 32 percent of these LOC accidents.
Boeing, 2013
Bateman, et al., 2011

5. Aircraft State Perception and Susceptibility to SD
Contributor #1: The Vestibular System
Vestibular System
Semi-Circular Canals
Otolith
Human vestibular system evolved in a 1-g environment (walking, running, sitting)
So, to understand SD better, let’s see how we perceive orientation and what makes us susceptible to SD in aviation. There are 3 major contributors to it;
Vestibular system located in the inner ear, and helps us keep our balance and posture. However, since this system evolved in a 1-g environment, it can deceive and can lead to illusions during flight, Especially when the visual information is lacking.
Limitations:
Threshold Values (No sensation in case of a sub-t maneuver)
Sensor Dynamics (Signals exponentially decay during a sustained stimulus)

6. Vestibular Illusions in Aviation
Limitations are causing illusions (especially when visual cues are lacking)
Somatogyral illusion occurs due to the vestibular thresholds for sensing. For example, in case of a sub-threshold roll maneuver, pilot cannot perceive the roll unless there is visual sensing involved.
Somatogravic illusion occurs due to ambiguous nature of the otolith measurments. It’s basically a false pitch-up sensation during a forward linear acceleration.
Somatogyral Illusions (mostly due to SCC)
(e.g., dead-man’s spiral, leans)
Somatogravic Illusions (mostly due to otolith)
(e.g., false sensation of pitch)

7. Aircraft State Perception and Susceptibility to SD
Contributor #2: The Visual System
Flight desk displays are the most reliable source of information (If scanned)
Contributor #3: Pilot Knowledge of the Aircraft Dynamics
Pilot expertise through training and experience
Ability to generate internal expectations of the aircraft state based on sensory cues
Second contributor to orientation sensation is the visual sensors. Flight desk displays are the most reliable source of information in cockpit. All crucial aircraft states can be perceived through the flight instruments.
The third major component is pilot’s knowledge of the aircraft dynamics. Pilots can generate internal expectations of aircraft states. These internal expectations form through training and experience.

8. Aircraft State Perception and Susceptibility to SD
Contributor #2: The Visual System
Flight desk displays are the most reliable source of information (If scanned)
Problem 1: How does a pilot incorporate these sensory inputs and the expertise into their expectation of spatial orientation?
Contributor #3: The Knowledge of the Aircraft Dynamics
Pilot Expertise through training and experience
Ability to generate internal expectations of the aircraft state based on sensory cues

9. Countermeasures to SD Training Alerting Systems
Simulators
“Believe your flight instruments” trainings
Alerting Systems
Auditory
Tactile
Visual
Flight Deck Display Designs
NextGen Flight Deck Displays
Software/Hardware Enhancements
Don’t say too much.
Since SD is still has the highest risk for fatalities and pilots are prone to SD many researches have been done to find feasible countermeasures to SD. (related accidents/incidents)
There are wide ranges of SD countermeasure propositions including training designs, auditory and tactile alerting systems, and flight instrument improvements/designs.

10. Countermeasures to SD
Training
Simulators
“Believe your flight instruments” trainings
Alerting Systems
Auditory
Tactile
Visual
Flight Deck Display Designs
NextGen Flight Deck Displays
Software/Hardware Enhancements
Problem 2: How to identify the pilot’s information requirements?
Problem 3: How to help analyze potential flight deck technology interventions?
Problem 2: How to identify the pilot’s information requirements?
What is needed to maintain the spatial orientation?
Problem 3: How to help analyze potential flight deck technology interventions?

11. Introduction and Motivation Objectives 1 – Developing the Model
Overview
Introduction and Motivation
Objectives
1 – Developing the Model
2 – Verify & Validate the Model
Conclusion
Now I am going to list the objectives of this thesis, which can help answer the research questions stated earlier.

12. Objectives
Develop a computational model (Model-Based Observer) to predict the pilot’s best possible expectation of the aircraft state given vestibular and visual cues.
Parameterize and verify & validate the model using:
Preliminary scenarios (brief turns, banking maneuvers etc.)
Empirical data from the literature

13. Introduction and Motivation Objectives 1 – Developing the Model
Overview
Introduction and Motivation
Objectives
1 – Developing the Model
2 – Verify & Validate the Model
Conclusion

14. The Model-Based Observer1
Aircraft Dynamics ⌃ 2
Measurements of Aircraft State
Best Possible 3
Pilot’s “Internal Simulation” of the Aircraft
So, this model is looking at predicting the best possible pilot expectation by developing a Model Based Observer. Let us have a look at how we can construct such a model.
We first need an aircraft… Here is a representation of aircraft dynamics.
Here, is the pilot aspect of the MBO and this portion looks similar to the upper portion and represents an expert pilot that knows about the aircraft dynamics.
Measurements represents the sensor outputs, say an airspeed indicator displaying the airspeed. I will talk about the implemented measurement systems and measurement errors in the next slides.
As you can see, pilot has an estimation of this airspeed.
… actual and estimated measurements are then compared and residuals are weight by a Gain to propagate the expectations.
Since we are aiming for the best possible expectation, we should find the optimal gains. 4
Discrepancy between estimated and actual measurements

15. Discrete Visual Scanning Model
Discrete Measurements …
Measurement error (v) for visual scan
Sensor noise
Error due to display design (thick needle -> elevated error)
Pilot perception error
MBO stable for a range of error values
The measurement error for visual scan includes several errors. It covers….
It is impossible to find a definite value for the visual measurement errors.
That’s why a range that satisfies MBO’s stability has been determined by trial and error.
timeline

16. Vestibular Model
Continuous measurements of the aircraft states and state derivatives
The SCC Dynamics
(based on Merfeld’s model)
The Otolith Dynamics
(based on Grant & Best’s model)
ySCC =
yOTO = * *
The vestibular system has already been extensively researched. For sensor dynamics of the vestibular components, we have used the previously developed models.
The SCC takes the angular rates as inputs and outputs the afferent firing rate for angular rates.
The otolith takes the specific forces and measures the specific force afferent firing rates.
Just like the discrete visual model, the continuous vestibular model has measurement error. The error values for vestibular measurements can also be found from the previous studies and can be parameterize for the MBO
Measurement error (v) for vestibular model
Error values given in the previous work
Merfeld 1990, Grant & Best 1986

17. MBO Structure: Hybrid Kalman Filter
Continuous-time non-linear system dynamics (aircraft dynamics)
P -> the error covariance matrix
(a measure of the estimated accuracy of the state estimate)
(Discrete RE)
(Continuous RE)
Here is how the MBO calculates the optimal gain values shown in the block diagram earlier.
Riccati equations defines error covariance matrices both for continuous and for discrete cases. By minimizing the diagonal entries of this matrices, continuous and discrete optimal gain values can be found.
They are called Kalman Gains and these gains help us propagate the best possible pilot expectation.
(Discrete Kalman Gain )
(Continuous Kalman Gain)
Kd & Kc are the optimal gains (Kalman Gains) to generate the best possible estimate

18. Introduction and Motivation Objectives 1 – Developing the Model
Overview
Introduction and Motivation
Objectives
1 – Developing the Model
2 – Verify & Validate the Model
Conclusion

19. Verification of the Model Components
The Hybrid Kalman Filter
The Semi-Circular Canal Model
The Otolith Model
The MBO components
The Dryden Model
Implemented in aircraft simulation to emulate turbulence
Three MBO components can be listed as such.
A fourth component I built is not included into the MBO but indirectly affects by emulating turbulence in the aircraft.

20. i) Kalman Filter Verification
Gaussian Error is used in the Kalman Filter
Estimation error is expected to be Gaussian
Diagonal Entries = Predicted estimation error variances of each state
Here is the 2-sigma bounds of the roll rate in a brief test case..
We can also do a normality test for the estimation error to make sure that it is Gaussian ->NEXT

21. i) Kalman Filter Verification
Gaussian Error is used in the Kalman Filter
Estimation error is expected to be Gaussian
Diagonal Entries = Predicted estimation error variances of each state

22. i) Kalman Filter Verification – Measurements Impact
Sub-Threshold
(no visual – no SCC)
Here is a simple distraction scenario. Pilot is not scanning the flight instruments from 3rd seconds to 15th seconds.
You can see how the error variance grows when there is no visual measurements.
You can also observe the effects of vestibular measurements in these graph.
These are from a bank experiment. And in the sub-threshold case, there is neither visual nor SCC measurements. This leads to a more inflated variance bound compared to the above-t case.
Above-Threshold
(no visual)

23. ii) SCC Model Verification
To verify the SCC, we compared the implemented SCC outputs to a previously developed models outputs.
I built a simple test case that has a similar stimulus. And investigated the canal afferent responses of both. Time-crossing zero and peak values of both graphs are approximately the same.
Here is the SCC implemented to the MBO. A similar aircraft state stimulus causes a similar SCC response.
A previously developed model’s responses (Borah et. al, 1988)

24. ii) SCC Model Verification (sub-threshold behavior)
The SCC does not provide accurate information in case of a sub-threshold maneuver
Does it provide no measurement? (is it completely inactive)
Does it provide measurements of zero?

25. ii) SCC Model Verification (sub-threshold behavior)
(1) The SCC provides measurements of zero when maneuver is sub-threshold
(2) The SCC provides no measurement when maneuver is sub-threshold …
In the second case, sub-threshold regions cause an bigger expansion compared to the first configuration. This is because the second case does not provide any measurement when it’s sub-threshold.
It is nothing more than ignoring the presence of the SCC in sub-threshold cases. Therefore, the MBO implements the first case, which is a SCC that provides measurements of zero when maneuver is sub-threshold.
Sub-Threshold
Above-T
Sub-Threshold

26. iii) The Otolith Model Verification
Forward Acceleration Experiment
sf_x = –θ.g – ů
Otoliths, in the literature, are not studied as much as the SCC. That’s why we didn’t have as much information for comparison and verification.
However, we can at least check if the implemented model is satisfying the theoretical relationships.
By looking at the otolith measurements during a no-pitch-forward-acceleration maneuver, we can see that it establishes the specific force relation.

27. iii) The Otolith Model Verification
Pitch-up Experiment
sf_x = –θ.g – ů
Here is another test case. This is a pitch-up experiment. There is no commanded deceleration but aircraft decelerates due to the dynamics.
And again the specific force relation is established.
(While we didn’t command a deceleration the pitch up did cause it)

28. iv) The Dryden Model Verification
Linear gust verification
Angular gust verification
I ran an FFT analysis for gusts from Matlab and form the implemented Dryden Models. Magnitude spectrums has been compared.
Both linear and angular gusts from the implemented model show a good match with the Matlab’s validated Dryden Model

29. Validation of the Integrated MBO
Ability of the model to predict known problems with pilot SD.
Specifically, to reproduce the illusions that occur due to vestibular limitations when visual cues are lacking.
Impact of visual scanning. Do the visual corrections help overcome the illusion?

30. Somatogyral Illusion
Above-Threshold Banking
Sub-Threshold Banking

31. Visual Corrections on Somatogyral Illusion
Above-Threshold Banking
Sub-Threshold Banking

32. Somatogravic Illusion
Forward Acceleration
Deceleration

33. Visual Corrections on Somatogravic Illusion
Forward Acceleration

34. Introduction and Motivation Objectives 1 – Developing the Model
Overview
Introduction and Motivation
Objectives
1 – Developing the Model
2 – Verify & Validate the Model
Conclusion

35. Examples & Potential Design Interventions
Concerns
Model Representation
Potential Design Intervention
Pilot distraction
No scan for some or all of the instruments
Alerting for rare (and rarely-sampled) flight conditions.
Inaccurate pilot perception of state from instruments
Inaccurate or noisy measurement
More accurate/higher resolution presentation of key aircraft states.
Here, I would like to give an example on how this model can be used to represent certain critical SD concerns, and how can this model be used to give design insights to the NextGen flight instruments.
So, one of the concerns is an inaccurate pilot perception of state from instruments. Say we have an airspeed indicator which is hard to read or interpret.
This would correspond to an elevated measurement error for airspeed can can be represented in the model by having a bigger error value for it.
Critical maneuvers can be run to see if this inaccurate measurement causes substantial deviation in pilot’s best possible expectation from the actual aircraft state.
If so, this can give insight for the NextGen designers to aim for a higher resolution presentation of the airspeed indicator.

36. Summary & Contributions
The MBO enables several analyses:
Investigate the mechanism of spatial disorientation
Predict the best possible pilot’s expectations of the aircraft state with a given visual scan pattern
Identify the pilot’s information requirements (regarding the appropriate energy-state and attitude awareness)
Analyze potential flight deck technology interventions and/or provide design insights for the NextGen flight deck display designs.

37. Thank You!