Key Findings: During the project's fourth year, we focused on: merging MIT's Observer model with Alion's SDAT; enhancing SDAT with N-SEEV (noticing-salience, expectancy, effort, and value) and with three new illusion models, verification tests, and comparisons of analytical results produced by SDAT and Observer; validation of SDAT with anonymous data sets of helicopter pilots who experienced SD; and administering an Institutional Review Board (IRB)-approved Space Shuttle spatial orientation survey.

Observer was 'packaged' as a DLL (dynamically linked library) within SDAT. SDAT users can select whether they wish to use Observer algorithms for predicted perception, or SDAT's algorithms. While Observer may be more physiologically accurate, Observer requires data sets to be of a fixed rate and fairly high frequency (10-100 Hz). Unfortunately, actual vehicle flight data recordings rarely meet these requirements. In addition, Observer does not account for misperceptions due to sub-threshold motions, which are critical to many SDAT illusion models. Therefore, we give users the option to select Observer or SDAT algorithms for attitude perception predictions.

We designed three new illusion models to SDAT based upon vertical landing vehicle scenarios that we observed in data sets provided by an anonymous source of helicopter data -- data sets that included confirmed SD events. The three models are: (1) "Undetected loss of lift," which occurs when the pilot unwittingly flies out of ground effect with insufficient thrust to maintain the new altitude, resulting in a sudden plunge toward the surface; (2) "Inadvertent drift during hover" that could result in the vehicle striking an obstacle; and (3) "Undetected drift during landing" that could cause the vehicle to tip-over.

SDAT has also been enhanced with a pilot attention model called N-SEEV. N-SEEV elevates applied countermeasures when SDAT predicts that the pilot is suffering from SD and has not attended to a lower level of countermeasures. We created an updated version of SDAT's user manual and delivered SDAT and its user manual to the National Space Biomedical Research Institute (NSBRI). We did not undertake an experiment to validate the newly enhanced SDAT because we could use existing data sets plus the new ones acquired from our anonymous source of helicopter data sets. We also judged that a simulator validation experiment would use resources needed to do the best possible job of integrating Observer into SDAT.

FORT (frame of reference transformation) tool cost scores were not integrated into SDAT. The FORT tool remains a separate stand-alone tool. We performed additional FORT tool validation, and submitted an article for the Human Factors Journal.

We received 40 usable survey responses, analyzed the data from the 71 missions in the responses, and submitted an article to Aviation, Space, and Environmental Medicine reporting our method and results. We also sent de-identified data to our customer, NASA-Johnson Space Center (JSC's) Dr. Jacob Bloomberg, and will make the full set of de-identified data available to anyone who wishes it.

We designed three new illusion models to SDAT based upon vertical landing vehicle scenarios that we observed in data sets provided by an anonymous source of helicopter data -- data sets that included confirmed SD events. The three models are: (1) "Undetected loss of lift" which occurs when the pilot unwittingly flies out of ground effect with insufficient thrust to maintain the new altitude, resulting in a sudden plunge toward the surface; (2) "Inadvertent drift during hover" that could result in the vehicle striking an obstacle; and (3) "Undetected drift during landing" that could cause the vehicle to tip-over.

SDAT has also been enhanced with a pilot attention model called N-SEEV. N-SEEV elevates applied countermeasures when SDAT predicts that the pilot is suffering from SD and has not attended to a lower level of countermeasures. We created an updated version of SDAT's user manual and delivered SDAT and its user manual to NSBRI. We did not undertake an experiment to validate the newly enhanced SDAT because we could use existing data sets plus the new ones acquired from our anonymous source of helicopter data sets. We also judged that a simulator validation experiment would use resources needed to do the best possible job of integrating Observer into SDAT.

FORT tool cost scores are not integrated into SDAT because it is inappropriate to do so; they could be integrated into Observer. The FORT tool remains a separate stand-alone tool. We performed additional FORT tool validation, and submitted an article to the Human Factors Journal.

We received 40 usable survey responses, analyzed the data from the 71 missions in the responses, and submitted an article to Aviation, Space, and Environmental Medicine reporting our method and results. We also sent de-identified data to our customer, NASA-JSC's Dr. Jacob Bloomberg, and will make the full set of de-identified data available to anyone who wishes it.

Key Findings: During the project's third year, we focused on: merging MIT's Observer perception models with Alion's SDAT; enhancing both Observer and SDAT based upon data set analyses, verification tests, and comparisons of analytical results produced by the two models; validation of Observer via comparison to perception data from a NASA-Ames vertical motion simulator (VMS) lunar landing simulator experiment (in collaboration with Dr. Young's NSBRI-funded lunar landing project at MIT), and a dynamic swinging experiment (in collaboration with Drs. Rader and Merfeld at the Massachusetts Eye and Ear Infirmary); obtaining helicopter spatial disorientation event data sets for verification and validation tests; validating and enhancing our visual frame of reference transformation (FORT) tool; and, creating a Space Shuttle orientation survey for commanders and pilots. Due to Constellation program changes, we broadened our focus to include helicopter and fixed-wing aircraft disorientation scenarios and Space Shuttle orientation issues. MIT's Observer has been enhanced with visual inputs and calculations to account for the impact of visual cues on a human's perception of attitude, velocity and displacement. Validations of Observer included: linear and angular acceleration steps; post-rotation tilt; vertical yaw rotations (with and without vision cues); somatogravic illusions (linear acceleration with and without vision cues, and fixed and variable radius centrifugation); static and dynamic roll tilt; off vertical axis rotation; large amplitude horizontal and vertical sinusoidal displacements; circular vection ; linear vection; Coriolis and pseudo-Coriolis illusions; and, astronaut post-flight tilt-gain and tilt-translation illusions. To facilitate integration with SDAT, eObserver, a stand-alone version of Observer was developed; eObserver does not require a MatLab license to run. It includes a GUI (developed by Alion) to ease data file selection and processing step selections. Vestibular threshold literature was studied to determine the best way to represent SCC and otolith thresholds within Observer as a prerequisite for replacing SDAT's vestibular attitude calculator with Observer. The FORT tool was validated and modified using the available literature to adjust FORT costs and achieve higher correlations with previous experimental data. SDAT/SOAS is being enhanced with illusions heuristics for pilots of vertical landing vehicles. The IRB-approved survey of Shuttle commanders and pilots seeks to capture the experiences of these crew members before the Shuttle is retired. It asks questions about illusory sensations and how each respondent coped with those sensations, if he/she experienced any. The survey was peer reviewed and will be revised to accommodate peer comments as well as suggestions from the first two respondents.

Impact of Key Findings on Original Aims: The most important impacts from Year 3 are: we now have a method for integrating Observer into SDAT, a plan for integrating FORT costs into SDAT, new illusion heuristics for vertical landing vehicles (e.g., helicopters and lunar landers), helicopter SD event data sets for verifying and validating perception models and enhancements, and a survey with which to capture the prevalence and severity of Space Shuttle orientation issues.

Proposed Research Plan for Year 4: In the fourth year of this project, the Alion-MIT team will: (1) Complete enhancements to, and the merging of, SDAT and Observer, and continue comparing analytical results of common data sets. (2) Validate enhancements with previous aircraft flight data sets and new data sets (from actual vehicles and simulators). Included will be helicopter SD event data sets and data sets from space vehicle simulators (e.g. NASA-Ames' VMS). (3) Incorporate FORT tool scores into SDAT calculations of SD probability. (4) Characterize the prevalence and severity of Space Shuttle re-entry and landing orientation issues. (5) Plan and conduct experiments to generate validation data (e.g., vertical motion threshold data) and/or to demonstrate the value of SD countermeasures.

• Merged a compiled version of Observer (eObserver) with SDAT;

• Investigated how to incorporate thresholds into Observer;

• Submitted a manuscript to Biological Cybernetics comparing Observer and Kalman filter models of human orientation perception;

• Obtained actual helicopter spatial disorientation event data sets for verification and validation tests;

• Gathered perception data from a lunar landing simulator experiment (in collaboration with Dr. Young's NSBRI lunar landing project at MIT);

• Validated and enhanced our visual frame of reference transformation (FORT) tool;

• Submitted a FORT tool paper for the annual Human Factors and Ergonomics Society (HFES) meeting in San Francisco in October 2010 that was accepted;

• Created a Space Shuttle orientation survey, for past and present commanders and pilots, that was approved by the MIT and JSC IRBs; and,

• Updated the SDAT user guide.

Previous technical reports from this project were also published at the FAA's Civil Aero Medical Institute (CAMI) spatial disorientation web site: http://www.faa.gov/library/online_libraries/aerospace_medicine/sd/advancedtech/

The goal of this industry-university research and development project is to extend Alion’s spatial disorientation mitigation software – originally developed for aeronautical use – to NASA’s space applications including the Shuttle, CEV, Altair, and Mars exploration missions. Alion’s Spatial Disorientation Analysis Tool (SDAT) is designed for post hoc analyses of aircraft trajectory data from U.S. Navy, Air Force and NTSB mishaps to determine the presence or absence of vestibular SD. SOAS (Spatial Orientation Aiding System) is a real-time cockpit aid that has been evaluated in simulators with rated pilots. Both tools incorporate models of the vestibular system and assessor heuristics to predict the epoch and probability of an SD event such as Leans, Coriolis, or Graveyard Spiral illusions, as well as any other significant disparities between actual and perceived pitch attitude (somatogravic), roll rate, or yaw/heading rate. SOAS assesses multi-sensory workload to determine the types of countermeasures to trigger and when to trigger them.

This project will: 1) Enhance the utility of SDAT/SOAS by including appropriate mathematical models for vestibular and visual sensory cues, and CNS gravito-inertial force resolution into perceived tilt and translation estimates from MIT’s Observer model, and revalidating it using existing aeronautical data sets. 2) Extend the models to describe 0-G, Shuttle, and Altair landing illusions, validating the models using Shuttle and Altair simulator data sets, current theories (e.g., ROTTR observer or Bayesian particle filter), as well as archived Apollo LM data, if available. 3) Extend SDAT/SOAS to consider multiple visual frames of reference, the effects of visual attention and sensory workload, and the cognitive costs of mental rotation and reorientation. The enhanced SDAT/SOAS from Aims 1-3 will be validated via simulator and/or flight experiments. 4) SOAS will be tailored for a lunar landing, using multi-sensory workload to choose appropriate countermeasures and their timing. Countermeasures will include one or more of the following, as conditions warrant: control command displays; 2D and perspective synthetic/enhanced vision displays; attitude indicator formats tailored for physically redirected, off velocity vector viewing; and, auditory cues and commands.

SDAT will also help human factors engineers analyze the following: past Shuttle landing incidents; Orion/CEV/Altair landing and ascent trajectory planning; Altair cockpit displays, and caution and warning system design; workload evaluation; and, crew training and mission simulation. SDAT could assist flight surgeons with post-flight medical debriefings.

Key Findings

During the project’s second year, we focused on: understanding the separate Alion & MIT perception models; assessing how to combine them; obtaining vehicle data sets for verification and validation tests; and, prototyping a visual frame of reference transformation (FORT) tool.

We have access to Shuttle, VMS (vertical motion simulator), Altair simulator data sets, and helicopter simulator data sets. Apollo data sets are unavailable.

We also began to understand VMS washout algorithms with the goal to assist the VMS engineers fine-tune those algorithms to better account for lunar gravity.

MIT’s Observer has been enhanced with visual inputs and calculations to account for the impact of visual cues on a human's perception of attitude.

SDAT/SOAS is being enhanced with micro-gravity illusions heuristics.

Impact of Key Findings on Original Aims

The most important impacts from Year 2 is that we now have a prototype FORT tool to help designers calculate the costs of various cognitive display-control transformations, and Observer has been enhanced with visual cues.

Proposed Research Plan for Year 3

In the third year of this NSBRI sensorimotor adaptation project, the Alion-MIT team will: 1) Continue enhancing and merging SDAT and Observer, and continue comparing analytical results of common data sets.

2) Validate enhancements with previous flight data sets and new data sets (from actual vehicles and simulators). Included may be Shuttle landing data outlier analyses (compared to non-outliers), and data sets from Altair simulators.

3) Further develop the FORT tool from a prototype into a usable tool.

4) Help VMS engineers tune their washout algorithms to better account for lunar gravity.

5) Plan in detail for simulator and possible flight validation experiments in the second half of Year 3 and in Year 4.

MIT’s Observer model has aided investigators of aircraft accidents (e.g., 2004 Flash Air 737 fatal crash).

The new FORT tool is intended as a design aid for all vehicle control-display engineers. The tool will help designers objectively assess the costs of frame-of-reference transformations in terms of increased workload, slower response times, and more control reversal errors.

1) Extend SDAT by incorporating MIT’s Observer models. Enhance SDAT with pilot head movement data, and visual attention cues. Validate enhancements with existing and new flight data sets.

_Progress: SDAT is ready to incorporate MIT’s Observer algorithms. SDAT can also accept head movement data, and Observer includes visual orientation cues for perception calculations. We have obtained new data sets (Shuttle, Altair simulator, helicopter simulator, VMS), but were unable to obtain Apollo data, as those data sets were apparently not archived, according to our sources.

2) Extend SDAT assessments to include typical space vehicle illusions: Inversion, Visual Reorientation, Tilt Gain, and Otolith Tilt-Translation Reinterpretation. Validation will include assessment of Shuttle landing data, and Altair simulator data.

_Progress: See above. When Observer is incorporated into SDAT, SDAT will be able to assess all the illusions listed above.

3) Further extend SDAT by examining alternative visual reference frames. The FORT model is used to predict the cognitive cost of transitioning between reference frames. Validation of Aims 1-3 for SDAT will include parabolic flight experiments.

_Progress: We designed and prototyped a FORT tool to help designers calculate the cognitive costs of FORT. It is a stand-alone tool, not included in either SDAT or Observer. FORT costs include the increased potential for control errors, response time delays, and increased cognitive workload. We analyzed and applied the tool to Shuttle-ISS docking, and to Shuttle-Hubble rendezvous and robotic arm tasks. We have begun to plan flight and simulator experiments to validate all enhancements to SDAT, although parabolic flight experiments may not be included.

4) To further enhance SDAT/SOAS assessor performance, pilot multi-sensory workload is considered in countermeasure selection. Validation experiments are not detailed, but will involve evaluations in ground-based simulators.

_Progress: Once we have verified and validated our models, we will assess the efficacy of various countermeasures triggered by SOAS during years three or four.

The goal of this industry-university research and development project is to extend Alion’s spatial disorientation mitigation software – originally developed for aeronautical use – to NASA applications in the Shuttle, CEV, Altair, and Mars exploration mission. Alion’s Spatial Disorientation Analysis Tool (SDAT) is for post hoc analyses of aircraft trajectory data from U.S. Navy, Air Force and NTSB mishaps to determine the presence or absence of vestibular SD. SOAS (Spatial Orientation Aiding System) is a real-time cockpit aid that has been evaluated in simulators with rated pilots. Both tools incorporate models of the vestibular system and assessor heuristics to predict the epoch and probability of an SD event such as Leans, Coriolis, or Graveyard Spiral illusions, as well as any other significant disparities between actual and perceived pitch attitude (somatogravic), roll rate, or yaw/heading rate. SOAS assesses multi-sensory workload to determine the types of countermeasures to trigger and when to trigger them.

This project will: 1) Enhance the utility of SDAT/SOAS by including appropriate mathematical models for vestibular and visual sensory cues, and CNS gravito-inertial force resolution into perceived tilt and translation estimates from MIT’s Observer model, and revalidating it using existing aeronautical data sets.

2) Extend the models to describe 0-G, Shuttle, and Altair landing illusions, validating the models using Shuttle and Altair simulator data sets, current theories (e.g., ROTTR observer or Bayesian particle filter), as well as archived Apollo LM data, if available.

3) Extend SDAT/SOAS to consider multiple visual frames of reference, the effects of visual attention and sensory workload, and the cognitive costs of mental rotation and reorientation. The enhanced SDAT/SOAS from Aims 1-3 will be validated via simulator and/or flight experiments.

4) SOAS will be tailored for a lunar landing, using multi-sensory workload to choose appropriate countermeasures and their timing. Countermeasures will include one or more of the following, as conditions warrant: control command displays; 2D and perspective synthetic/enhanced vision displays; attitude indicator formats tailored for physically redirected, off velocity vector viewing; and, auditory cues and commands.

SDAT will also help human factors engineers analyze the following: past Shuttle landing incidents; Orion/CEV/Altair landing and ascent trajectory planning; Altair cockpit displays, and caution and warning system design; workload evaluation; and, crew training and mission simulation. SDAT could assist flight surgeons with post-flight medical debriefings.

Key Findings

During the project’s first year, we focused on: understanding the domain and astronaut susceptibility to various disorienting situations; understanding each other’s perception models; obtaining vehicle data sets for verification and validation tests; and, designing a visual frame of reference transformation (FORT) model.

We have access to Shuttle, VMS (vertical motion simulator), and Altair simulator data sets, and are still pursuing Apollo data sets. To supplement these data, we are also pursuing rotary wing data from Ft. Rucker, as well as other sources. The reason we seek rotary wing data is that they represent the closest analogs to lunar landers on Earth, and they are plentiful. Their lateral and vertical motions are significantly different from fixed-wing aircraft, thus giving us data with which to verify and validate enhancements to SDAT and Observer.

MIT’s Observer has been enhanced with a user interface for condition selection (e.g., gravity environment) and results presentation. It now uses improved calculations for perceived “down” and azimuth, as well as perceived linear velocities and displacements.

Impact of Key Findings on Original Aims

The most important impact from Year 1 is that we must re-double our efforts to obtain relevant data sets with which to verify and validate improvements to SDAT and Observer. We also determined that Altair simulators would be suitable platforms for validation experiments, supplementing or replacing parabolic flight experiments, based on cost considerations, and availability.

After designing a FORT model, we decided that it should not be incorporated into SDAT per se. Rather, the FORT cost function should be used to help determine an astronaut’s risk of SD in a given situation, yielding design guidelines to help determine the severity of an SD event, and to help select countermeasures when a frame-of-reference transformation SD occurs.

Proposed Research Plan for Year 2

In the second year of this NSBRI sensorimotor adaptation project, the Alion-MIT team will: 1) Continue enhancing and merging SDAT and Observer, and continue comparing analytical results of common data sets.

2) Validate enhancements with previous flight data sets and new data sets (from actual vehicles and simulators). Included will be Shuttle landing data outlier analyses (compared to non-outliers), and data sets from Altair simulators.

3) Incorporate the FORT model’s cost function into SDAT and develop a separate FORT design tool (if schedule and budget conditions permit).

4) Plan in detail for simulator and parabolic flight validation experiments in Years 3 and 4.

We will not pursue applying SDAT or Observer to elderly falls, as stated in our proposal, because that presents a significant distraction to the main focus of our research, which is to understand astronaut SD events and to apply appropriate countermeasures to help astronauts who experience SD. In Year 2 we will focus more attention on designing validation experiments in parabolic flight and in suitable ground-based simulators (e.g., VMS and Desdemona). Parabolic flight experiments will borrow ideas from Borah and Young’s TIFS experiments in the 1980s, which measured in-flight perception of vertical and translatory motions.

Lastly, MIT’s Observer model has aided investigators of aircraft accidents (e.g., 2004 Flash Air 737 fatal crash).

1) Extend SDAT by incorporating MIT’s Observer models. Enhance SDAT with pilot head movement data, and visual attention cues. Validate enhancements with existing and new flight data sets.

Progress: We have enhanced SDAT with MIT models and designed a visual frame-of-reference transformation (FORT) model. The MIT Observer model has been modified so that the strength of gravity is variable, an additional parameter has been incorporated to facilitate dynamic modeling of entry and post-landing OTTR and Tilt-Gain illusions, and the model was extended to calculate perceived velocity and displacement in the perceptual horizontal plane. We have obtained new data sets (Shuttle, Altair simulator) and still have more to obtain (e.g., Apollo data) with which we will validate the enhancements.

2) Extend SDAT assessments to include typical space vehicle illusions: Inversion, Visual Reorientation, Tilt Gain, and Otolith Tilt-Translation Reinterpretation. Validation will include assessment of Shuttle landing data, and Altair simulator data.

Progress: See above. In addition, we devised scenarios to examine predicted perceived orientation via SDAT and Observer analyses, and have begun those analyses of Shuttle and Altair simulator data sets. SDAT has been enhanced with additional illusion sequences, specifically for somatogravic and lateral drift perception illusions.

3) Further extend SDAT by examining alternative visual reference frames. The FORT model is used to predict the cognitive cost of transitioning between reference frames. Validation of Aims 1-3 for SDAT will include parabolic flight experiments.

Progress: See above. In addition, we designed a FORT model and will incorporate its cost portion into SDAT. We have begun to plan flight and simulator experiments to validate all enhancements to SDAT.

4) To further enhance SDAT/SOAS assessor performance, pilot multi-sensory workload is considered in countermeasure selection. Validation experiments are not detailed, but will involve evaluations in ground-based simulators.

Progress: Once we have verified and validated our models, we will assess the efficacy of various countermeasures triggered by SOAS during years three or four, based upon the scenarios devised in item 2, above.

The Spatial Orientation Aiding System (SOAS) is a real-time cockpit aid that has been evaluated in simulators with rated pilots. Both tools incorporate models of the vestibular system and assessor heuristics to predict the epoch and probability of an SD event such as leans, coriolis or graveyard spiral illusions and any other disparities between actual and perceived pitch attitude (somatogravic), roll rate or yaw/heading rate. SOAS assesses multi-sensory workload to determine the types of countermeasures to trigger and when to trigger them.

This project will:

1. Enhance the utility of SDAT/SOAS by including comprehensive mathematical models for vestibular and visual sensory cues, help translate CNS gravitoinertial force resolution into perceived tilt and translation estimates, and revalidate existing aeronautical data sets;

2. Extend the models to describe zero gravity and Shuttle/LSAM landing illusions, validating the models using Shuttle data sets and existing (e.g. ROTTR) theory;

3. Extend SDAT/SOAS to consider multiple visual frames of reference (inside and outside), panel and heads-up (HUD) orientation displays, the effects of visual attention and sensory workload, and the cognitive costs of mental rotation and reorientation. The enhanced SDAT/SOAS from Aims 1-3 will be validated via flight experiments, and;

4. SOAS will be tailored for a lunar landing using multi-sensory workload to choose appropriate countermeasures and their timing.

Countermeasures will include:

* Control command displays;

* Two-dimensional and perspective synthetic/enhanced vision displays;

* Attitude indicator formats tailored for physically redirected, off-velocity vector viewing, and;

* Auditory cues and commands.

SDAT also will help human factors engineers at NASA Johnson Space Center analyze past Shuttle landing incidents and will aid CEV/LSAM landing and ascent trajectory planning. It can aid LSAM cockpit displays, caution and warning system design, workload evaluation, and crew training and mission simulation. SDAT could assist flight surgeons with postflight medical debriefings.