2. Key Findings: In project year 4, we completed our investigations of flight performance and failure detection in the NASA Ames VMS, quantification of operator visual attention in the Draper Laboratory fixed-base simulator, closed-loop human-system model for representing the human in a complex system, and analyzed dynamic task allocation between the human and the system in operational implementations. In the two VMS experiments, participants with flight experience detected and diagnosed system-level failures with varying levels of control and with and without motion cues. The effect of how often a failure appeared in the test matrix was also evaluated. Results indicated that the pilots had a correct hit rate of 90.7 percent of failure trials and a correct rejection rate of 90.2 percent of no-failure trials. There was no effect of motion cues on flight performance or failure detection. Failures were also more easily detected in the high frequency (75% of trials had a failure) condition than in the low frequency (25%) condition. In all cases, workload and situation awareness decreased following a planned mode transition from automatic flight to manual control. In the Draper Laboratory visual attention experiment, there was an effect of mode transition on the average dwell duration and number of visual fixations, as measured by an eye tracker. These effects were due to the transition's final mode, and not the initial mode or the direction of the mode transition (whether increasing or decreasing level of automation). Average dwell duration prior to the failure detection was found to be higher on the instruments that were used to detect the failure, as compared to the no-failure conditions. Lastly, the results of the experiments have been used to update the integrated human-system model. The modeling effort represents the cognitive processes and action responses of the human, who can act as both a flying pilot as well as a supervisory pilot.

3. Impact of Key Findings on hypotheses, technology requirements, objectives and specific aims of the original proposal: The results of the pilot visual attention experiment (Aim 3) and VMS failure detection experiments (Aim 4) are critical for the understanding of pilot performance in nominal and off-nominal scenarios. The data collected from the experiments provides critical information for parameter justification for the identified operational scenario within the integrated human-system model (Aim 2). The experiments further investigated our previous identification of the effect of flight control mode transitions on workload and situation awareness, by quantifying visual attention on primary and supervisory flight instruments. The experimentation represents typical spacecraft command and control tasks where the flying pilot was responsible for selecting a landing aimpoint using information from an on-board hazard detection system, and then either supervising the autoflight system or manually commanding a representative lunar landing vehicle. The system-level failures that were introduced represent plausible failures that would be detected on either the primary flight displays or secondary displays. By quantifying performance, workload, and situation awareness across these experimental conditions, we've demonstrated a robust set of metrics that can be applied throughout a system's design and verification cycle to benchmark and evaluate the implementation (SHFE-TASK-01: How can workload measures and tools be developed to unobtrusively monitor and trend workload throughout the mission design and verification cycle in a consistent manner?). The integrated human-system performance modeling work and MATLAB/Simulink library (Aim2) addresses the NASA Human Research Program (HRP) Risk of Poor Critical Task Design, specifically the Gap associated with model-based tools that can assist in the early-stage research and drafting of spacecraft systems and task procedures (SHFE-TASK-02: What model-based HF Tools can assist with the design and evaluations of spacecraft systems and task procedures?). The parameterized model-based simulation approach enables a systematic evaluation of task allocation, task parameters, and human parameters on system performance. The partnership with NASA Ames Research Center and the use of the VMS experimentation has both provided us valuable data for the modeling and simulation of typical spacecraft command and control tasks, and has advanced their trajectory simulation capability through the implementation of a landing point designation phase and off-nominal scenarios.

4. Proposed research plan for the coming year: This is the final report for this project. Aspects of the project, specifically the flight performance, workload, and situation awareness metrics will be further investigated, developed and evaluated in additional spaceflight operational scenarios through National Space Biomedical Research Institute (NSBRI) Project HFP03401--Metrics and Methods for Real-Time Task Performance Assessment. This newly funded project aims to transition these metrics to other NSBRI Human Factors and Performance (HFP) Team investigators, and make the capability more broadly available to the research community.

The research has also developed a new situation awareness metric – one that allows for continuous measurement without interrupting the reporter or simulation – and has been used in several experiments. This non-invasive method requires the participant to verbally callout specific vehicle/system states that are pertinent to the task that they are executing, and we record the correctness of the callout. Within a trial this gives an indication of which points they missed callouts, and across trials it provides temporal comparisons of task sequences/events that result in lower situation awareness. This method could be applied to many land, sea, or space-based systems where there is a need to assess operator situation awareness over time without interfering with the activity, or interrupting the simulation. Specific examples for space operations and exploration may include teleoperation/remote manipulator operation and near-Earth object/asteroid rendezvous and proximity operations.

In VMS Experiment 1, the pilots performed a landing point designation and then transitioned to one of three manual modes. Three failure types were modeled. Results indicated that the pilots had a correct hit rate of 90.7% of failure trials and a correct rejection rate of 90.2% of no-failure trials. There was no effect of motion cues on flight performance or failure detection. There was a significant effect of control mode and failure type in time to detect failures, although the significance of control mode depended on the failure type. In all cases, quantitatively evaluated workload and situation awareness decreased following a planned mode transition from automatic flight to manual control (as seen in prior work).

In VMS Experiment 2, the pilots initially performed a landing aimpoint designation in autopilot, then initiated a transition where the vehicle would maintain in high automation (autopilot) or enter a low automation condition. Their tasks were to fly to the designated landing aimpoint, while making call-outs and detecting and diagnosing a failure. Two failures were modeled, and the failure frequency (25% or 75%) was a between-subjects variable. Failures were more easily detected in the high frequency (75%) condition than in the low frequency condition. Additionally, participants were more hesitant to declare they observed a failure when they occurred more frequently and when participants were manually in control. The Draper Laboratory fixed-base simulator was modified to include an eye tracker for measuring operator visual fixations. Each trial began in one of three vehicle control modes, and then transitioned to one of the other two. The same three failures as VMS Experiment 1 were modeled. Subjects were asked to fly the vehicle, report workload, make verbal callouts, and detect and diagnose a failure. There was an effect of mode transition on both average visual dwell duration and number of fixations for all instruments. These effects were due to the transition's final mode, and not the initial mode or the direction of the mode transition (whether increasing or decreasing level of automation).

The results of the VMS and Draper Laboratory experiments have been used to update the integrated human-system model. The model represents the cognitive processes and action responses of the human, who can act as both a flying pilot as well as a supervisory pilot. The model blocks have been updated in the Human Performance Model (HPM) library, and the integrated model has been used to run sensitivity analyses to the effect of parameter variation on simulated system performance.

2. Key Findings: In project year 3, we completed our investigation of human decision making and risk-taking during simulated lunar landing and further developed the human performance model library. In the experimentation, subjects were asked to select a landing aimpoint within a pre-determined region, and the probability of a human-piloted vs. automatic landing was varied. We hypothesized that the placement of the landing aimpoint would vary with the probability of human-piloted versus automatic flight and whether estimated touchdown errors were remembered by the subjects from earlier in the experiment or presented graphically on scatter plots. Results showed that subjects designated landing points that compensated for estimated touchdown dispersions and knowledge of the probabilities of manual versus automated flight. Subjects made more complete landing selection compensations when estimating touchdown dispersion from graphical plots rather than from memories of previous simulated landings. Parameterized human performance models were integrated in MATLAB/Simulink as a library that can be used by system designers to represent the human in a common modeling framework. In project year 3, three models of human performance -- Action, Attention, and Perception – have been implemented. Future implementations will include blocks of the detection and response to failures, spatial orientation, and variants on decision making. In preparation for the Aim 4 experiments at NASA ARC, significant modifications have been made to the VMS lunar landing cab and simulation. These include 1) a landing point designation phase in the trajectory, 2) the transition to manual control from automatic flight, and 3) the implementation of system failures that the experimental subject will be tasked with detecting and identifying.

3: Impact of Key Findings on hypotheses, technology requirements, objectives and specific aims of the original proposal: The results of the fixed-base lunar landing experiment (Aim 3) are important for quantifying flight performance under varying task allocation (probability of human piloted vs. automatic flight), and the influence of prior landing accuracy on the selection of landing aimpoints. This work leverages the hierarchical task analysis performed in Aim 1, and provides key data for the development of representative models of human performance for Aim 2. The experimentation represents typical spacecraft command and control tasks where the flying pilot was responsible for selecting a landing aimpoint from using information from an on-board hazard detection system, and then either supervising the autoflight system or manually commanding the flight path and attitude of a representative lunar landing vehicle. The human performance modeling work addresses the NASA HRP Risk of Poor Critical Task Design, specifically the Gap associated with model-based tools that can assist in the design and evaluation of spacecraft systems and task procedures. The model-based simulation approach enables a systematic evaluation of task allocation, task parameters, and human parameters on system performance. The partnership with NASA ARC and the use of the VMS for Aim 4 experimentation has advanced their trajectory simulation capability through the implementation of a landing point designation phase and off-nominal scenarios.

4. Proposed research plan for the coming year: In project year 4, there are four elements of our research plan that we aim to advance: 1) Implementation of models of human failure detection and response to off-nominal scenarios in the human performance model library. This will allow us to analyze the system response to degraded states, and analyze the human-automation task allocation and system performance using Draper Laboratory's performance and reliability analysis techniques. 2) Conduct fixed-base and motion-base experimentation using the Draper Laboratory simulator and NASA Ames VMS to examine the effects of simulator motion on failure detection and identification, workload, situation awareness, and flying performance. 3) Complete the packaging of human performance models within MATLAB/Simulink so that they are a stand-alone library. This will include parameterized models for simulation and analysis scripts for model performance and parameter sensitivity analysis. 4) Further the analysis of dynamic task allocation through investigation of pilot performance during adaptive and adaptable automation. Develop an adaptive automation taxonomy and model for evaluating human performance in complex systems. In addition, we will continue to work with other NSBRI HFP Team investigators to identify potential future collaborations that leverage the Team's expertise and integrate this simulation capability and model-based human-system design work.

The research has also developed a new situation awareness metric – one that allows for continuous measurement without interrupting the reporter or simulation. This non-invasive method requires the participant to verbally callout specific vehicle/system states that are pertinent to the task that they are executing, and we record the correctness of the callout. Within a trial this gives a indication of which points they missed callouts, and across trials it provides temporal comparisons of task sequences/events that result in lower situation awareness. This method could be applied to many land, sea, or space-based systems where there is a need to assess operator situation awareness over time. Specific examples for space operations and exploration may include teleoperation/remote manipulator operation and near-earth object/asteroid rendezvous and proximity operations.

The Human Performance Library (HPL) is a collection of human performance models that enables the user to model the human within an existing system model. The HPL integrates with the user's Simulink Library Browser. Three models of human performance -- Action, Attention, and Perception – have been implemented. The Attention block includes three separate attention models (selected signals, fixed-interval scan, and maximum error) which drive the simulated spotlight of attention. The Perception block simulates the perception of an input by allowing a user to specify bias, gain, and noise. The Action block allows a user to input a bias, gain, and time delay to simulate response to an input. Future implementations of the HPL will include blocks of the detection and response to failures, spatial orientation, and variants on decision making. All will include user-configurable parameters to allow for the specification of a number of different scenarios and models of human response.

In preparation for the Aim 4 experiments at NASA ARC, significant modifications have been made to the VMS lunar landing cab and simulation. These include the 1) implementation of a landing point designation phase in the trajectory, 2) the transition to manual control modes from automatic flight, and 3) the implementation of system failures (attitude control thruster stuck on, landing radar failure, and fuel leak) that the experimental subject will be tasked with detecting and identifying. Companion pilot experiments have been started in the Draper Laboratory fixed-base simulator.

2. Key Findings: In project year 2, we completed our initial human-system model development. Our simulation varied the human performance model parameters, and recorded system performance to determine the sensitivity to variations in human activity. Landing accuracy and fuel usage were two system-level parameters to score performance against. Two key findings include: 1) the best combined fuel and landing accuracy was found when the human does the decision making, but aided by automated flight, and 2) variability in fuel savings when the human is flying is greater than automated flying, but the average is not always greater. This supports the human-system collaboration hypothesis that the combined human-automation performance is better than either the human or the automation alone. We also completed an experiment investigating "graceful" mode transitions within complex systems. Subjects flew trajectories that transitioned from a fully automatic flight control mode to one of three manual flight control modes. Workload was measured using the Modified Bedford Scale and secondary task response time. Verbal callouts of altitude, fuel, and location - provided a measure of situation awareness (SA). Flight performance was evaluated using the pitch axis tracking error. The key findings include: 1) secondary task response time and subjective workload all significantly increased following the mode transition, 2) the subjects' Modified Bedford reports, when ranked, showed unanimous agreement that workload was lowest prior to the transition, and highest during it, and 3) SA callout accuracy decreased significantly after the transition. With the new SA metric, we were able to demonstrate for the first time a short term decrease in SA during difficult mode transitions. We also found that workload depended on the number of control loops the subject was required to close.

3: Impact of Key Findings on hypotheses, technology requirements, objectives and specific aims of the original proposal: The results of the human-automation task allocation modeling and simulation supports the human-system collaboration hypothesis that the combined human-automation performance is better than either the human or the automation alone. This work leverages the hierarchical task analysis performed in Aim 1, and expands on Aim 2. In addition to the implementation of human performance models, we developed analogous automation performance models - allowing us to simulate performance under varying task allocations. The results of the simulation informed the landing point designation and human decision making hypotheses for Aim 3 experimentation. Several of these tasks models will be implemented as "typical spacecraft command and control tasks." The objective of research Aim 1 was expanded to investigate "graceful" transitions between automation modes. Accidents and incidents in aviation, maritime, and space were analyzed to produce a generalized set of design guidelines and metrics to quantify the workload, situation awareness, and system performance changes during mode changes. An experiment in the context of piloted lunar landing was performed to quantify the "gracefulness" of transitions between automation modes. This investigation filled a void in the task allocation analysis literature by considering dynamic task allocation, as well as utilizing a novel method for assessing situation awareness in near-real-time using verbal callout accuracy. The results of this experimentation contribute to Aims 1 and 3, and the system performance, workload, and situation awareness metrics will be used to inform the modeling effort in Aim 2.

4. Proposed research plan for the coming year In project year 3, there are four elements of our research plan that we plan to advance: 1) Expand on the model development in Aim 2 to include human failure detection and response to off-nominal scenarios. This will allow us to analyze the system response to degraded states, and analyze the human-automation task allocation and system performance using Draper Laboratory's performance and reliability analysis techniques. 2) Complete the human decision making and action experiment during landing point designation and approach-to-landing tasks. This experiment was started in year 2, and the results will be used to validate the Aim 2 models. 3) Initiate the packaging of the human performance models within MATLAB/Simulink to be a stand-alone library. 4) Coordinate the Aim 4 research plan using the NASA Ames VMS. In addition, we will continue to work with other NSBRI HFP Team investigators to identify potential future collaborations that leverage the Team's expertise and integrate this model-based human-system design work.

The research has also developed a new situation awareness metric - one that allows for continuous measurement without interrupting the reporter or simulation. This non-invasive method requires the participant to verbally callout specific vehicle/system states that are pertinent to the task that they are executing, and we record the correctness of the callout. Within a trial this gives an indication of which points they missed callouts, and across trials it provides temporal comparisons of task sequences/events that result in lower situation awareness. This method could be applied to many land, sea, or space-based systems where there is a need to assess operator situation awareness over time. Specific examples for space operations and exploration may include teleoperation/remote manipulator operation and near-earth object/asteroid rendezvous and proximity operations.

Case studies of accidents and incidents in aviation, maritime, and space were analyzed to produce a generalized set of automation design guidelines and metrics to quantify the workload, situation awareness, and system performance changes during automation mode transitions. We completed an experiment in year 2 investigating mode transitions within complex systems - to determine system attributes which allow the transitions to occur "gracefully." We used piloted lunar landing was our scenario. Subjects flew approach trajectories that transitioned from a fully automatic flight control mode to one of three manual flight control modes. Workload was measured using the Modified Bedford Scale and secondary task response time. A tertiary task - verbal callouts of altitude, fuel, and location - provided a measure of situation awareness. Flight performance was evaluated using the pitch axis tracking error. The key findings include: 1) Secondary task response time and subjective workload all significantly increased following the mode transition, 2) the subjects' Modified Bedford reports, when ranked, showed unanimous agreement that workload was lowest prior to the transition, and highest during it, 3) situation awareness verbal callout accuracy decreased significantly after the transition, and 4) pitch axis tracking mean square error following the transition was significantly greater in re-designation trials. A manuscript is in preparation. Since this experiment utilized a new SA metric, we were able to demonstrate for the first time a short term decrease in SA during difficult mode transitions. We also found that workload depended on the number of control loops the subject was required to close.

2. Key Findings - During the first year of the project, we created a hierarchical task analysis (HTA) for lunar landing, starting with Apollo and adding Autonomous Landing and Hazard Avoidance Technology (ALHAT) Project functions and capabilities, to generate sufficient detail for model development. Several task themes emerged - decision making, flying, and sub-system supervisory/monitoring tasks - which provide the basis for the development of several classes of "typical space vehicle command and control tasks" for the model library development. Given the ALHAT requirement to support crewed, cargo, and robotic landings, we identified Landing Point Designation (LPD) as an important activity for the case study. A systems-level model architecture was developed that includes a task network (Stateflow), human performance model (HPM) library (attention, perception, decision making, action), and a vehicle dynamics model (ALHAT Entry Descent and Landing Simulation) - all within MATLAB/Simulink. Within the model, each task or function can be assigned to either a human or automation. Also, failure modes, probabilities, and conditions can be specified a various points within the parameterized model.

During the HTA development, we found that previous research had focused on the analysis of a static allocation of functions between humans and automation. However, in most complex systems, operators switch between levels of automation resulting from switching between modes. An important aspect of these transitions is the ease of it, so that there is not a discontinuity between distant levels of automation, or there is not an unknown or ambiguous status of the system when switching into a new mode. We have defined the term "graceful transition" as "the ability of an operator of a complex system to change between levels of automation or levels of supervisory control (including automation modes) while maintaining control and awareness of the system without sacrificing system performance or mission objectives."

3. Impact of Key Findings on hypotheses, technology requirements, objectives and specific aims of the original proposal - The identification of decision making, flying, and supervisory/monitoring task classes during the HTA development provided a structure for the model library development. In addition to the types of "typical space vehicle command and control tasks," we were able to identify the key elements within those tasks to develop the human and automation blocks that could be parameterized within the overall model. The implementation of the human-system model within the MATLAB/Simulink environment has eased the development due to the mix of existing and customizable blocks, and eliminates the need for communication protocols between software modules.

We expanded the objective of the first aim to include the kinds of interfaces that facilitate graceful transitions in automation modes and in levels of supervisory control. During our HTA, we recognized that previous research had focused on the analysis of only static allocation of functions between humans and automation which represents a gap in the research and analysis methods of complex systems. To support this aim, we are analyzing multiple complex systems (air, space, sea) to produce a generalized set of automation design guidelines and metrics to quantify these transitions for evaluation of complex systems with multiple modes.

4. Proposed research plan for the coming year - In year 2 of the project, we will expand on the development of the human-system model to include multiple phases within a lunar landing trajectory. We will continue to refine the model for landing point designation, developing the ALHAT-based representation, as well as that used during Apollo. Simulation and analyses will be conducted to quantitatively analyze the performance differences between the Apollo and ALHAT function allocations as well as to determine sensitivities to critical parameters within the model. A human subject experiment is planned to validate the model and key findings. In addition, we plan to deliver a comprehensive review of interactions with multiple automation modes and mode transitions within complex systems - maritime, aircraft, and spacecraft. This review will also include proposed methodologies for analyzing the transitions between modes ("graceful transition"), including metrics for quantifying the gracefulness of a system as well as design rules to assist in the development of a system to ensure the ease of transition.

The systems-level model architecture includes a task network, human performance model (HPM) library, and a vehicle dynamics model - all within MATLAB/Simulink. The LPD task network is implemented using Stateflow, which allows task completion and transition conditions. The vehicle dynamics builds on the ALHAT Entry Descent and Landing Simulation which is a representation of the vehicle dynamics and guidance, navigation, and control algorithms, and the HPM library includes functions that follow the "see-think-decide-do" paradigm, and include abstract representations of attention, perception, decision making, and action. Within the network, functions can be assigned to either the human or the automation so subsequent analyses can inform the relative benefit of one function allocation versus an alternative. Failure modes, associated probabilities and conditions can also be specified and added to the parameterized model.

During the development of the HTA, we identified a gap in the analysis of complex system automation mode transitions. Previous research focused on the analysis of static allocation of functions between humans and automation; however, in most complex systems, operators switch between levels of automation as a result from switching between modes. These transitions are usually planned and initiated by the operator, but can also result from unplanned events such as a failure or mode confusion. An important aspect of these transitions is the ease of it, so that there is not an abrupt jump between distant levels of automation, or unfamiliarity with the status of the system when operating in the new mode. We have defined the term "graceful transition" as "the ability of an operator of a complex system to change between levels of automation / levels of supervisory control (including automation modes) while maintaining control and awareness of the system without sacrificing system performance or mission objectives." We are analyzing complex systems to produce a generalized set of design guidelines and metrics to quantify these transitions and evaluate systems with multiple modes.

There are four integrated specific aims:

(1) Perform a critical analysis of Apollo human-automation interactions and task allocation during terminal descent through touchdown, as well as the information requirements, decision making process and selection of action,

(2) Develop a closed-loop pilot-vehicle model, integrating vehicle dynamics, human perception, decision making and action, and analyzed using reliability analysis techniques in MATLAB/Simulink® to quantify system performance.

(3) Conduct experiments in the Draper Laboratory fixed-base lunar landing cockpit simulator to validate critical parameters within the integrated pilot-vehicle model, and determine decrements in flight control performance and pilot workload during nominal and off-nominal scenarios.

(4) Extend the dynamic model to include the effect of spatial orientation on system performance and conduct experiments on the NASA Ames Vertical Motion Simulator to investigate the effects of motion cues on pilot perception, decision making, and control during instrument failures, or loss of visual references during terminal descent through touchdown.

This proposed research will produce an integrated human-system model that includes perception, decision making, and action as an early-stage model-based simulation design tool to identify the appropriate human-automation task allocation and information requirements to enable safe and successful lunar landing.