2. Key Findings: In project year 3, our final project year, we completed the development of the simulation capability by integrating the piloted lunar lander, EVA SAFER return to the International Space Station (ISS), EVA SAFER ISS solar array inspection, and MPCV/Orion docking with the ISS vehicle models. The simulation software, along with an operations manual, was made freely available to those who agree to the Draper software licensing agreement. A copy of the Draper simulation station was delivered to NSBRI (National Space Biomedical Research Institute), and is fully capable of executing all the experimentation and demonstration use cases that were developed. In this final project year, an automated flight instructor was designed by our collaborators at University of California (UC) Davis to investigate the effects of three variations of an instructor-model performance-feedback strategy on human performance in a novel SAFER inspection task. Thirty subjects flew SAFER to perform an inspection of the ISS solar arrays. Subjects were initially placed 40 ft away from the array, and were asked to close to 30 ft and hold this distance while inspecting 4 damage" points using a guidance display for navigation to the waypoints. In order of priority, the task included: 1) maintaining the 30 ft distance, 2) minimizing their roll angle to a relative angle of 0 degrees, and 3) navigating the waypoints as quickly and accurately possible. Subjects were also asked to respond to a secondary task in the form of a comm light" as a proxy for workload. Finally, the subjects also made verbal callouts about their fuel level every 5% as a measure of situational awareness.

Subjects were placed into one of three groups. Group 1 acted as a control group, and had an analog distance error display that read from -10 to 10 ft, and digital displays with one significant figure. Group 2 had an analog distance error display that read from -5 to 5 ft, and digital display with two significant figures. Group 3 had the same analog and digital distance error displays as Group 1, and experienced real-time feedback in the form of displays that changed color. Results suggest Groups 2 and 3 both perform better than the control group on the primary and secondary flight tasks. The data also suggests that the real-time performance feedback immediately provided a performance improving effect, even with novice operators. While both Groups 2 and 3 showed performance improvements over the control group, Group 2's workload was increased, as was their time to complete the task.

3. Impact of Key Findings on hypotheses, technology requirements, objectives and specific aims of the original proposal: The development of the integrated simulation platform for running vehicle models, logging data, unobtrusively estimating workload and situation awareness, and providing visualizations and feedback to the pilot has significantly enhanced the capabilities for developing real-time performance metrics. By using typical spacecraft command and control tasks, such as piloted lunar landing, SAFER self-rescue, and Orion/MPCV docking, we have several operational scenarios to test our metrics. The Human Research Program (HRP) Integrated Research Plan gap (SHFE-TASK-01) states, in part, that, …The successful management or evaluation of workload must include a consideration of the nature of individual tasks that operators must perform, the combinations of tasks that are performed during a work period, priorities among tasks, and individual differences among operators. The development and evaluation of real-time performance metrics in representative operational settings—which include task performance, workload, and situational awareness, and are measured objectively as well as subjectively—will provide valuable data for the validity assessment. In project year 3, through conducting the experiment investigating the effects of an instructor-model performance-feedback strategy on human performance in a novel Simplified Aid for EVA Rescue (SAFER) inspection task revealed many interesting aspects. Not surprisingly, providing real-time performance feedback immediately provided a performance improving effect, even with novice operators. However, as the sensitivity of the feedback was increased (i.e., tighter performance criteria), the time to complete the task increased as well as the reported operator workload. The significance of this is that there is likely an optimum set of feedback sensitivity parameters that balance the operators task completion performance with their workload. The installation of a copy of our simulation station at NSBRI Headquarters is a valuable tool for collaborative use by researchers. Our team's continual pursuit of scientific investigation into the dynamic interaction between a pilot and their spacecraft led to a workshop on Piloted Spacecraft Guidance & Control Systems and Human Performance.

4. Proposed research plan for the coming year: This was the final year of the project. There is no proposed research plan for the coming year.

In this final project year, an automated flight instructor was designed by our collaborators at UC Davis to investigate the effects of three variations of an instructor-model performance-feedback strategy on human performance in a novel SAFER inspection task. Thirty subjects flew SAFER to perform an inspection of the ISS solar arrays. Subjects were tasked with: 1) maintaining the 30 ft distance, 2) minimizing their roll angle to a relative angle of 0 degrees, and 3) navigating the waypoints as quickly and accurately possible. Subjects were also asked to respond to a secondary task as a proxy for workload, and made verbal fuel percentage callouts as a measure of situational awareness. Group 1 had an analog distance error display that read from -10 to 10 ft, and digital displays with one significant figure. Group 2 had an analog distance error display that read from -5 to 5 ft, and digital display with two significant figures.

Results suggest Groups 2 and 3 both perform better than Group 1 on the primary and secondary flight tasks. The data also suggests that the real-time performance feedback immediately provided a performance improving effect, even with novice operators. While both Groups 2 and 3 showed performance improvements over Group 1, Group 2's workload was increased, as was their time to complete the task. The development of the integrated simulation platform for running vehicle models, logging data, unobtrusively estimating workload and situation awareness, and providing visualizations and feedback to the pilot has significantly enhanced the capabilities for developing real-time performance metrics and addressing the HRP TASK-01 Gap. By using typical spacecraft command and control tasks, such as piloted lunar landing, SAFER self-rescue, and Orion/MPCV docking, we have several operational scenarios to test our metrics.

The HRP Integrated Research Plan gap (SHFE-TASK-01) states, in part, that, …The successful management or evaluation of workload must include a consideration of the nature of individual tasks that operators must perform, the combinations of tasks that are performed during a work period, priorities among tasks, and individual differences among operators. The installation of a copy of our simulation station at NSBRI Headquarters is a valuable tool for collaborative use by researchers. Our team's continual pursuit of scientific investigation into the dynamic interaction between a pilot and their spacecraft led to a workshop on Piloted Spacecraft Guidance & Control Systems and Human Performance.

2. Key Findings: In project year 2, we furthered the development of the simulation capability through integration of additional vehicle models, and testing the calculation of pilot flight performance, workload, and situation awareness metrics in real-time, without adding additional hardware to the simulator or requiring interrupts to the flight task. The baseline piloted lunar landing simulation was modified to include real-time estimation and plotting of flight performance (entropy of the hand controller inputs, root mean square error of the difference between the actual and guidance recommended pitch attitude), response time to the two-choice secondary task for workload estimation, and an estimate of situation awareness via the accuracy of required key system state callouts through an automatic speech recognition engine. These metrics were calculated in real-time throughout the trial, and are displayed through an engineering-level view. A human subject experiment (n = 26) was conducted using the lunar landing simulation in an effort to validate the performance of the automatic speech recognition (ASR) engine, as well as to collect data to develop unobtrusive workload estimation metrics that do not require a two-choice secondary task. The ASR analysis found that the system performed well (the ASR algorithm correctly recognized 838 of the 1035 valid callouts, for a precision of 0.81), and also identified several areas for further development to improve performance. We also developed a new real-time statistical approach to estimating flight performance by computing the percentage of pitch axis error in relation to pre-set bounds (comparison of actual vs. guidance recommended). The real-time performance metrics (flight, workload, and situation awareness) infrastructure was integrated with an extravehicular activity (EVA) simplified aid for EVA rescue (SAFER) jetpack International Space Station (ISS) self-rescue simulation in our portable ground station for future experimentation. The baseline Orion/MPCV piloted simulation for docking with the ISS was also implemented and tested in the portable ground station. It is these simulations and the data that we collect from the experiment that will enable the development of robust metrics that can be presented to the pilot for making operations more safe and efficient.

3. Impact of Key Findings on hypotheses, technology requirements, objectives and specific aims of the original proposal: The development of the integrated simulation platform for running the vehicle models, recording/logging data, unobtrusively estimating workload and situation awareness, and providing visualizations and feedback to the pilot has significantly enhanced the capabilities for developing real-time performance metrics. By using typical spacecraft command and control tasks, such as piloted lunar landing, SAFER self-rescue, and Orion/MPCV docking, we have several operational scenarios to test our metrics. The Human Research Program (HRP) Integrated Research Plan gap (SHFE-TASK-01) states, in part, that, …The successful management or evaluation of workload must include a consideration of the nature of individual tasks that operators must perform, the combinations of tasks that are performed during a work period, priorities among tasks, and individual differences among operators. The development and evaluation of real-time performance metrics in representative operational settings—which include task performance, workload, and situational awareness, and are measured objectively as well as subjectively—will provide valuable data for the validity assessment.

4. Proposed research plan for the coming year: In project year 3, we aim to conclude the analysis of the piloted lunar landing study. This analysis will produce a quantitative assessment of the ASR engine performance as a real-time situation awareness metric, as well as an unobtrusive workload estimation metric based on flight performance analysis (compared against the two-choice secondary task a as the gold standard). The presentation of the results to affect flight and mission performance will be prototyped in the simulator in a manner that seamlessly integrated with the existing displays. In collaboration with our team members at the University of California (UC) – Davis and the NASA Johnson Space Center (JSC) Virtual Reality Laboratory, we aim to conduct an experiment using the SAFER self-rescue simulation to develop and test our metrics engine, and to compare its robustness to novel scenarios through data collected during JSC VR Lab simulations. We will also complete the development of the Orion/MPCV docking simulation for human subject testing that is planned for later in project year 3. Lastly, we will deliver and install a copy of our simulation station at National Space Biomedical Research Institute (NSBRI) Headquarters to provide a foundational capability for subsequent test and evaluation with operators and subject matter experts.

2. Key Findings: In project year 1, we completed the development of the system architecture for integrating the vehicle models with our simulation framework, calculating objective workload metrics from flight performance data, performing speech recognition for situation awareness estimation, and rendering an out-the-window view using NASA's Engineering Dynamic On-board Ubiquitous Graphics (DOUG) Graphics for Exploration (EDGE) package. We developed a portable ground station, which includes a single PC, two joysticks (a rotational hand controller (RHC) and a translational hand controller (THC)), five monitors, speakers, and a microphone. Three prior lunar landing simulations – all developed under prior NSBRI projects (Project SA01604 Fuel Contour Display, HFP02001 Mode Transition and Failure Detection models) – and the SAFER EVA self-rescue scenario were integrated in the simulation. These simulations will provide the basis for the planned year 2 experimentation for developing and validating the implemented real-time performance metrics. Three real-time objective workload metrics were prototyped for implementation with the Mode Transition simulation (Hainley et al., 2013). These included the baseline implementation of a two-chose secondary task response time, an analysis of the entropy of the RHC inputs, and the root mean square error (RMSE) of the difference between the actual and guidance recommended attitude during piloted flight. The RHC entropy and attitude RMSE metrics were chosen based on post-hoc analysis of prior experiment data. In addition, we implemented a real-time speech recognition engine to analyze the accuracy of required key system state callouts as a measure of situation awareness (see Hainley et al., 2013). In this implementation, the utterances spoken by the user are recognized by the speech engine and then compared against the actual simulated vehicle state and are scored correct based on temporal (e.g., did they make the callout within 2 seconds?) and spatial (e.g., did they make the callout within 10 feet of the actual altitude?) boundaries. It is this feedback – both from the secondary task response time, mental workload, and situation awareness metrics – that we plan to further develop in the out years of the project to refine the presentation to the pilot for making operations more safe and efficient.

3. Impact of Key Findings on hypotheses, technology requirements, objectives, and specific aims of the original proposal: The development of the integrated simulation platform for running the vehicle models, recording/logging data, unobtrusively estimating workload and situation awareness, and providing visualizations and feedback to the pilot has significantly enhanced the capabilities for developing real-time performance metrics. By using typical spacecraft command and control tasks, such as piloted lunar landing, we have an initial operational scenario to test our metrics. The HRP Integrated Research Plan gap (SHFE-TASK-01) states, in part, that, …The successful management or evaluation of workload must include a consideration of the nature of individual tasks that operators must perform, the combinations of tasks that are performed during a work period, priorities among tasks, and individual differences among operators. The development and evaluation of real-time performance metrics in representative operational settings—which include task performance, workload, and situational awareness, and are measured objectively as well as subjectively—will provide valuable data for the validity assessment.

4. Proposed research plan for the coming year: In project year 2, we aim to begin conducting human-in-the-loop experiments with our lunar landing Mode Transition experiment to quantitatively compare the real-time performance metrics calculations with our baseline post-hoc measures of flight technical error, workload, and situation awareness. The real-time metrics that are analyzed and collected on-the-fly will be quantitatively compared against the prior approach to analyze the data after-the-fact. This will be done in close collaboration with our team mates at the University of California, Davis. In addition, an operationally relevant approach for fusing the vehicle flight data, workload, and situation awareness data will be developed, as well as an approach for providing feedback will be prototyped and evaluated. In project year 2, we also aim to develop and integrate Orion/MPCV rendezvous and docking and atmospheric entry models with our simulation. Real-time performance metrics will also be developed and integrated with this simulation, with the goal of having consistent metrics between classes of simulations. Lastly, we propose to deliver and install a copy of our simulation station at NSBRI Headquarters to provide a foundational capability for subsequent test and evaluation with operators and subject matter experts.

Three real-time objective workload metrics were prototyped for implementation with the lunar landing Mode Transition simulation (Hainley et al., 2013). These included the baseline implementation of a two-chose secondary task response time, an analysis of the entropy of the RHC inputs, and the root mean square error (RMSE) of the difference between the actual and guidance recommended attitude during piloted flight. In addition, we implemented a real-time speech recognition engine with our simulation to analyze the accuracy of required key system state callouts as a measure of situation awareness (see Hainley et al., 2013). In this implementation, the utterances spoken by the user are recognized by the speech engine and then compared against the actual simulated vehicle state and are scored correct based on temporal and spatial boundaries. It is this feedback – both from the secondary task response time, mental workload, and situation awareness metrics – that we plan to further develop in the out years of the project to refine the presentation to the pilot for making operations more safe and efficient.

Planning for the start of human subject experimentation in year 2 was initiated. This included the submission and approval of an experimental protocol to the University of California, Davis IRB, the shipment of a copy of the simulation control station with integrated lunar landing simulations, and the identification of research objectives and testable hypotheses for the investigation and validation of in-situ real-time flight performance, workload, and situation awareness metrics. In addition, the framework for fusing these metrics for providing feedback to the pilot and/or control system to make operations more safe and efficient was also completed.

REFERENCE (reported in NSBRI "Human-Automation Interactions and Performance Analysis of Lunar Lander Supervisory Control" project):

Hainley CJ Jr, Duda KR, Oman CM, Natapoff A. "Pilot performance, workload, and situation awareness during lunar landing mode transitions." Journal of Spacecraft and Rockets. 2013 Jul;50(4):793-801. http://dx.doi.org/10.2514/1.A32267

1) Define the system architecture, integrate vehicle, system and environment models, and perform a critical analysis of the operationally relevant tasks to identify the specifics of candidate metrics for performance, workload, and situation awareness,

2) Develop and integrate real-time performance analysis techniques with the vehicle/system models that can run in real-time and provide immediate feedback to the operator, and

3) Conduct a series of simulations and experiments to baseline performance, workload, and situation awareness in each of the tasks.

Vehicle, system, and environment models, as well as task-specific displays and controls will be available to the operator for the following selectable scenarios: a) piloted MPCV/Orion atmospheric entry, b) piloted MPCV/Orion rendezvous, proximity operation, and docking with the ISS, c) ISS EVA/SAFER operations, and d) piloted lunar landing. We intend to leverage extensively the performance assessment methods developed under NSBRI Project HFP02001 to quantify performance, workload, and situation awareness as temporal measures during complex system automation mode transitions (e.g., Hainley, 2011; Hainley, Duda, et. al, in review).