NOTE: End date changed to 03/31/2025 per A. Beitman/JSC (Ed., 9/19/24)

NOTE: End date changed to 12/31/2024 per A. Beitman/JSC (Ed., 1/20/23)

NOTE: End date changed to 5/14/2023 per S. Huppman/HRP and NSSC information (Ed., 3/3/2020)

Future deep space missions will present new challenges for crew, and increased risks to human performance due to the stress, fatigue, radiation exposure, and isolation that characterizes these missions. In addition, crew will no longer be able to depend on timely support from Mission Control due to distance from the Earth, but will have to work autonomously, while maintaining high performance. Mission Controllers may not be available to answer questions, check system status, assist with procedures, monitor for errors, or troubleshoot problems. Greater crew autonomy will increase dependence on automated systems, and design of these automated systems must be driven by sound human-system integration standards and guidelines in order to ensure mission success. Historically, crew have had very limited dependence on automated systems, thus crew will be faced with a new way of working that may put situation awareness (SA) at risk. We must develop methods for promoting good situation awareness in the automated systems that will most certainly be part of future deep space vehicles and habitats.

Procedure automation is a promising technology for reducing crew workload. We define procedure automation as technology that automates the selection or execution of procedural tasks. Structuring the work of automation according to human procedures should improve the transparency of automation actions. This approach provides a means for establishing common ground about ongoing tasks to improve operator understanding of automation behavior.

New technologies such as adaptive, multimodal, augmented reality displays can offer the benefits of information presentation tailored to meet the needs of each crewmember, taking into consideration the current state of that crewmember (e.g., sleep-deprived, high workload), as well as the current state of his/her environment and ongoing activities (e.g., emergency situation, time-critical operations).

We combine technology for procedure automation with technology for augmented reality multi-modal (ARMM) user interfaces using Microsoft Hololens head-mounted display to provide a virtual task assistant to assist crew in performing procedural work. This virtual task assistant identifies procedures to perform, aids performing actions in crew procedures, and summarizes actions taken by the human-automation team to assist crew in preparing for tasks and taking over tasks from other team members.

Four studies were conducted to evaluate the effects of a virtual task assistant combining procedure automation with augmented reality multi-modal (AARM) user interfaces on human task performance. These studies will achieve the following aims:

Aim. Determine best methods to improve situation awareness and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance.

Aim. Determine best practices to improve crew performance when using virtual task assistant to aid in task transitions (task handovers, automation interruption).

Aim. Determine best methods for display of procedural work to improve crew autonomy and satisfaction

The proposed work addresses a number of gaps in the Human Research Program Human Factors and Behavioral Performance risks. This project will provide guidelines for designing effective human-automation systems and evaluate human-automation performance for exemplar procedure automation systems. This project also will provide guidance for the application of multi-modal and adaptive displays and control to Human-Computer Interaction (HCI) design for long duration operations.

The Virtual Intelligent Task Assistant (VITA) project is leveraging augmented reality platforms and toolkits for augmented reality apps to develop a virtual task assistant that can be used to assist users with procedural task work on the job. Our technical approach is innovative in that new procedural tasks can be supported without custom software development. Our experimental research is distinguished by investigating effective task assistance for maintenance or assembly tasks where hands-free operation of task assistance is beneficial. For the first study we investigated best techniques for using augmented reality task assistance when assembling small rovers that are held in the hands during assembly. Subsequent studies investigate other aspects of using augmented reality for assembly tasks, including use of 3D images in procedures and virtual notification for task milestones and transitions.

This technology and associated research findings have potential benefit to NASA for the assembly, maintenance, and repair of aircraft, spacecraft, habitats, and robotics. This technology and associated research findings also have broader potential benefit for any organization performing assembly and maintenance procedural work. This includes assembly and maintenance of drilling equipment for the oil and gas industry, equipment used in chemical processing plants, and maintenance and repair of commercial aircraft.

The aim of the Virtual Intelligent Task Assistant (VITA) study in NASA Human Exploration Research Analog (HERA) Campaign 7 is to determine the best methods for the display of procedural work to improve crew autonomy and satisfaction. Specifically, this study compares crew performance and satisfaction on rover assembly tasks similar to those used in HERA Campaign 6 but with procedures that vary the level of detail in text instruction and the visual aids, to improve support for flexible execution of actions. These differences are illustrated in existing VITA procedures for rover assembly and disassembly.

A new version of the VITA task assistant was developed for use in HERA Campaign 7. The new task assistant is an augmented reality (AR) application developed in Unity. This application runs on the augmented reality headset Microsoft’s HoloLens 2, instead of the HoloLens 1 used in HERA Campaign 6. The VITA augmented reality display was redesigned, including changing from gaze interaction used in HERA C6 to voice interaction for HERA C7. To address our aim of the best methods for display of procedure work, we investigate procedures that vary the visual aid and the level of detail in text. Specifically, we compare performance with figures displaying Photographs to 3D models with diagrams in figures. We also look at the level of detail in text instructions, comparing the procedure with and without more detailed text instructions.

The VITA study in HERA Campaign 7 had a total of 16 participants. Participants were selected by HERA to be astronaut-like. Participants were trained to use VITA prior to each HERA mission. Data collection includes situation awareness, workload, usability, and task timing at the end of each session. A survey was administered at the end of each session (called the end of condition survey) and after the final VITA session (called the end of mission survey). These surveys include questions about user satisfaction and perception of autonomy when using different procedure styles will be added to the end of the mission survey.

Since the annual report was submitted in March 2025, the analysis of data collected during HERA C7 was completed. Findings are summarized on human performance using the different visual aids during the VITA study in HERA C7. We also summarize findings of our investigation into procedure style and crew sense of work autonomy.

Visual Aids in AR procedures

Augmented reality interfaces afford the opportunity to situate virtual overlays relative to the real world. A common technique is to use image recognition software like Vuforia https://www.ptc.com/en/products/vuforia to locate objects in the world, then overlay virtual cues on the view of the object to highlight or annotate it. We found that such an approach did not work well for our rover assembly task. One issue is that the rover has many small parts that can be difficult to distinguish and track with software like Vuforia. Small components could also be occluded by the annotation. Another issue is that the rover's appearance changed with every assembly or disassembly action taken, making it challenging to capture and recognize all the possible appearances of the real object during the assembly or disassembly task. For such applications, we situated cues relative to a 3D model of the object. In HERA C7, we investigate the use of 3D models for small rover assembly, and compare performance with 3D models to performance with Photographs. This investigation of visual aids addresses our aims of the best methods for the display of procedural work as well as support for hands-free assembly and maintenance.

There is a significant reduction in workload when using 3D models for the first assembly session instead of Photographs. Workload is low (Median=3) and the same; however, for both conditions in the second assembly session. A Wilcoxon signed-rank test was done for median workload in the first session of both visual aids. Workload was higher for participants during the first rover assembly when using Photographs (Median=6) for a visual aid than for the 3D model (Median=3.5). This workload difference is significant.

This difference indicates that the 3D model may be easier to learn than the Photographs, when the task is unfamiliar to the user. Once the user practices with the visual aid, these workload differences disappear.

In addition to the effects of these visual aids on workload, some workload differences could be due to the task differences. Different but comparable procedures were used for the 3D and Photograph conditions.

In the NASA Johnson Space Center (JSC) Human Factors Engineering Laboratory (HFEL) study, we enhanced the 3D model to include a legend of parts and to show arrow overlays identifying which parts to assemble at each step. Participants rated the visual aids for both HERA C7 and the JSC HFEL lab study using a 5-point Likert scale (1-strongly disagree to 5-strongly agree). Overall, the mean and median ratings were higher for the enhanced 3D model used in the JSC HFEL study than for the original 3D model used in the HERA C7 study, indicating that participants viewed these changes to the 3D model as an improvement.

Procedure Style and Work Autonomy

During HERA Campaign 7, data were collected about the effects of procedure style on crew's sense of autonomy. The VITA study used two styles of procedure in HERA C7. The assembly procedures were highly structured, with sequences of detailed instructions (similar to ISS procedures) accompanied by figures or 3D models for every step. The disassembly procedures were much less structured, with limited details about how to disassemble the rover, similar to those used in HERA C6. Disassembly procedures were performed using both the VITA task assistant on a HoloLens 2 and a web browser on a Tablet. A work autonomy survey [Breaugh, 1985] was administered after every session using both of these procedure types. This survey asks the user to rate the statements about work method autonomy using a 5-point Likert scale (1- strongly disagree to 5-strongly agree).

The effect of Task Type (assembly vs disassembly) and Practice (Sessions 1, 2, 3, and 4) on this work method autonomy rating was analyzed using a 2x4 within-subjects ANOVA. The main effect for Task Type is statistically significant, indicating that the participants’ perceived work autonomy was significantly higher for the Disassembly Task than the Assembly Task.

These findings from the work autonomy survey data collected in HERA C7 substantiate the crew observations from HERA C6. Participants found that the less structured disassembly procedures gave them a significantly greater sense of work autonomy than the more structured assembly procedures.

Participants performing the Disassembly Task have recent device familiarity due to the rover assembly they performed a few days earlier. Research indicates that when users are familiar with either the device or the task using the device, they are less likely to seek out and follow procedural instructions [Eiriksdottir and Catrambone, 2011]. Thus, a Disassembly Task that follows an Assembly Task is an example of a task situation where detailed procedural instruction may not be needed (or heeded) by users.

These less structured procedures are not appropriate for all types of mission work, such as high-risk tasks or tasks requiring skilled performance. Less structured procedures can be effective; however, when the crew member has recent task experience, such as disassembly, not long after assembly seen in HERA C7. Thus, when appropriate to the task, less structured procedures can be used as a mitigation to increase crew's sense of autonomy for long-duration exploration missions.

II. Laboratory Study of VITA Support for Task Transitions

A laboratory study was performed investigating aid from the VITA task assistant for task transitions. The aim of this study is to determine the best practices to improve crew performance when using the VITA virtual task assistant to aid in task transitions, such as handling the interruption of a manual task by an automated task. In this study, the user is responsible for the manual assembly of the rover gripper. Concurrently, they are responsible for the supervision of an automated task (e.g., the automated startup of a habitat system). Supervisory tasks include monitoring the progress on the startup and intervening in the automated task when automation pauses.

The participant monitors progress on automated tasks using virtual notices displayed in the VITA task assistant. These notices are issued when task milestones are achieved, corresponding to changes in system states or environmental levels.

This study evaluated two conditions: with and without virtual notices from the automated task displayed in the task assistant. The condition With Virtual Notices displayed notices in the HoloLens. The condition Without Virtual Notices did not display notices in the HoloLens or in a Tablet. Instead, the participant monitored automated task progress and the need for intervention by periodically checking the procedure display on the laptop. User intervention with the automation was done for both conditions using PRIDE procedures displayed in a browser on a laptop. Task performance was compared with and without virtual notices in the VITA task assistant. This is a within-subjects study.

The VITA laboratory study had 16 participants. The study was conducted in the Human Factors Engineering Laboratory (HFEL) onsite at JSC using subjects from the Human Test Subject Facility (HTSF). Participants were trained prior to the study session. After training, each participant assembled the gripper, but without responsibility to supervise automation. Then a second gripper assembly was performed, during which the participant was responsible for supervising automation. During the second gripper assembly, participants performed supervisory tasks both with and without virtual notices from the automated task. Sessions were counterbalanced for these notification conditions. Since the annual report was submitted in March 2025, the VITA project has completed the analysis of data from the JSC HFEL lab study.

AR Notices to Supervise Automation

Findings are summarized comparing human performance with AR Notices compared to Tablet Monitoring when supervising automation while also performing a rover assembly.

The JSC HFEL lab study evaluated two conditions for supervision of an automated system concurrent with performing another manual task: AR Notices and Tablet Monitoring. All participants (n = 16) preferred monitoring automation using AR Notices in the HoloLens over watching the automation execute using Tablet Monitoring. Participants reported they felt that monitoring the automation was easier using AR notices (M = 5.94, SD = 0.68, Median = 6) than using Tablet Monitoring (M = 3.94, SD = 1.24, Median = 4). A Wilcoxon signed-rank test shows that AR Notices are significantly easier to use when monitoring automation than Tablet Monitoring.

We expected that the time to respond to an event in the automated system (called response time) would be shorter when using AR Notices than when using Tablet Monitoring. The response times were comparable for both conditions, however. We believe response time for Tablet Monitoring was shorter than expected because users were hyper-vigilant. Additionally, response time for AR Notices was longer than expected because users did not feel a sense of urgency to report immediately. As one participant put it, “I also felt less pressure with the HoloLens. I felt less urgency to log the event, whereas with the scrolling steps, I wanted to log the event right when I saw the step”.

Workload, however, was significantly lower for AR Notices than for Tablet Monitoring. These workload effects are discussed below.

A Wilcoxon signed-rank test was done for median workload of the automation supervision task. Workload for the automation supervision task was significantly less for AR Notices (Median=2) than for Tablet Monitoring.

When supervising the automation using AR notices, participants felt they could focus on the assembly task while responding to notices from the habitat automation in a timely manner. They attributed this to the auditory cue emitted when a new notice was received. Participants heavily relied on the auditory cue to shift their attention to the notice pane when a new notice was received. This reduced the effort needed to monitor the monitor automation. In noisy environments like space habitats, other non-visual modalities may be preferred to cue a new notice.

User-reported reliance on auditory cues to detect incoming notices is consistent with the findings of a pilot study conducted prior to the JSC HFEL Study. In that pilot study, five users monitored for new notices while performing an assembly task. Time to detect the notice was measured for two conditions – with and without an auditory tone when a new notice was received. This was a within-subject study. With no auditory cue, 4 of 5 participants took at least 90 seconds to recognize a new notice. With an auditory cue, all participants recognized a new notice within a few seconds.

When supervising the automation using Tablet Monitoring, most participants felt that the combination of the primary and secondary task was “doable,” but left them with little spare capacity with minor delays. Only one participant directly mentioned that they felt they had enough bandwidth for other tasks if needed.

References 1. Breaugh, J. A. The Measurement of Work Autonomy. Human Relations. Volume 38. Number 6. 1985. Pp. 5551-570. 2. Eiriksdottir, E. and R. Catrambone. Procedural Instructions, Principles, and Examples : How to Structure Instructions for Procedural Tasks to Enhance Performance, Learning, and Transfer. Human Factors. 2011.

NOTE: End date changed to 03/31/2025 per A. Beitman/JSC (Ed., 9/19/24)

NOTE: End date changed to 12/31/2024 per A. Beitman/JSC (Ed., 1/20/23)

NOTE: End date changed to 5/14/2023 per S. Huppman/HRP and NSSC information (Ed., 3/3/2020)

Future deep space missions will present new challenges for crew, and increased risks to human performance due to the stress, fatigue, radiation exposure, and isolation that characterizes these missions. In addition, crew will no longer be able to depend on timely support from Mission Control due to distance from the Earth, but will have to work autonomously, while maintaining high performance. Mission Controllers may not be available to answer questions, check system status, assist with procedures, monitor for errors, or troubleshoot problems. Greater crew autonomy will increase dependence on automated systems, and design of these automated systems must be driven by sound human-system integration standards and guidelines in order to ensure mission success. Historically, crew have had very limited dependence on automated systems, thus crew will be faced with a new way of working that may put situation awareness (SA) at risk. We must develop methods for promoting good situation awareness in the automated systems that will most certainly be part of future deep space vehicles and habitats.

Procedure automation is a promising technology for reducing crew workload. We define procedure automation as technology that automates the selection or execution of procedural tasks. Structuring the work of automation according to human procedures should improve the transparency of automation actions. This approach provides a means for establishing common ground about ongoing tasks to improve operator understanding of automation behavior.

New technologies such as adaptive, multimodal, augmented reality displays can offer the benefits of information presentation tailored to meet the needs of each crewmember, taking into consideration the current state of that crewmember (e.g., sleep-deprived, high workload), as well as the current state of his/her environment and ongoing activities (e.g., emergency situation, time-critical operations).

We combine technology for procedure automation with technology for augmented reality multi-modal (ARMM) user interfaces using Microsoft Hololens head-mounted display to provide a virtual task assistant to assist crew in performing procedural work. This virtual task assistant identifies procedures to perform, aids performing actions in crew procedures, and summarizes actions taken by the human-automation team to assist crew in preparing for tasks and taking over tasks from other team members.

Four studies were conducted to evaluate the effects of a virtual task assistant combining procedure automation with augmented reality multi-modal (AARM) user interfaces on human task performance. These studies will achieve the following aims:

Aim. Determine best methods to improve situation awareness and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance.

Aim. Determine best practices to improve crew performance when using virtual task assistant to aid in task transitions (task handovers, automation interruption).

Aim. Determine best methods for display of procedural work to improve crew autonomy and satisfaction

The proposed work addresses a number of gaps in the Human Research Program Human Factors and Behavioral Performance risks. This project will provide guidelines for designing effective human-automation systems and evaluate human-automation performance for exemplar procedure automation systems. This project also will provide guidance for the application of multi-modal and adaptive displays and control to Human-Computer Interaction (HCI) design for long duration operations.

The Virtual Intelligent Task Assistant (VITA) project is leveraging augmented reality platforms and toolkits for augmented reality apps to develop a virtual task assistant that can be used to assist users with procedural task work on the job. Our technical approach is innovative in that new procedural tasks can be supported without custom software development. Our experimental research is distinguished by investigating effective task assistance for maintenance or assembly tasks where hands-free operation of task assistance is beneficial. For the first study we investigated best techniques for using augmented reality task assistance when assembling small rovers that are held in the hands during assembly. Subsequent studies investigate other aspects of using augmented reality for assembly tasks, including use of 3D images in procedures and virtual notification for task milestones and transitions.

This technology and associated research findings have potential benefit to NASA for the assembly, maintenance, and repair of aircraft, spacecraft, habitats, and robotics. This technology and associated research findings also have broader potential benefit for any organization performing assembly and maintenance procedural work. This includes assembly and maintenance of drilling equipment for the oil and gas industry, equipment used in chemical processing plants, and maintenance and repair of commercial aircraft.

The aim of the Virtual Intelligent Task Assistant (VITA) study in NASA Human Exploration Research Analog (HERA) Campaign 7 (C7) is to determine best methods for the display of procedural work to improve crew autonomy and satisfaction. Specifically, this study compares crew performance and satisfaction on rover assembly tasks similar to those used in HERA C6 but with procedures that vary the level of detail in text instruction and the visual aids, to improve support for flexible execution of actions. These differences are illustrated in existing VITA procedures for rover assembly and disassembly.

A new version of the VITA task assistant was developed for use in HERA C7. The new task assistant is an augmented reality application developed in Unity. This application runs on the augmented reality headset Microsoft’s HoloLens 2, instead of the HoloLens 1 used in HERA C6. The VITA augmented reality display was redesigned, including changing from gaze interaction used in HERA C6 to voice interaction for HERA C7.

To address our aim of best methods for the display of procedure work, we investigate procedures that vary the visual aid and the level of detail in text. Specifically, we compare performance with figures displaying Photographs to 3D models with diagrams in figures. We also look at the level of detail in text instructions, comparing procedures with and without more detailed text instructions.

The VITA study in HERA C7 had a total of 16 participants. Participants were selected by HERA to be astronaut-like. Participants were trained to use VITA prior to each HERA mission. Data collection includes situation awareness, workload, usability, and task timing at the end of each session. A survey was administered at the end of each session (called the end of condition survey) and after the final VITA session (called the end of mission survey). These surveys include questions about user satisfaction and perception of autonomy when using different procedure styles will be added to the end of mission survey.

During the reporting period, the VITA project met all HERA C7 delivery milestones. This included training the crews of C7M2-4, and supporting VITA sessions as needed during C7M2-4. It also includes analysis of data collected during HERA C7, which is ongoing at the time of this report.

Preliminary findings are summarized on human performance using the different visual aids and level of detail during the VITA study in HERA C7. The Bedford workload scale was administered at the end of each assembly session. For the first time an assembly task is performed, the Photograph condition has a median workload of 6, while the 3D condition has a much lower median workload of 3.5. When the procedure is repeated; however, the median workload is the same at 3.0 for both visual aids. Some workload differences may be due to task differences.

System usability is measured using the System Usability Scale (SUS). SUS was assessed at the end of each assembly session. Mean usability ratings for the Photograph and the 3D model conditions are comparable for both the first and second repetition of the procedure. Means values are above 70 for all conditions.

Situation awareness (SA) was assessed by the participant at two points in the assembly task, one halfway through the task (SA1) and one just after the task was finished (SA2). The participants rated SA from a low of 1 to a high of 7. For the interim rating SA1, the median SA for the Photograph condition is lower at 5 while the median SA for the 3D model is 6. The final rating SA2; however, is similar for both visual aids, with a median value of 5-5.5 for the first repetition, and a median value of 6 for the second repetition. All SA ratings are moderately high.

Participants were asked which visual aid they preferred. Ten of 16 crew participants preferred using the 3D model. A key reason given was that the 3D model provides more information more quickly. Some participants also felt the 3D model was easier to use. Additionally, the 3D model shows the relationship between the parts to be assembled and makes this relationship available at all steps of the assembly. Four of 16 participants preferred figures using Photographs. These participants felt photographs were straightforward to use and captured more detail than the 3D model. Finally, two of 16 participants had no preference but mentioned that they liked having both options for different situations.

Participants also were asked which level of text detail they preferred: Expanded text or Collapsed text. Ten of 16 participants preferred the Expanded text because it gave them additional information complementary to the visual aids or not available in the visual aids. Some also felt the text instruction provided useful context in interpreting the visual aids. Two participants preferred having less text detail, feeling the visual aids were sufficient to perform the task. Three participants felt the text instruction was useful sometimes, such as the first time they built a rover configuration, or if an instruction was particularly complex.

II. Laboratory Study of VITA Support for Task Transitions

A laboratory study was performed investigating aid from the VITA task assistant for task transitions. This study aims to determine the best practices to improve crew performance when using the VITA virtual task assistant to aid in task transitions, such as handling the interruption of a manual task by an automated task. In this study, the user is responsible for the manual assembly of the rover gripper. Concurrently they are responsible for supervision of an automated task (e.g., the automated startup of a habitat system). Supervisory tasks include monitoring the progress on the startup, and intervening in the automated task when automation pauses.

The participant monitors progress on automated tasks using virtual notices displayed in the VITA task assistant. These notices are issued when task milestones are achieved, corresponding to changes in system states or environmental levels.

This study evaluated two conditions – with and without virtual notices from the automated task displayed in the task assistant. The condition With Virtual Notices displayed notices in the HoloLens. The condition Without Virtual Notices did not display notices in the HoloLens or in a Tablet. Instead, the participant monitored automated task progress and the need for intervention by periodically checking the procedure display on the laptop. User intervention with the automation was done for both conditions using PRIDE procedures displayed in a browser on a laptop. Task performance was compared with and without virtual notices in the VITA task assistant. This is a within-subjects study.

The VITA laboratory study had 16 participants. The study was conducted in the Human Factors Engineering Laboratory (HFEL) onsite at NASA Johnson Space Center (JSC) using subjects from the Human Test Subject Facility (HTSF). Participants were trained prior to the study session. After training, each participant assembled the gripper, but without responsibility to supervise automation. Then a second gripper assembly was performed, during which the participant was responsible for supervising automation. During the second gripper assembly, participants performed supervisory tasks both with and without virtual notices from the automated task. Sessions were counterbalanced for these notification conditions.

During the reporting period, the VITA project completed data collection for the lab study and began analysis of these data, which is ongoing at the time of this report.

Preliminary findings are summarized by comparing human performance on the first gripper assembly using the different visual aids during the VITA study in HERA C7 and the VITA lab study. This comparison was performed to gain insight into differences in the visual aids used for each study. The HERA C7 visual aids combine a 3D model of the gripper with annotated 2D images of the model. The VITA lab study replaces the 2D images used in HERA by overlaying the annotations on the 3D model (“enhanced 3D model”).

Situation Awareness was measured during the assembly task. Two SA ratings, from a low of 1 to a high of 7, were assessed by the participant, one halfway through the task (SA1) and one just after the task was finished (SA2). Both studies have a mean value at SA1 just below 6 and a median value of 6 for the HERA study and 6.5 for the lab study. The ratings at SA2; however, show larger differences. Both the mean and median values using the modified visual aids for the Lab Study are essentially the same at both SA1 and SA2. However, the SA2 ratings for the HERA C7 study have both a mean and median value of 5. The mean value is one point lower than SA1 and the median value is 1.5 points lower than SA1. We are investigating the possible causes of this loss.

Situation awareness was also assessed using the Modified Situation Awareness Rating Technique (SART) to measure SA. The SART technique assesses situation awareness in three areas: task demand, crew resources, and crew understanding.

Task Demand: Both the mean and median Task Demand ratings are over a point lower for the VITA lab study, indicating a less demanding task. These differences in the Task Demand scale may be affected by the design changes in the visual aids. They also may be influenced by the longer session and more comprehensive build of the HERA C7 session.

Crew Resources: The mean and median ratings for the Supply of Crew Resources are similar for both studies, possibly indicating that the design changes had no notable effect on the availability of crew resources.

Crew Understanding: The mean rating for Crew Understanding is over half a point higher, and the median rating is over one point higher for the lab study than for the HERA C7 study. This difference may indicate that the modified 3D model provided participants with an advantage relative to the original 3D model with 2D diagrams. This difference also may be influenced by the need for additional knowledge in the HERA C7 study to build the rover base as well as the gripper.

The Bedford workload scale was used to assess workload at the end of each assembly task. Participants in HERA C7 reported a median workload of 4, slightly higher than the median workload of 3 reported in the Lab Study. Differences in the visual aids between the two studies do not appear to notably impact subjective workload.

The Usability Scale (SUS) is used to assess usability at the end of each assembly task. Participants in the VITA Lab Study rated the usability of VITA with the modified 3D model near or just above 80. Participants in the HERA C7 study rated the usability of VITA with the original 3D model with 2D diagrams just below 80. Thus, the changes in the 3D model do not appear to affect usability in any notable way.

NOTE: End date changed to 12/31/2024 per A. Beitman/JSC (Ed., 1/20/23)

NOTE: End date changed to 5/14/2023 per S. Huppman/HRP and NSSC information (Ed., 3/3/2020)

Future deep space missions will present new challenges for crew, and increased risks to human performance due to the stress, fatigue, radiation exposure, and isolation that characterizes these missions. In addition, crew will no longer be able to depend on timely support from Mission Control due to distance from the Earth, but will have to work autonomously, while maintaining high performance. Mission Controllers may not be available to answer questions, check system status, assist with procedures, monitor for errors, or troubleshoot problems. Greater crew autonomy will increase dependence on automated systems, and design of these automated systems must be driven by sound human-system integration standards and guidelines in order to ensure mission success. Historically, crew have had very limited dependence on automated systems, thus crew will be faced with a new way of working that may put situation awareness (SA) at risk. We must develop methods for promoting good situation awareness in the automated systems that will most certainly be part of future deep space vehicles and habitats.

Procedure automation is a promising technology for reducing crew workload. We define procedure automation as technology that automates the selection or execution of procedural tasks. Structuring the work of automation according to human procedures should improve the transparency of automation actions. This approach provides a means for establishing common ground about ongoing tasks to improve operator understanding of automation behavior.

New technologies such as adaptive, multimodal, augmented reality displays can offer the benefits of information presentation tailored to meet the needs of each crewmember, taking into consideration the current state of that crewmember (e.g., sleep-deprived, high workload), as well as the current state of his/her environment and ongoing activities (e.g., emergency situation, time-critical operations).

We propose to combine technology for procedure automation with technology for augmented reality multi-modal (ARMM) user interfaces using Microsoft Hololens head-mounted display to provide a virtual task assistant to assist crew in performing procedural work. This virtual task assistant will be capable of identifying which procedures should be performed, performing actions in crew procedures, and summarizing actions taken by the human-automation team to assist crew in preparing for tasks and taking over tasks from other team members.

Four studies are planned to evaluate the effects of a virtual task assistant combining procedure automation with augmented reality multi-modal (AARM) user interfaces on human task performance. These studies will achieve the following aims:

Aim. Determine best methods to improve situation awareness and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance.

Aim. Determine best practices to improve crew performance when using virtual task assistant to aid in task transitions (task handovers, automation interruption).

Aim. Determine best methods for display of procedural work to improve crew autonomy and satisfaction

The proposed work addresses a number of gaps in the Human Research Program Human Factors and Behavioral Performance risks. This project will provide guidelines for designing effective human-automation systems and evaluate human-automation performance for exemplar procedure automation systems. This project also will provide guidance for the application of multi-modal and adaptive displays and control to Human-Computer Interaction (HCI) design for long duration operations.

The Virtual Intelligent Task Assistant (VITA) project is leveraging augmented reality platforms and toolkits for augmented reality apps to develop a virtual task assistant that can be used to assist users with procedural task work on the job. Our technical approach is innovative in that new procedural tasks can be supported without custom software development. Our experimental research is distinguished by investigating effective task assistance for maintenance or assembly tasks where hands-free operation of task assistance is beneficial. For the first study we investigated best techniques for using augmented reality task assistance when assembling small rovers that are held in the hands during assembly. Subsequent studies investigate other aspects of using augmented reality for assembly tasks, including use of 3D images in procedures and virtual notification for task milestones and transitions.

This technology and associated research findings have potential benefit to NASA for the assembly, maintenance, and repair of aircraft, spacecraft, habitats, and robotics. This technology and associated research findings also have broader potential benefit for any organization performing assembly and maintenance procedural work. This includes assembly and maintenance of drilling equipment for the oil and gas industry, equipment used in chemical processing plants, and maintenance and repair of commercial aircraft.

The Virtual Intelligent Task Assistant (VITA) project conducted a study in NASA Human Exploration Research Analog (HERA) Campaign 6 (C6) to determine the best methods to improve situation awareness (SA) and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance and assembly. This study attempts to answer the question posed in the proposal:

“Can the virtual task assistant stand-in for Mission Control Center (MCC) and help crew prepare for and perform manual tasks that are not done frequently, such as equipment maintenance and assembly?”

The VITA study conducted in HERA Campaign 6 started in September 2021 and ended in March 2023. We completed data collection for HERA C6 during the reporting period. Prior to each mission, we trained the crew on how to interact with the VITA. After each mission, we debriefed each crew member about their experience using VITA during the mission. Data collection and analysis for the VITA study in HERA C6 was performed during the reporting period. Lori Chappell, a NASA biostatistician, performed a statistical analysis of the VITA data collected during HERA Campaign 6.

The findings from all four missions, with 16 participants, are reported in the VITA annual report and in a paper to be presented at International Conference on Environmental Systems (ICES) 2024 (Schreckenghost et al., 2024). These findings are summarized below.

The VITA study investigates the use of techniques to aid crewmembers with procedural tasks when MCC cannot assist. A key aspect of that assistance is to prompt user actions during procedural tasks when MCC is not available in real-time. Three approaches were considered – HoloLens, Tablet, and Team. This study demonstrates that on average a single crewmember using either the Tablet or HoloLens system can complete assembly tasks with fewer total crew hours than required when a second crew member replaces MCC support, as needed for the Team condition.

This study is unique from earlier studies of augmented reality for procedural work displays (Furuya et al., 2018; Markov-Vetter et al., 2013). We use a modified Subjective Awareness Rating Technique (SART) (Taylor, 1990) for collecting SA data during procedure execution. We found only small differences in SA across all conditions, with one exception. Task demand in SART for HoloLens was on average one point higher than the other conditions. We believe this difference is due to increased demand to resolve the technology reliability issues described below.

Not all the expected benefits of hands-free operation were observed with the HoloLens, however. Rover assembly using the HoloLens was expected to take less schedule time and to reduce workload over other conditions. Instead, workload and schedule time were higher for the HoloLens than for both Tablet and Team. A key contributor to this increased workload and schedule time was some reliability and usability issues with the HoloLens system, as evidenced by the lower usability assessment values for this system. Another contributor to reduced usability and increased workload for some crew members was insufficient practice with the HoloLens.

We addressed these performance issues in the VITA study in HERA Campaign 7. Modifications include updating from the HoloLens 1 to the HoloLens 2 virtual reality headset, investigating voice interaction with VITA, and increasing the time users have to familiarize themselves with the HoloLens system.

II. VITA Study in HERA Campaign 7

The aim of the VITA study in HERA Campaign 7 (C7) is to determine the best methods for the display of procedural work to improve crew autonomy and satisfaction. Specifically, this study compares crew performance and satisfaction on rover assembly tasks similar to those used in HERA Campaign 6 but with procedures that vary the level of detail in text instruction and the style of figures, to improve support for flexible execution of actions. These differences are illustrated in existing VITA procedures for rover assembly and disassembly.

The VITA task assistant from HERA C6 was modified for use in HERA C7. A number of changes were made to the VITA task assistant software. The VITA task assistant now runs on the HoloLens 2, instead of the HoloLens 1 used in Campaign 6. The VITA augmented reality display was redesigned, including changing from gaze interaction used in HERA C6 to voice interaction for HERA C7.

To address our aim of best methods for the display of procedure work, we investigate procedures that vary the style of figure and the level of detail in the text. Specifically, we look at the style of the figure, comparing performance with 2D photos to 3D figures with diagrams. We also look at the level of detail in text instructions, comparing procedures with and without more detailed text instructions.

One of the findings from HERA C6 was the need to improve crew familiarity with augmented reality technology. For HERA C7 the time using VITA has been increased by adding more practice with the system. Each crew member uses the VITA AR system with two training procedures during pre-mission training. Then each crew member uses the VITA AR system for six of eight sessions during the mission. For HERA C7 all assembly procedures are performed twice, while for HERA C6 procedures were never repeated. The time between repetitions is counterbalanced between 1 week and 3 weeks.

The VITA study in HERA Campaign 7 will have a total of 16 participants. Participants are selected by HERA to be astronaut-like. Participants are trained prior to HERA and sessions are counterbalanced. Data collection includes situation awareness, workload, usability, and task timing at the end of each session. A survey is administered at the end of each session (called the end of condition survey) and after the final VITA session (called the end of mission survey). These surveys will include questions about user satisfaction and perception of autonomy when using different procedure styles will be added to the end-of-mission survey.

During the reporting period, the VITA project met all HERA C7 delivery milestones. This includes delivering modified rover kits, finishing software upgrades to the VITA task assistant, training the crew of C7M1, and supporting VITA sessions as needed during C7M1.

III. Laboratory Study of VITA Support for Task Transitions

A laboratory study will be performed investigating aid from the VITA task assistant for task transitions. The aim of this study is to determine the best practices to improve crew performance when using the VITA virtual task assistant to aid in task transitions, such as handling the interruption of a manual task by an automated task. In this study, the user is responsible for manually assembling one of the rover configurations. Concurrently they are responsible for supervising an automated task (e.g., the automated startup of a habitat system). Supervisory tasks include monitoring the progress of the startup and intervening when automation pauses for user interventions.

The participant monitors progress on automated tasks using virtual notices displayed in the VITA task assistant. These notices are issued when task milestones are achieved, corresponding to changes in system states or environmental levels.

The VITA virtual notification software design groups notices into panes associated with key habitat systems. Notice panes are shown in the periphery of the field of view (FOV), but can be moved to a different location. Notice panes do not follow head motion.

Notices also are issued in the VITA task assistant when automation pauses, waiting for the supervisor to intervene. The participant intervenes when configuration information is needed or an automated task fails.

A request for user intervention is considered a high saliency notice. The user’s attention should be shifted to high saliency notices as soon as they are received. Each high saliency notice is displayed in a separate pane. The background of this pane is different than in the group panes. These notices follow head motion to remain in the user FOV. They must be acknowledged to be moved out of the FOV.

This study evaluates two conditions – with and without virtual notices from the automated task displayed in the task assistant. The condition With Virtual Notices will display notices in the HoloLens as described above. The condition Without Virtual Notices will not display notices at all, neither in the HoloLens nor in a Tablet. Instead, the participant will monitor automated task progress and the need for intervention by periodically checking the procedure display on a laptop. Intervention into the automation will be done for both conditions using PRIDE procedures displayed in a browser on a laptop. Task performance will be compared with and without virtual notices in the VITA task assistant. This is a within-subjects study.

The augmented reality interface used for the VITA task assistant in this study incorporates the changes made for HERA Campaign 7, such as running as an app on the HoloLens 2. Users interact with procedural task assistance using voice commanding and gestures. The VITA task assistant has been modified to include virtual notification, as described earlier.

The VITA laboratory study will target up to 16 participants. The study will be conducted in the Human Factors Engineering Laboratory (HFEL) onsite at NASA Johnson Space Center (JSC) using subjects from the Human Test Subject Facility (HTSF). Participants will be trained prior to the study session and sessions will be counterbalanced.

Data collection will include situation awareness, workload, usability, and task timing at the end of each session. A survey will be administered after the VITA session to collect subjective ratings and comments.

References

1. Furuya, Hiroshi and Wang, Lui and Elvezio, Carmine and Feiner, Steven. “A Comparative Ground Study of Prototype Augmented Reality Task Guidance for International Space Station Stowage Operations.” 69th International Astronautical Congress. Bremen, Germany, October 1-5, 2018.

2. Markov-Vetter, Daniela and Millberg, Jessica and Staadt, Oliver. “Mobile Augmented Reality for Space Operation Procedures: A Generic Approach of Authoring and Guiding On-Board Payload Activities.” 64th International Astronautical Congress (IAC). Beijing, China, September 23-27, 2013.

3. Schreckenghost D, Holden K, Milam T, Robertson, IWT. “Performance of a Virtual Intelligent Task Assistant for NASA Astronauts Working Remotely during Deep Space Missions.” 53rd International Conference on Environmental Systems (ICES). Louisville, KY, July 21-15, 2024.

4. Taylor RM. Situational awareness rating technique (SART): The development of a tool for aircrew systems design. In: Situational Awareness in Aerospace Operations (AGARD-CP-478). Neuilly Sur Seine, France: NATO - AGARD. p. 3/1–3/17.

NOTE: End date changed to 5/14/2023 per S. Huppman/HRP and NSSC information (Ed., 3/3/2020)

Future deep space missions will present new challenges for crew, and increased risks to human performance due to the stress, fatigue, radiation exposure, and isolation that characterizes these missions. In addition, crew will no longer be able to depend on timely support from Mission Control due to distance from the Earth, but will have to work autonomously, while maintaining high performance. Mission Controllers may not be available to answer questions, check system status, assist with procedures, monitor for errors, or troubleshoot problems. Greater crew autonomy will increase dependence on automated systems, and design of these automated systems must be driven by sound human-system integration standards and guidelines in order to ensure mission success. Historically, crew have had very limited dependence on automated systems, thus crew will be faced with a new way of working that may put situation awareness (SA) at risk. We must develop methods for promoting good situation awareness in the automated systems that will most certainly be part of future deep space vehicles and habitats.

Procedure automation is a promising technology for reducing crew workload. We define procedure automation as technology that automates the selection or execution of procedural tasks. Structuring the work of automation according to human procedures should improve the transparency of automation actions. This approach provides a means for establishing common ground about ongoing tasks to improve operator understanding of automation behavior.

New technologies such as adaptive, multimodal, augmented reality displays can offer the benefits of information presentation tailored to meet the needs of each crewmember, taking into consideration the current state of that crewmember (e.g., sleep-deprived, high workload), as well as the current state of his/her environment and ongoing activities (e.g., emergency situation, time-critical operations).

We propose to combine technology for procedure automation with technology for augmented reality multi-modal (ARMM) user interfaces using Microsoft Hololens head-mounted display to provide a virtual task assistant to assist crew in performing procedural work. This virtual task assistant will be capable of identifying which procedures should be performed, performing actions in crew procedures, and summarizing actions taken by the human-automation team to assist crew in preparing for tasks and taking over tasks from other team members.

Four studies are planned to evaluate the effects of a virtual task assistant combining procedure automation with augmented reality multi-modal (AARM) user interfaces on human task performance. These studies will achieve the following aims:

Aim 1. Determine best methods to improve situation awareness and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance.

Aim 2. Determine best methods to improve situation awareness and reduce workload when a virtual task assistant is used to handover maintenance tasks between users.

Aim 3. Determine best methods to improve situation awareness and reduce workload when using a virtual task assistant to help manage concurrent manual and automated tasks.

Aim 4. Determine best methods for display of procedural work to improve crew autonomy and satisfaction

The proposed work addresses a number of gaps in the Human Research Program Human Factors and Behavioral Performance risks. This project will provide guidelines for designing effective human-automation systems and evaluate human-automation performance for exemplar procedure automation systems. This project also will provide guidance for the application of multi-modal and adaptive displays and control to Human-Computer Interaction (HCI) design for long duration operations.

The Virtual Intelligent Task Assistant (VITA) project is leveraging augmented reality platforms and new WebXR standards to develop a virtual task assistant that can be used to assist users with procedural task work on the job. Our technical approach is innovative in that new procedural tasks can be supported without custom software development. Our experimental research is distinguished by investigating effective task assistance for maintenance or assembly tasks where hands-free operation of task assistance is beneficial. For the first year we are investigating best techniques for using augmented reality task assistance when assembling small devices that are held in the hands during assembly.

This technology and associated research findings have potential benefit to NASA for the assembly, maintenance, and repair of aircraft, spacecraft, habitats, and robotics. This technology and associated research findings also have broader potential benefit for any organization performing assembly and maintenance procedural work. This includes assembly and maintenance of drilling equipment for the oil and gas industry, equipment used in chemical processing plants, and maintenance and repair of commercial aircraft.

The Virtual Intelligent Task Assistant (VITA) study to be conducted in HERA Campaign 6 started on September 2021. At the time this report was submitted, the VITA sessions for three missions in the NASA Human Exploration Research Analog (HERA) Campaign 6 (C6) were completed, and the fourth mission was in progress. We worked with our HERA Experiment Support Scientists (ESSs) Michael Merta and Reinhold Polvilaitis to prepare for each mission. Prior to each mission, we train the crew on how to interact with the VITA. After each mission, we debrief each crew member about their experience using VITA during the mission. Data collection and preliminary data analysis for the VITA study in HERA C6 were performed on data from the first three missions during the reporting period as well. We met with Millenia Young, the NASA biostatistician working on VITA, to define statistical analysis that will be performed once data from C6 Mission 4 is available.

We report preliminary results from 12 participants in HERA C6 Missions 1-3 in the remainder of this section.

Situation Awareness (SA)

Two types of situation awareness data are collected, situation awareness during performance of the task and situation awareness after completion of the task. We report preliminary descriptive statistics for each.

Situation Awareness during the Task

Two assessments are collected for each session, one mid-way through the task (interim) and one near the end of the task (final). Each assessment consists of the user indicating their situation awareness as a value ranging from 1 (lowest SA) to 7 (highest SA). Users are given the following definition of situation awareness to help make this assessment:

Situation awareness is the perception, understanding, and anticipation of factors impacting the effective completion of the tasks. • Are you getting the information that you need to do the task? • Do you understand what you are doing? • Do you have a good feel for what is coming next?

The descriptive statistics for situation awareness during the task by condition for three HERA C6 missions indicate the Team Tablet condition received the highest SA ratings (M = 6.14, 6.33), followed closely by the Solo Tablet (M = 6.04) while HoloLens was rated lower (M = 5.04).

Situation Awareness after the Task

The second type of SA data was collected using a questionnaire administered at the end of the task. This questionnaire uses the Situation Awareness Rating Technique (SART) to measure SA. The SART questionnaire has 10 questions grouped into the following categories:

• Task Demand. The amount of attention needed to perform the task and the task complexity and dynamics that affects this. A lower number indicates a less demanding task. • Supply of Crew Resource. Crew alertness, spare mental capacity, and the ability to focus on the task. A higher number indicates more crew resource is available for the task. • Availability of Information for Crew Understanding. How much information is provided to the crew, how easy it is to access this information, and the crew’s familiarity with the tasks. A higher number indicates a better task understanding.

The descriptive statistics for situation awareness after the task by condition for three HERA C6 missions indicate the task was considered the least demanding for the Team Reader (M=3.14) while other conditions were slightly more demanding. The crew resources available were slightly less when using Tablet (M=4.68) and slightly more for Team (M=5.02, 5.11). Crew resources for HoloLens condition were between Tablet and Team conditions. Crew information for understanding was the highest for the Team Reader (M=5.39) and the lowest for HoloLens (M=4.78).

Workload

The Bedford workload scale is used to assess the workload for each condition. Workload varies from 1 (least workload) to 10 (highest workload). Workload is measured at the end of each assembly session (condition). The descriptive statistics for workload by condition for three HERA C6 missions suggest that workload for HoloLens (M=5.58) is higher than other conditions, all of which have a mean workload of less than 3. A plot of the mean workload during missions 1-3 for each rover assembly procedure indicates the mean workload for HoloLens varies between a mean of 3-7 for the different rover assembly procedures.

Usability

The System Usability Scale (SUS) is used to assess usability of the different conditions. The SUS scale is rated from 1 (“Strongly Disagree”) to 5 (“Strongly Agree.”) Values are then scaled to be between 0 and 100. Usability is measured at the end of each rover assembly session.

For Campaign 6 Missions 1-3, preliminary descriptive statistics indicate the Team condition (M=80.42, 84.55) was considered the most usable, closely followed by the Tablet condition (M=77.29). The HoloLens condition (M=48.96) was considered less usable than other techniques.

Discussion of Preliminary Findings

Descriptive statistics for mean workload and usability for Missions 1-3 indicate the VITA augmented reality interface on the HoloLens 1 requires more workload and is less usable than either the Team or Tablet conditions. There are some observations about this augmented reality interface that might have affected these findings.

First, system reliability may have influenced the users’ ratings of augmented reality. Occasionally the HoloLens 1 would stop working and require a system restart. This would usually happen after 50 minutes or more of use and is suspected to be due to overheating. Our protocol required users to take a 5-minute break after 50 minutes of use to give the HoloLens time to cool. Even with this protocol, typically, one user per mission experienced a system crash. This system restart potentially affected both the users’ perception of the technology as well as the workload and SA measures during the session.

Second, the augmented reality system usability ratings may be affected by a few observations. First, most crew are not familiar with augmented technology, meaning it requires some users more practice to master. Additionally, some crew experienced fatigue when using the augmented reality headset. Also, gaze control was reported as fatiguing to some users.

Finally, the augmented reality system performance was observed to become less responsive at times. This likely was due to a combination of occasional slowness in the VITA software and intermittent network slowness due to loading.

To address some of these issues and improve the use of augmented reality in later VITA studies, the VITA augmented reality interface is being moved to the HoloLens 2 [ https://www.microsoft.com/en-us/hololens/ ] running Unity [ https://unity.com/ ] on the HoloLens 2 headset.

In addition to these findings based on data that were collected at the end of each condition (session), a questionnaire was administered after the final VITA session (called the end of mission questionnaire). This questionnaire was intended to collect crew's subjective assessment of the different techniques after they had used all of them.

One question in this survey asked users to indicate whether they preferred figures or text instructions for rover assembly:

“For all methods, to what extent did you rely on / prefer text vs. figure instructions to successfully complete the procedure actions?”

Through Mission 3, most participants (N = 8) reported preferring figures to text instructions (N = 2), while only a few had no preference (N =2). Thus, in HERA Campaign 7, we will investigate a style of procedures that relies more on figures.

NOTE: End date changed to 5/14/2023 per S. Huppman/HRP and NSSC information (Ed., 3/3/2020)

Future deep space missions will present new challenges for crew, and increased risks to human performance due to the stress, fatigue, radiation exposure, and isolation that characterizes these missions. In addition, crew will no longer be able to depend on timely support from Mission Control due to distance from the Earth, but will have to work autonomously, while maintaining high performance. Mission Controllers may not be available to answer questions, check system status, assist with procedures, monitor for errors, or troubleshoot problems. Greater crew autonomy will increase dependence on automated systems, and design of these automated systems must be driven by sound human-system integration standards and guidelines in order to ensure mission success. Historically, crew have had very limited dependence on automated systems, thus crew will be faced with a new way of working that may put situation awareness (SA) at risk. We must develop methods for promoting good situation awareness in the automated systems that will most certainly be part of future deep space vehicles and habitats.

Procedure automation is a promising technology for reducing crew workload. We define procedure automation as technology that automates the selection or execution of procedural tasks. Structuring the work of automation according to human procedures should improve the transparency of automation actions. This approach provides a means for establishing common ground about ongoing tasks to improve operator understanding of automation behavior.

New technologies such as adaptive, multimodal, augmented reality displays can offer the benefits of information presentation tailored to meet the needs of each crewmember, taking into consideration the current state of that crewmember (e.g., sleep-deprived, high workload), as well as the current state of his/her environment and ongoing activities (e.g., emergency situation, time-critical operations).

We propose to combine technology for procedure automation with technology for augmented reality multi-modal (ARMM) user interfaces using Microsoft Hololens head-mounted display to provide a virtual task assistant to assist crew in performing procedural work. This virtual task assistant will be capable of identifying which procedures should be performed, performing actions in crew procedures, and summarizing actions taken by the human-automation team to assist crew in preparing for tasks and taking over tasks from other team members.

Four studies are planned to evaluate the effects of a virtual task assistant combining procedure automation with augmented reality multi-modal (AARM) user interfaces on human task performance. These studies will achieve the following aims:

Aim 1. Determine best methods to improve situation awareness and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance.

Aim 2. Determine best methods to improve situation awareness and reduce workload when a virtual task assistant is used to handover maintenance tasks between users.

Aim 3. Determine best methods to improve situation awareness and reduce workload when using a virtual task assistant to help manage concurrent manual and automated tasks.

The proposed work addresses a number of gaps in the Human Research Program Human Factors and Behavioral Performance risks. This project will provide guidelines for designing effective human-automation systems and evaluate human-automation performance for exemplar procedure automation systems. This project also will provide guidance for the application of multi-modal and adaptive displays and control to Human-Computer Interaction (HCI) design for long duration operations.

The Virtual Intelligent Task Assistant (VITA) project is leveraging augmented reality platforms and new WebXR standards to develop a virtual task assistant that can be used to assist users with procedural task work on the job. Our technical approach is innovative in that new procedural tasks can be supported without custom software development. Our experimental research is distinguished by investigating effective task assistance for maintenance or assembly tasks where hands-free operation of task assistance is beneficial. For the first year we are investigating best techniques for using augmented reality task assistance when assembling small devices that are held in the hands during assembly.

This technology and associated research findings have potential benefit to NASA for the assembly, maintenance, and repair of aircraft, spacecraft, habitats, and robotics. This technology and associated research findings also have broader potential benefit for any organization performing assembly and maintenance procedural work. This includes assembly and maintenance of drilling equipment for the oil and gas industry, equipment used in chemical processing plants, and maintenance and repair of commercial aircraft.

The second VITA study being conducted in HERA Campaign 6 started in September 2021. The HERA study has four participants per mission and there will be four missions in Campaign 6, for a total of 16 participants. At the current time, the VITA sessions for two missions in HERA C6 are complete. We worked with our HERA Experiment Support Scientist (ESS), Michael Merta, to prepare for each mission. Prior to each mission, we train the crew on how to interact with the VITA. After each mission, we debrief each crewmember about his/her experience using VITA during the mission. Data collection and analysis for the VITA study in HERA C6 was started in Year 3 as well. We report preliminary results from the Pilot study and HERA C6 Mission 1 (C6M1) below.

Findings on Gaze-activated Control

Multi-modal interaction when using the VITA augmented reality software in a HoloLens includes visual presentation of task cues, hand gestures, and gaze-activated control. To improve support for hands-free operation, we are investigating the use of gaze to interact with the VITA user interface, including advancing to the next instruction, returning to a prior instruction, and recording data.

The operator uses gaze to mark instructions done and advance to the next instruction. Gaze is also used to zoom closer to or away from the VITA display, and to rotate a 3D model of the rover.

Gaze-activated controls are investigated as a means to reduce workload during assembly tasks by enabling interaction with the VITA intelligent agent without moving the user’s hands away from the assembly task. Subjective feedback from subjects on gaze-activated controls indicates such controls can increase workload, if not properly designed. If button response is too sensitive to user gaze, buttons can be accidentally activated, causing the user additional work to “undo” unintended actions. If button response is not sensitive enough to user gaze, repeated gaze actions and extended gaze times can be required, making it difficult to activate these controls and frustrating the user.

In response to feedback from the pilot study, a number of design changes were made to the gaze-activated controls in VITA. In the initial VITA design, buttons used to navigate through task instructions were located below the textual cue, to be near the bottom of the virtual field of view and closer to the user’s line of sight during assembly. However, this location appeared to possibly contribute to accidental button activations as the eyes moved during assembly, so the navigation buttons were moved above the task cue to be further away from the user’s line of sight when assembling the rover. The time that a user must gaze at a control button to activate it (called dwell time) was also increased to 250 msec. Navigation buttons were modified to blink briefly when activated, to improve user awareness of button activation. Finally, accidental activation of critical controls (like marking a task done and moving to the next task) is prevented by using an “arm-and-fire” design that requires two button activations to take an action.

Even with these changes, subjective feedback from the first mission of HERA Campaign 6 indicates that users felt gaze-control was slow and not sensitive enough. The HoloLens 1 tracks head direction but does not track eye movement. This reduced the precision of gaze direction, which can make it harder to activate buttons.

Findings on Placement of Virtual Cues

The VITA user interface arranges virtual task cues and gaze-activated controls in a planar layout. By default, this plane is placed at eye-level when the head is raised and looking forward. The user can use hand gestures to adjust the placement of this plane relative to head position and focal length.

During the pilot study, users were trained how to adjust the placement of this plane. Some subjects found it difficult to make this adjustment. Initially, many users intuitively placed the plane near their hand position when assembling the rover. Eventually, most users moved the plane above their hands and to one side, to prevent accidental activation of gaze-controlled buttons. If the plane was placed too far away, however, users had to move their heads more to see task cues.

During HERA C6M1, placement of virtual cues continued to be challenging for users. Some crew mentioned that shifting the focal plane between virtual task cues and the rover can be tiring over time.

Preliminary findings indicate that additional study of the placement of virtual cues with respect to the focal plane of the task is merited. The user interface design should make it easy to adjust placement of virtual cues relative to the location of task components. The user interface design should also try to minimize shifts in focal length between the physical task and the virtual task cue, as frequent shifts can cause visual fatigue. Designs should be investigated that simplify aligning the focal length of the virtual cues with the focal length of the task components, even when virtual cues are not placed near the task components.

Results

The VITA study is investigating workload when using only gaze-activated control, which earlier studies do not address. Subjective response to gaze-activated control has been mixed, with some users preferring it while others suggest using gesture or voice control. The reliance on gaze for all interaction with VITA makes it more likely that users may experience some visual fatigue, which is substantiated by observations during both the pilot study and HERA C6M1.

Observations from the pilot study and HERA C6M1 indicate that procedure information may need to be organized differently than in the tablet display for more effective use in virtual space. Currently, figures are associated with specific instructions. Users can easily glance at an earlier figure, when using a tablet to view the procedure. When using VITA, however, access to prior figures requires navigating back one instruction at a time. A number of users observed that the effort to go back using VITA discouraged them from looking at figures that would have helped with the current instruction.

Some participants in the pilot study and in HERA C6M1 reported discomfort when using the HoloLens 1 continuously for over 50 minutes. These reports are consistent with a study of simulator sickness. Gaze control was reported as fatiguing to some users in the pilot study. One participant in HERA C6M1 reported that having more than one session with an augmented reality headset in a day made symptoms worse, even when using different headsets (HoloLens 1 in one session, HoloLens 2 in another session). We are investigating in the HERA C6 study whether users adapt to this with repeated use.

We submitted a paper entitled “Lessons on Developing an Augmented Reality Interface to a Virtual Intelligent Task Assistant” to the Human Factors and Ergonomics Society annual meeting to be held in Atlanta, GA, on October 10 - 14, 2022. This paper reports preliminary results from the VITA pilot study and the first two missions in HERA Campaign 6.

Future deep space missions will present new challenges for crew, and increased risks to human performance due to the stress, fatigue, radiation exposure, and isolation that characterizes these missions. In addition, crew will no longer be able to depend on timely support from Mission Control due to distance from the Earth, but will have to work autonomously, while maintaining high performance. Mission Controllers may not be available to answer questions, check system status, assist with procedures, monitor for errors, or troubleshoot problems. Greater crew autonomy will increase dependence on automated systems, and design of these automated systems must be driven by sound human-system integration standards and guidelines in order to ensure mission success. Historically, crew have had very limited dependence on automated systems, thus crew will be faced with a new way of working that may put situation awareness (SA) at risk. We must develop methods for promoting good situation awareness in the automated systems that will most certainly be part of future deep space vehicles and habitats.

Procedure automation is a promising technology for reducing crew workload. We define procedure automation as technology that automates the selection or execution of procedural tasks. Structuring the work of automation according to human procedures should improve the transparency of automation actions. This approach provides a means for establishing common ground about ongoing tasks to improve operator understanding of automation behavior.

New technologies such as adaptive, multimodal, augmented reality displays can offer the benefits of information presentation tailored to meet the needs of each crewmember, taking into consideration the current state of that crewmember (e.g., sleep-deprived, high workload), as well as the current state of his/her environment and ongoing activities (e.g., emergency situation, time-critical operations).

We propose to combine technology for procedure automation with technology for augmented reality multi-modal (ARMM) user interfaces using Microsoft Hololens head-mounted display to provide a virtual task assistant to assist crew in performing procedural work. This virtual task assistant will be capable of identifying which procedures should be performed, performing actions in crew procedures, and summarizing actions taken by the human-automation team to assist crew in preparing for tasks and taking over tasks from other team members.

Four studies are planned to evaluate the effects of a virtual task assistant combining procedure automation with augmented reality multi-modal (AARM) user interfaces on human task performance. These studies will achieve the following aims:

Aim 1. Determine best methods to improve situation awareness and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance.

Aim 2. Determine best methods to improve situation awareness and reduce workload when a virtual task assistant is used to handover maintenance tasks between users.

Aim 3. Determine best methods to improve situation awareness and reduce workload when using a virtual task assistant to help manage concurrent manual and automated tasks.

The proposed work addresses a number of gaps in the Human Research Program Human Factors and Behavioral Performance risks. This project will provide guidelines for designing effective human-automation systems and evaluate human-automation performance for exemplar procedure automation systems. This project also will provide guidance for the application of multi-modal and adaptive displays and control to Human-Computer Interaction (HCI) design for long duration operations.

The Virtual Intelligent Task Assistant (VITA) project is leveraging augmented reality platforms and new WebXR standards to develop a virtual task assistant that can be used to assist users with procedural task work on the job. Our technical approach is innovative in that new procedural tasks can be supported without custom software development. Our experimental research is distinguished by investigating effective task assistance for maintenance or assembly tasks where hands-free operation of task assistance is beneficial. For the first year we are investigating best techniques for using augmented reality task assistance when assembling small devices that are held in the hands during assembly.

This technology and associated research findings have potential benefit to NASA for the assembly, maintenance, and repair of aircraft, spacecraft, habitats, and robotics. This technology and associated research findings also have broader potential benefit for any organization performing assembly and maintenance procedural work. This includes assembly and maintenance of drilling equipment for the oil and gas industry, equipment used in chemical processing plants, and maintenance and repair of commercial aircraft.

Aim 1. Determine best methods to improve situation awareness and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance.

Aim 2. Determine best methods to improve situation awareness and reduce workload when a virtual task assistant is used to handover maintenance tasks between users

Aim 3. Determine best methods to improve situation awareness and reduce workload when using a virtual task assistant to help manage concurrent manual and automated tasks.

Research during the second year of this project addresses Aim 1. To achieve Aim 1, we defined and prepared for a study in the Human Exploration Research Analog (HERA). This study investigates the usability and effectiveness of the virtual task assistant to improve crew autonomy in the HERA Campaign 6. Effectiveness in increasing crew autonomy is indicated by the number and type of interactions with MCC (Mission Control Center) or other crew members made during this experiment. Usability is measured using the System Usability Scale (SUS). The VITA project is conducting two studies to determine the best methods to improve situation awareness and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance and assembly (Project Aim 1). These studies attempt to answer the question posed in the proposal:

“Can the virtual task assistant stand-in for MCC and help crew prepare for and perform manual tasks that are not done frequently, such as equipment maintenance and assembly?”

The HERA Campaign 6 experiment and pilot study were defined in the first year of the VITA project. We made progress in these studies in the second year as described below.

During the second year the VITA project met a number of milestones for HERA Campaign 6 that define our HERA experiment. The hardware and software to be used for the VITA experiment was delivered to HERA. And the functional testing of VITA in HERA facility was completed. We worked with our HERA Experiment Support Scientist (ESS) to accomplish these activities.

We finalized and tested a full set of electronic procedures for the rover assembly and disassembly tasks. We integrated these procedures with the VITA software. Multi-modal interaction when using the augmented reality software in a HoloLens includes visual presentation of information, hand gestures, and gaze tracking. To improve support for hands-free operation, we are investigating the use of gaze to interact with the VITA user interface, including advancing to the next instruction, recording data, and manipulating the 3D model.

Conducting research safely during the COVID-19 pandemic required modification of the Institutional Review Board (IRB) used for the VITA studies. Specifically, we modified the study procedures in July 2020 to include both participants and researchers wearing masks and gloves for the duration of the pilot session. We also renewed the VITA IRB in October 2020. This renewal included similar provision for conducting studies during the COVID-19 pandemic.

The VITA experiment as originally planned ran the VITA software on a laptop resident inside HERA. Restrictions going onsite Johnson Space Center (JSC) due to COVID-19 made it difficult to validate technology running on a laptop in HERA. During the reporting period we worked with HERA to modify our experiment protocol to improve access to software and procedures by running the VITA software on a cloud server instead of a laptop resident inside HERA. This approach has been implemented and tested by HERA and by the VITA research team.

Pilot sessions were originally planned to be conducted in the Human Factors Engineering Laboratory (HFEL) onsite at Johnson Space Center (JSC) using subjects from the Human Test Subject Facility (HTSF) at JSC. Restrictions going onsite Johnson Space Center (JSC) due to COVID-19 prevented us from conducting pilot sessions when originally planned.

To make progress on some pilot study objectives, we conducted walkthroughs at a team member’s home with family members and work colleagues. Safety protocols from IRB were followed. Although JSC is still under Phase 3 restrictions, the VITA project recently received a waiver to conduct pilot sessions onsite at JSC. We are currently contacting candidate participants through HTSF to resume pilot sessions. Preliminary findings to date from pilot sessions and walkthroughs include 1) more accurate estimate of session timing for the HERA study, 2) improved procedures for rover assembly and disassembly, and 3) techniques for more automated computation of task metrics.

The Principal Investigator met monthly with Dr. S. Robinson, Dr. B. Gore, and Principal Investigators for the Virtual NASA Specialized Center of Research (VNSCOR) for “Human Capabilities Assessments for Autonomous Missions” (HCAAM). These meetings improved communication among these projects and promoted coordination between projects.

A paper describing our approach for automatically computing task performance metrics for VITA was accepted for presentation at SpaceOps 2021 [1]. This approach combines actions logged in the electronic procedure database with tasks defined in the electronic procedure to compute performance metrics upon request.

The last quarter of Year 2 will focus on finishing the pilot study for HERA Campaign 6. This study evaluates experimental techniques for the HERA C6 study. The study to be conducted in HERA Campaign 6 should start in Year 3 (expected start in September 2021).

REFERENCE

1. Debra Schreckenghost, Tod Milam, David Kortenkamp, and Alize Nguyen. Near Realtime Computation of Task Performance using Electronic Procedures. SpaceOps 2021. May 2021.

Future deep space missions will present new challenges for crew, and increased risks to human performance due to the stress, fatigue, radiation exposure, and isolation that characterizes these missions. In addition, crew will no longer be able to depend on timely support from Mission Control due to distance from the Earth, but will have to work autonomously, while maintaining high performance. Mission Controllers may not be available to answer questions, check system status, assist with procedures, monitor for errors, or troubleshoot problems. Greater crew autonomy will increase dependence on automated systems, and design of these automated systems must be driven by sound human-system integration standards and guidelines in order to ensure mission success. Historically, crew have had very limited dependence on automated systems, thus crew will be faced with a new way of working that may put situation awareness (SA) at risk. We must develop methods for promoting good situation awareness in the automated systems that will most certainly be part of future deep space vehicles and habitats.

Procedure automation is a promising technology for reducing crew workload. We define procedure automation as technology that automates the selection or execution of procedural tasks. Structuring the work of automation according to human procedures should improve the transparency of automation actions. This approach provides a means for establishing common ground about ongoing tasks to improve operator understanding of automation behavior.

New technologies such as adaptive, multimodal, augmented reality displays can offer the benefits of information presentation tailored to meet the needs of each crewmember, taking into consideration the current state of that crewmember (e.g., sleep-deprived, high workload), as well as the current state of his/her environment and ongoing activities (e.g., emergency situation, time-critical operations).

We propose to combine technology for procedure automation with technology for augmented reality multi-modal (ARMM) user interfaces using Microsoft Hololens head-mounted display to provide a virtual task assistant to assist crew in performing procedural work. This virtual task assistant will be capable of identifying which procedures should be performed, performing actions in crew procedures, and summarizing actions taken by the human-automation team to assist crew in preparing for tasks and taking over tasks from other team members.

Four studies are planned to evaluate the effects of a virtual task assistant combining procedure automation with augmented reality multi-modal (AARM) user interfaces on human task performance. These studies will achieve the following aims:

Aim 1. Determine best methods to improve situation awareness and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance.

Aim 2. Determine best methods to improve situation awareness and reduce workload when a virtual task assistant is used to handover maintenance tasks between users.

Aim 3. Determine best methods to improve situation awareness and reduce workload when using a virtual task assistant to help manage concurrent manual and automated tasks.

The proposed work addresses a number of gaps in the Human Research Program Human Factors and Behavioral Performance risks. This project will provide guidelines for designing effective human-automation systems (Human and Automated/Robotic Interactions (HARI)-02) and evaluate human-automation performance for exemplar procedure automation systems (HARI-03). This project also will provide guidance for the application of multi-modal and adaptive displays and control to Human-Computer Interaction (HCI) design for long duration operations (HCI-04).

The VITA project is leveraging augmented reality platforms and new WebXR standards to develop a virtual task assistant that can be used to assist users with procedural task work on the job. Our technical approach is innovative in that new procedural tasks can be supported without custom software development. Our experimental research is distinguished by investigating effective task assistance for maintenance or assembly tasks where hands-free operation of task assistance is beneficial. For the first year we are investigating best techniques for using augmented reality task assistance when assembling small devices that are held in the hands during assembly.

This technology and associated research findings have potential benefit to NASA for the assembly, maintenance, and repair of aircraft, spacecraft, habitats, and robotics. This technology and associated research findings also have broader potential benefit for any organization performing assembly and maintenance procedural work. This includes assembly and maintenance of drilling equipment for the oil and gas industry, equipment used in chemical processing plants, and maintenance and repair of commercial aircraft.

Aim 1. Determine best methods to improve situation awareness and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance.

Aim 2. Determine best methods to improve situation awareness and reduce workload when a virtual task assistant is used to handover maintenance tasks between users

Aim 3. Determine best methods to improve situation awareness and reduce workload when using a virtual task assistant to help manage concurrent manual and automated tasks.

Research during the first year of this project addresses Aim 1. To achieve Aim 1, we will conduct a study in the Human Exploration Research Analog (HERA). This study will investigate the usability and effectiveness of the virtual task assistant to improve crew autonomy in the HERA Campaign 6. Effectiveness in increasing crew autonomy will be indicated by the number and type of interactions with Mission Control Center (MCC) or other crewmembers made during this experiment. Usability will be measured using the System Usability Scale (SUS). The VITA project is conducting two studies to determine the best methods to improve situation awareness and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance and assembly (Project Aim 1). These studies attempt to answer the question posed in the proposal:

“Can the virtual task assistant stand-in for MCC and help crew prepare for and perform manual tasks that are not done frequently, such as equipment maintenance and assembly?”

During the definition phase of the VITA project, the research team focused on the initialization of the VITA project including detailed coordination with the other teams in the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR), and with NASA’s Flight Analogs Program to ensure seamless integration for the HERA Campaign 6. NASA Johnson Space Center Institutional Review Board (JSC IRB) was submitted and approved for the VITA project. An integrated plan for using the HERA facility was developed with support from the HERA Experiment Support Scientist (ESS). The Science Requirements Document (SRD) for the VITA HERA experiment was created, reviewed, and signed. The VITA study timeline was developed with the HERA ESS.

The studies for VITA project aim 1 were defined and implementation of these studies was begun. The first study is a pilot laboratory study to determine techniques for use in the second study, to be conducted in HERA Campaign 6. Both studies will investigate situation awareness and workload when using VITA to help the user prepare for and perform a manual assembly task. Specifically, VITA will assist the human in assembling and disassembling a small rover. The rover can be configured with a gripper or with multiple alternative means of locomotion. The virtual task assistant will use augmented reality and multi-modal techniques to prompt and inform the user when performing assembly tasks on the rover. Participants will complete the rover assembly procedure task under three procedure completion conditions: 1) individual participants with electronic procedures on a tablet, 2) individual participants with VITA task assistant, and 3) a team of two participants with electronic procedures on a tablet. Participants will cycle across a unique set of rover procedures, following random assignment. SA and Workload measures while using the VITA task assistant will be compared with baseline performance using assembly procedures in typical electronic procedure displays for NASA (such as Orion or International Space Station (ISS) displays)) available on a portable tablet.

The technology for the VITA task assistant to be used in this experiment also was integrated during the definition phase. The HoloLens 1 was integrated with the PRIDE electronic procedure software extended for augmented reality (PRIDEAVR). This integrated technology was validated to produce contextual virtual-assistant directions to crewmember. The VITA task assistant user interface was designed and implemented. This user interface is implemented as a web user interface, which permits displaying it in either the HoloLens or on a tablet. During the definition phase, equipment also was procured for both the pilot study and the HERA Campaign 6 study.

At the time this report was submitted, the first VITA study was in progress. We do not yet have experimental findings to report from this study. The second study to be conducted in HERA Campaign 6 has been designed, technology has been integrated, and preparation for hardware and software delivery to HERA is in progress.

The last quarter of Year 1 will focus on preparation for the study to be conducted in HERA Campaign 6. This includes continuing the pilot study started in the definition phase to evaluate experimental techniques for the HERA C6 study. The study to be conducted in HERA Campaign 6 should start in Year 2 (expected start in August 2020).

Future deep space missions will present new challenges for crew, and increased risks to human performance due to the stress, fatigue, radiation exposure, and isolation that characterizes these missions. In addition, crew will no longer be able to depend on timely support from Mission Control due to distance from the Earth, but will have to work autonomously, while maintaining high performance. Mission Controllers may not be available to answer questions, check system status, assist with procedures, monitor for errors, or troubleshoot problems. Greater crew autonomy will increase dependence on automated systems, and design of these automated systems must be driven by sound human-system integration standards and guidelines in order to ensure mission success. Historically, crew have had very limited dependence on automated systems, thus crew will be faced with a new way of working that may put situation awareness (SA) at risk. We must develop methods for promoting good situation awareness in the automated systems that will most certainly be part of future deep space vehicles and habitats.

Procedure automation is a promising technology for reducing crew workload. We define procedure automation as technology that automates the selection or execution of procedural tasks. Structuring the work of automation according to human procedures should improve the transparency of automation actions. This approach provides a means for establishing common ground about ongoing tasks to improve operator understanding of automation behavior.

New technologies such as adaptive, multimodal, augmented reality displays can offer the benefits of information presentation tailored to meet the needs of each crewmember, taking into consideration the current state of that crewmember (e.g., sleep-deprived, high workload), as well as the current state of his/her environment and ongoing activities (e.g., emergency situation, time-critical operations).

We propose to combine technology for procedure automation with technology for augmented reality multi-modal (ARMM) user interfaces using Microsoft Hololens head-mounted display to provide a virtual task assistant to assist crew in performing procedural work. This virtual task assistant will be capable of identifying which procedures should be performed, performing actions in crew procedures, and summarizing actions taken by the human-automation team to assist crew in preparing for tasks and taking over tasks from other team members.

Four studies are planned to evaluate the effects of a virtual task assistant combining procedure automation with augmented reality multi-modal (AARM) user interfaces on human task performance. These studies will achieve the following aims:

Aim 1. Determine best methods to improve situation awareness and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance.

Aim 2. Determine best methods to improve situation awareness and reduce workload when a virtual task assistant is used to handover maintenance tasks between users.

Aim 3. Determine best methods to improve situation awareness and reduce workload when using a virtual task assistant to help manage concurrent manual and automated tasks.

The proposed work addresses a number of gaps in the Human Research Program Human Factors and Behavioral Performance risks. This project will provide guidelines for designing effective human-automation systems (Human and Automated/Robotic Interactions (HARI)-02) and evaluate human-automation performance for exemplar procedure automation systems (HARI-03). This project also will provide guidance for the application of multi-modal and adaptive displays and control to Human-Computer Interaction (HCI) design for long duration operations (HCI-04).