This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

Crew extravehicular activity (EVA) is limited on spaceflight missions. Multiple, small robotic spacecraft with varying levels of autonomy are needed to perform tasks that might have been completed by an astronaut (e.g., an exterior surface inspection or repair). Crews on long duration exploration missions (LDEM) will have less access to ground support during task operations. As a result, they will need to process more information and communicate with autonomous robots effectively to ensure tasks are progressing safely and on schedule.

The objective of these studies is to investigate the use of augmented reality (AR) multimodal interface displays and communication pathways for improving human-robot communication, situation awareness (SA), trust, and task performance. This will lead to developing guidelines for designing human-robot system interactions that enable operational performance for crews on spaceflight missions.

The specific aims are to:

1) Develop a simulation testbed for examining communication between human-robot teams.

2) Develop a hardware testbed for examining communication between human-robot teams.

3) Evaluate human SA, trust, and task performance within a short duration and long-duration ground-based study (simulation and/or hardware) through testing various interface communication modalities and information displays.

4) (Option) Perform additional studies for alternate parameters of interest that could be tested using the study testbeds. Additional parameters include timing and persistence of information, gesture command mapping, varying the levels of robot automation, evaluating precision enabled by each command mode.

Our first two studies examined an inspection task in a simulation environment, where a small robotic agent was external to the space station being inspected. Both studies support decreased detection accuracy using AR for this task and increase in detection accuracy during synchronous interactions. Factors influencing the decrease in performance of AR for this task include the reduced contrast and resolution of the presented image, as well as reliability of hand interactions. For tasks that require higher resolution and contrast, like inspection, computer screens are recommend over AR implementations for current technology. For tasks that require short periods of vigilance, synchronous actions between human-robot teams are recommended. However, the optimal range of times for synchronous actions should be further studied.

We then adapted this simulation environment to examine agent movement as a communication modality. Autonomous agent motion plans can convey system intent to the monitor supporting the decision-making process. In this study, participants viewed an egocentric view from the agent in a supervisory role to determine accuracy and appropriateness of the docking maneuver. Participants appropriately took over control, but the takeover location was dependent on path characteristics. Path characteristics that led to egocentric views with the dock in the primary field of view led to earlier correct takeovers. Larger deviations from a central point-to-point connection were less preferred. Based on these findings, when supervising a robotic agent with an egocentric view, path directionality and heading should support a central view of the target location and paths should minimize deviations from the shortest-line path between waypoints. Path planning methods should incorporate human-aware constraints to support path selection from available solutions.

In our final set of studies, we further examined how AR impacts sensorimotor and balance related tasks. In these studies, participants performed a task using AR cues versus physical cues. Participants were able to complete all assessment tasks successfully in AR and meaningful measures of performance were obtained from the embedded sensors. These data support that AR has the potential to be used to track performance, BUT the baseline thresholds of performance can differ from other assessment modalities. Based on these studies, we recommend that AR interfaces should: (1) feature wider targets (minimum 0.025 m) than required in physical interfaces, (2) use visual cues to support depth perception (e.g., proximity lighting), and (3) take care in selecting the interaction region for a given object. Improvements are needed in AR to support increased permissibility for hand gesture detection and performance baselines should be determined within AR for use as an assessment platform.

This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

Crew extravehicular activity (EVA) is limited on spaceflight missions. Multiple, small robotic spacecraft with varying levels of autonomy are needed to perform tasks that might have been completed by an astronaut (e.g., an exterior surface inspection or repair). Crews on long duration exploration missions (LDEM) will have less access to ground support during task operations. As a result, they will need to process more information and communicate with autonomous robots effectively to ensure tasks are progressing safely and on schedule.

The objective of these studies is to investigate the use of augmented reality (AR) multimodal interface displays and communication pathways for improving human-robot communication, situation awareness (SA), trust, and task performance. This will lead to developing guidelines for designing human-robot system interactions that enable operational performance for crews on spaceflight missions.

The specific aims are to:

1) Develop a simulation testbed for examining communication between human-robot teams.

2) Develop a hardware testbed for examining communication between human-robot teams.

3) Evaluate human SA, trust, and task performance within a short duration and long-duration ground-based study (simulation and/or hardware) through testing various interface communication modalities and information displays.

4) (Option) Perform additional studies for alternate parameters of interest that could be tested using the study testbeds. Additional parameters include timing and persistence of information, gesture command mapping, varying the levels of robot automation, evaluating precision enabled by each command mode.

During this reporting period, our team performed synergistic follow-on studies using the simulation space station environment developed in a previous reporting period. The first study examined how the motion plan trajectory shape influenced a decision to takeover control during a docking maneuver from an egocentric viewpoint. Here takeovers occurred when the human supervisor deemed that the agent was going to an incorrect docking location, or a collision could occur. The second study examined how viewpoint (egocentric vs. exocentric) influenced the decision to takeover control during a docking maneuver. We observed that both path trajectory and viewpoints influenced the decisions to takeover control. We also observed that there were effects on the decision to handover control (i.e., when the human supervisor determined the agent was performing correctly and they no longer needed to monitor performance). Findings from these studies will be summarized for the NASA HIDH.

During the reporting period, our team also completed analysis from studies to support gaps in the NASA Human Integration Design Handbook (HIDH) for AR interfaces. We compared performance on sensorimotor and neurovestibular assessment tasks using physical objects and within AR. The sensorimotor task was a multi-directional Fitts’ Law target acquisition task, while the neurovestibular tasks included three clinical balance assessments (Four Square Step test (FSST), the Star Excursion, tandem walking/TW), and three operational tasks (capsule ingress and egress, geology sampling, and obstacle avoidance). For the sensorimotor task, the touchscreen modality yielded improved performance over AR as measured by accuracy, precision, error rates, throughput, and movement time. AR designers can improve performance when designing AR interfaces by selecting larger buttons when accuracy and efficiency are required and by embedding perception cues to button target surfaces such as depth and proximity cues. For the neurovestibular tasks, while participants were able to perform all assessment tasks successfully, their strategies differed when comparing AR performance to physical object use. Task completion times were longer when administered in AR. In the Ingress/Egress task and the Obstacle Avoidance task, head pitch increased while overall head motion and torso motion decreased. These adaptations align with maintaining head stability for viewing the holographic content. The participants were able to complete all tasks within AR and meaningful measures of task performance and postural control were obtained, indicating that AR may be a useful instrumentation solution with embedded sensors to evaluate task performance and postural control. However, care should be taken when comparing performance within AR to other assessment modalities. Results from these studies support research gaps identified in NASA’s Human Research Roadmaps, provide design guidance for AR in NASA’s Human Integration Design Handbook, and support determining whether AR is a viable tool for evaluating astronauts’ vestibular performance throughout mission timelines.

This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

Crew extravehicular activity (EVA) is limited on spaceflight missions. Multiple, small robotic spacecraft with varying levels of autonomy are needed to perform tasks that might have been completed by an astronaut (e.g., an exterior surface inspection or repair). Crews on long duration exploration missions (LDEM) will have less access to ground support during task operations. As a result, they will need to process more information and communicate with autonomous robots effectively to ensure tasks are progressing safely and on schedule.

The objective of these studies is to investigate the use of augmented reality (AR) multimodal interface displays and communication pathways for improving human-robot communication, situation awareness (SA), trust, and task performance. This will lead to developing guidelines for designing human-robot system interactions that enable operational performance for crews on spaceflight missions.

The specific aims are to:

1) Develop a simulation testbed for examining communication between human-robot teams.

2) Develop a hardware testbed for examining communication between human-robot teams.

3) Evaluate human SA, trust, and task performance within a short duration and long-duration ground-based study (simulation and/or hardware) through testing various interface communication modalities and information displays.

4) (Option) Perform additional studies for alternate parameters of interest that could be tested using the study testbeds. Additional parameters include timing and persistence of information, gesture command mapping, varying the levels of robot automation, evaluating precision enabled by each command mode.

During this reporting period, our team supported the NASA Human Exploration Research Analog (HERA) Campaign 6 missions in data collection and performed data analysis for the missions completed. This study used the Unity-based simulation testbed developed in an earlier year of the program for evaluating the aims of assessing visual presentation modality across two tasks. The visual modalities considered were: (1) 2D camera images from fixed cameras placed externally on a simulated space station, (2) 3D reconstruction shown on a 2D projection, and (3) 3D reconstruction shown in an Augmented Reality environment using the HoloLens v2 platform. The reconstruction simulates using 2D camera images from the inspector to create the 3D object. The tasks evaluated were Synchronous and Asynchronous inspection task. In the Synchronous task, participants flew an inspector robot around the spacecraft to identify any surface anomalies that required closer inspection. The inspector could be flown in automatic mode along a predetermined path or manual mode to move off the assigned path. The robot was controlled with a 3-DOF joystick. In the Asynchronous task, participants analyzed the imagery from a previous inspection flight to identify the potential anomalies on the spacecraft exterior. In both tasks, detected anomalies were captured by taking a picture of the anomaly within the viewpoint. Initial HERA results support that detection accuracy was the highest for the 2D display for the Synchronous Inspection task. Additional interactive 3D viewpoints decreased detection accuracy and increased task completion time. Augmented Reality provided no significant improvement to local navigation, i.e., minimum distance to station or portion of time within two meters, suggesting that the technology did not enhance the perception level of situation awareness. Based on these findings, mission planning operations, when applicable, should include synchronous human-in-the-loop presence for telerobotic inspection of spacecraft. Additional details on the single session study are available in Weiss et al. (2021) and on the initial HERA findings in Liu et al. (2022).

During the reporting period, our team also performed studies to support gaps in the NASA Human Integration Design Handbook (HIDH) for augmented reality (AR) interfaces. We compared performance on sensorimotor and neurovestibular assessment tasks using physical objects and within AR. The sensorimotor task was a multi-directional Fitts’ Law target acquisition task, while the neurovestibular tasks included three clinical balance assessments (Four Square Step test (FSST), the Star Excursion, tandem walking (TW)), and three operational tasks (capsule ingress and egress, geology sampling, and obstacle avoidance). For the sensorimotor task, the touchscreen modality yielded improved performance over AR as measured by accuracy, precision, error rates, throughput, and movement time. AR designers can improve performance when designing AR interfaces by selecting larger buttons when accuracy and efficiency are required and by embedding perception cues to button target surfaces such as depth and proximity cues. For the neurovestibular tasks, while participants were able to perform all assessment tasks successfully, their strategies differed when comparing AR performance to physical object use. Task completion times were longer when administered in AR. Preliminary results yielded higher step heights in the TW, FSST, and capsule egress task as well as higher foot placement variability in the FSST. To maintain head stability for viewing the holographic content within the TW task, participants restricted their torso movement. While performing the capsule egress task, the AR and physical objects differed significantly in the downward pitch of the head, but not the torso. The participants were able to complete all tasks within AR and meaningful measures of task performance and postural control were obtained, indicating that AR may be a useful instrumentation solution with embedded sensors to evaluate task performance and postural control. However, care should be taken when comparing performance within AR to other assessment modalities. Results from these studies support research gaps identified in NASA’s Human Research Roadmaps, provide design guidance for AR in NASA’s Human Integration Design Handbook, and support determining whether AR is a viable tool for evaluating astronauts’ vestibular performance throughout mission timelines.

This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

Crew extravehicular activity (EVA) is limited on spaceflight missions. Multiple, small robotic spacecraft with varying levels of autonomy are needed to perform tasks that might have been completed by an astronaut (e.g., an exterior surface inspection or repair). Crews on long duration exploration missions (LDEM) will have less access to ground support during task operations. As a result, they will need to process more information and communicate with autonomous robots effectively to ensure tasks are progressing safely and on schedule.

The objective of these studies is to investigate the use of augmented reality (AR) multimodal interface displays and communication pathways for improving human-robot communication, situation awareness (SA), trust, and task performance. This will lead to developing guidelines for designing human-robot system interactions that enable operational performance for crews on spaceflight missions.

The specific aims are to:

1) Develop a simulation testbed for examining communication between human-robot teams.

2) Develop a hardware testbed for examining communication between human-robot teams.

3) Evaluate human SA, trust, and task performance within a short duration and long-duration ground-based study (simulation and/or hardware) through testing various interface communication modalities and information displays.

4) (Option) Perform additional studies for alternate parameters of interest that could be tested using the study testbeds. Additional parameters include timing and persistence of information, gesture command mapping, varying the levels of robot automation, evaluating precision enabled by each command mode.

During this reporting period, our team supported the NASA Human Exploration Research Analog (HERA) Campaign 6 missions in data collection and performed data analysis for the missions completed. This study used the Unity-based simulation testbed for evaluating the aims of assessing visual presentation modality across two tasks. The visual modalities considered were: (1) 2D camera images from fixed cameras placed externally on a simulated space station, (2) 3D reconstruction shown on a 2D projection, and (3) 3D reconstruction shown in an Augmented Reality environment using the HoloLens v2 platform. The reconstruction simulates using 2D camera images from the inspector to create the 3D object. The tasks evaluated are Synchronous and Asynchronous inspection tasks. In the Synchronous task, participants fly an inspector robot around the spacecraft to identify any surface anomalies that require closer inspection. The inspector can be flown in automatic mode along a predetermined path or manual mode to move off the assigned path. The robot is controlled with a 3-DOF joystick. In the Asynchronous task, participants analyze the imagery from a previous inspection flight to identify the potential anomalies on the spacecraft exterior. In both tasks, detected anomalies are captured by taking a picture of the anomaly within the viewpoint. Initial HERA results support that detection accuracy was the highest for the 2D display for the Synchronous Inspection task. Additional interactive 3D viewpoints decreased detection accuracy and increased task completion time. Augmented Reality provided no significant improvement to local navigation, i.e., minimum distance to station or portion of time within two meters, suggesting that the technology did not enhance the perception level of situation awareness. Based on these findings, mission planning operations, when applicable, should include synchronous human-in-the-loop presence for telerobotic inspection of spacecraft. Additional details on the single session study are available in Weiss et al. (2021) and on the initial HERA findings in Liu et al. (2022).

During the reporting period, our team also continued to develop a hardware-based implementation of the anomaly-detection task. We previously built a scaled physical mock-up of a space station and configured a quadcopter for a hardware-based ground evaluation. During this report period, pilot testing was performed to examine the hardware testbed and communication between human-robot teams. Additional details on these tests are reported in Weiss et al. (2022). Localization of the quadcopter for the automated mode is performed with motion capture, while the manual mode is performed using a handheld controller. Effort continues on the automation algorithms to support coverage mapping and real-time object reconstruction using this testbed.

Finally, our team began our effort to compare augmented reality task performance with physical task performance to support new knowledge for the NASA Human Integration Design Handbook (HIDH). Tasks selected align with sensorimotor and neurovestibular assessment and include a Fitts’ Law task examining precision and accuracy, as well as dynamic and operational balance assessments that make use of the virtual environment. This reporting period included software development and initial user studies. Human studies testing has begun and will continue into the next reporting period.

This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

Crew extravehicular activity (EVA) is limited on spaceflight missions. Multiple, small robotic spacecraft with varying levels of autonomy are needed to perform tasks that might have been completed by an astronaut (e.g., an exterior surface inspection or repair). Crews on long duration exploration missions (LDEM) will have less access to ground support during task operations. As a result, they will need to process more information and communicate with autonomous robots effectively to ensure tasks are progressing safely and on schedule.

The objective of these studies is to investigate the use of augmented reality (AR) multimodal interface displays and communication pathways for improving human-robot communication, situation awareness (SA), trust, and task performance. This will lead to developing guidelines for designing human-robot system interactions that enable operational performance for crews on spaceflight missions.

The specific aims are to:

1) Develop a simulation testbed for examining communication between human-robot teams.

2) Develop a hardware testbed for examining communication between human-robot teams.

3) Evaluate human SA, trust, and task performance within a short duration and long-duration ground-based study (simulation and/or hardware) through testing various interface communication modalities and information displays.

4) (Option) Perform additional studies for alternate parameters of interest that could be tested using the study testbeds. Additional parameters include timing and persistence of information, gesture command mapping, varying the levels of robot automation, evaluating precision enabled by each command mode.

During this reporting period, our team used the Unity-based simulation testbed for evaluating the aims of assessing visual presentation modality across two tasks. The visual modalities considered were (1) 2D camera images from fixed cameras placed externally on a simulated space station, (2) 3D reconstruction shown on a 2D projection, and (3) 3D reconstruction shown in an Augmented Reality environment using the HoloLens v2 platform. The reconstruction simulates using 2D camera images from the inspector to create the 3D object. The tasks evaluated are Synchronous and Asynchronous inspection tasks. In the Synchronous task, participants fly an inspector robot around the spacecraft to identify any surface anomalies that require closer inspection. The inspector can be flown in automatic mode along a predetermined path or manual mode to move off the assigned path. The robot is controlled with a 3-DOF joystick. In the Asynchronous task, participants analyze the imagery from a previous inspection flight to identify the potential anomalies on the spacecraft exterior. In both tasks, detected anomalies are captured by taking a picture of the anomaly within the viewpoint. We found that detection accuracy was the highest for the 2D display for the Synchronous Inspection task. Additional interactive 3D viewpoints decreased detection accuracy and increased task completion time. Augmented Reality provided no significant improvement to local navigation, i.e., minimum distance to station or portion of time within two meters, suggesting that the technology did not enhance the perception level of situation awareness. Based on these findings, mission planning operations, when applicable, should include synchronous human-in-the-loop presence for telerobotic inspection of spacecraft. Additional details on this study are available in our Human Factors journal article, Weiss et al. (2021). [Ed. Note: see Bibliography].

During the reporting period, our team also designed and built a scaled physical mock-up of a space station and configured a quadcopter for a hardware-based ground evaluation. Localization of the quadcopter for the automated mode is performed with motion capture, while the manual mode is performed using a handheld controller. Initial pilot testing with the integrated hardware system is ongoing and will continue into the next reporting period.

Finally, our team prepared for NASA Human Exploration Research Analog (HERA) testing, which will perform the same simulated study with the Unity-based environment, but will be able to explore the effects of isolation and time duration on use of the visual modalities to support robotic-assisted inspections. As the reporting period is concluding, initial training on HERA-study participants is beginning. Data collection will continue into the next reporting period.

This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

Crew extravehicular activity (EVA) is limited on spaceflight missions. Multiple, small robotic spacecraft with varying levels of autonomy are needed to perform tasks that might have been completed by an astronaut (e.g., an exterior surface inspection or repair). Crews on long duration exploration missions (LDEM) will have less access to ground support during task operations. As a result, they will need to process more information and communicate with autonomous robots effectively to ensure tasks are progressing safely and on schedule.

The objective of these studies is to investigate the use of augmented reality (AR) multimodal interface displays and communication pathways for improving human-robot communication, situation awareness (SA), trust, and task performance. This will lead to developing guidelines for designing human-robot system interactions that enable operational performance for crews on spaceflight missions.

The specific aims are to:

1) Develop a simulation testbed for examining communication between human-robot teams.

2) Develop a hardware testbed for examining communication between human-robot teams.

3) Evaluate human SA, trust, and task performance within a short duration and long-duration ground-based study (simulation and/or hardware) through testing various interface communication modalities and information displays.

4) (Option) Perform additional studies for alternate parameters of interest that could be tested using the study testbeds. Additional parameters include timing and persistence of information, gesture command mapping, varying the levels of robot automation, evaluating precision enabled by each command mode.

During this reporting period, our team developed a Unity-based simulation testbed for evaluating the aims of assessing visual presentation modality across two tasks. The visual modalities considered are (1) 2D camera images from fixed cameras placed external on a simulated space station, (2) 3D reconstruction shown on a 2D projection, and (3) 3D reconstruction shown in an Augmented Reality environment using the HoloLens v2 platform. The reconstruction simulates using 2D camera images from the inspector to create the 3D object. The tasks evaluated are Telerobotic Control and Inspection Task (TCIT) and the Analysis Task (AT). In TCIT, participants fly an inspector robot around the spacecraft to identify any surface anomalies that require closer inspection. The inspector can be flown in automatic mode along a predetermined path or manual mode to move off the assigned path. The robot is controlled with a 3-DOF (degree of freedom) joystick. In AT, participants analyze the imagery from a previous inspection flight to identify the potential anomalies on the spacecraft exterior. In both tasks, detected anomalies are captured by taking a picture of the anomaly within the viewpoint. The 2D and 3D interfaces are shown on a 24" computer monitor, while the AR image is presented on the Microsoft HoloLens. A second 24" computer monitor is used to display additional task information.

For the development of the simulation environment, a mock-up of a space station and an inspector satellite were created. Different anomaly sets were designed of equivalent difficulty to enable randomization of anomaly placement across trials. The automated mode was designed and implemented enabling the inspector to have edge tracking of the space station. The joystick was enabled to permit control of the inspector, picture taking tools, and mode switching.

Due to the pandemic, user tests planned for the summer months were not able to be performed. However, the simulation environment was configured in the home of three research team members to perform initial user testing within the team. A pipeline for data analysis was also developed and tested. We recently received approval to restart human studies and are preparing to begin the study to evaluate visual presentation modality. We were also able to virtually support integration of the hardware and software into the Human Exploration Research Analog (HERA) for the upcoming campaign.

his task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

Crew extravehicular activity (EVA) is limited on spaceflight missions. Multiple, small robotic spacecraft with varying levels of autonomy are needed to perform tasks that might have been completed by an astronaut (e.g., an exterior surface inspection or repair). Crews on long duration exploration missions (LDEM) will have less access to ground support during task operations. As a result, they will need to process more information and communicate with autonomous robots effectively to ensure tasks are progressing safely and on schedule.

The objective of these studies is to investigate the use of augmented reality (AR) multimodal interface displays and communication pathways for improving human-robot communication, situation awareness (SA), trust, and task performance. This will lead to developing guidelines for designing human-robot system interactions that enable operational performance for crews on spaceflight missions.

The specific aims are to:

1) Develop a simulation testbed for examining communication between human-robot teams.

2) Develop a hardware testbed for examining communication between human-robot teams.

3) Evaluate human SA, trust, and task performance within a short duration and long-duration ground-based study (simulation and/or hardware) through testing various interface communication modalities and information displays.

4) (Option) Perform additional studies for alternate parameters of interest that could be tested using the study testbeds. Additional parameters include timing and persistence of information, gesture command mapping, varying the levels of robot automation, evaluating precision enabled by each command mode.

NOTE (Ed., 4/15/2020) this is a continuation of "HCAAM VNSCOR: Responsive Multimodal Human-Automation Communication for Augmenting Human Situation Awareness in Nominal and Off-Nominal Scenarios," grant 80NSSC19K0703, due to PI Dr. Leia Stirling's move to University of Michigan from Massachusetts Institute of Technology in fall 2019. See that project for previous reporting.