This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

Crew extravehicular activity (EVA) is limited on spaceflight missions. Multiple, small robotic spacecraft with varying levels of autonomy are needed to perform tasks that might have been completed by an astronaut (e.g., an exterior surface inspection or repair). Crews on long duration exploration missions (LDEM) will have less access to ground support during task operations. As a result, they will need to process more information and communicate with autonomous robots effectively to ensure tasks are progressing safely and on schedule.

The objective of these studies is to investigate the use of augmented reality (AR) multimodal interface displays and communication pathways for improving human-robot communication, situation awareness (SA), trust, and task performance. This will lead to developing guidelines for designing human-robot system interactions that enable operational performance for crews on spaceflight missions.

The specific aims are to:

1) Develop a simulation testbed for examining communication between human-robot teams.

2) Develop a hardware testbed for examining communication between human-robot teams.

3) Evaluate human SA, trust, and task performance within a short duration and long-duration ground-based study (simulation and/or hardware) through testing various interface communication modalities and information displays.

4) (Option) Perform additional studies for alternate parameters of interest that could be tested using the study testbeds. Additional parameters include timing and persistence of information, gesture command mapping, varying the levels of robot automation, evaluating precision enabled by each command mode.

During this reporting period, our team supported the NASA Human Exploration Research Analog (HERA) Campaign 6 missions in data collection and performed data analysis for the missions completed. This study used the Unity-based simulation testbed developed in an earlier year of the program for evaluating the aims of assessing visual presentation modality across two tasks. The visual modalities considered were: (1) 2D camera images from fixed cameras placed externally on a simulated space station, (2) 3D reconstruction shown on a 2D projection, and (3) 3D reconstruction shown in an Augmented Reality environment using the HoloLens v2 platform. The reconstruction simulates using 2D camera images from the inspector to create the 3D object. The tasks evaluated were Synchronous and Asynchronous inspection task. In the Synchronous task, participants flew an inspector robot around the spacecraft to identify any surface anomalies that required closer inspection. The inspector could be flown in automatic mode along a predetermined path or manual mode to move off the assigned path. The robot was controlled with a 3-DOF joystick. In the Asynchronous task, participants analyzed the imagery from a previous inspection flight to identify the potential anomalies on the spacecraft exterior. In both tasks, detected anomalies were captured by taking a picture of the anomaly within the viewpoint. Initial HERA results support that detection accuracy was the highest for the 2D display for the Synchronous Inspection task. Additional interactive 3D viewpoints decreased detection accuracy and increased task completion time. Augmented Reality provided no significant improvement to local navigation, i.e., minimum distance to station or portion of time within two meters, suggesting that the technology did not enhance the perception level of situation awareness. Based on these findings, mission planning operations, when applicable, should include synchronous human-in-the-loop presence for telerobotic inspection of spacecraft. Additional details on the single session study are available in Weiss et al. (2021) and on the initial HERA findings in Liu et al. (2022).

During the reporting period, our team also performed studies to support gaps in the NASA Human Integration Design Handbook (HIDH) for augmented reality (AR) interfaces. We compared performance on sensorimotor and neurovestibular assessment tasks using physical objects and within AR. The sensorimotor task was a multi-directional Fitts’ Law target acquisition task, while the neurovestibular tasks included three clinical balance assessments (Four Square Step test (FSST), the Star Excursion, tandem walking (TW)), and three operational tasks (capsule ingress and egress, geology sampling, and obstacle avoidance). For the sensorimotor task, the touchscreen modality yielded improved performance over AR as measured by accuracy, precision, error rates, throughput, and movement time. AR designers can improve performance when designing AR interfaces by selecting larger buttons when accuracy and efficiency are required and by embedding perception cues to button target surfaces such as depth and proximity cues. For the neurovestibular tasks, while participants were able to perform all assessment tasks successfully, their strategies differed when comparing AR performance to physical object use. Task completion times were longer when administered in AR. Preliminary results yielded higher step heights in the TW, FSST, and capsule egress task as well as higher foot placement variability in the FSST. To maintain head stability for viewing the holographic content within the TW task, participants restricted their torso movement. While performing the capsule egress task, the AR and physical objects differed significantly in the downward pitch of the head, but not the torso. The participants were able to complete all tasks within AR and meaningful measures of task performance and postural control were obtained, indicating that AR may be a useful instrumentation solution with embedded sensors to evaluate task performance and postural control. However, care should be taken when comparing performance within AR to other assessment modalities. Results from these studies support research gaps identified in NASA’s Human Research Roadmaps, provide design guidance for AR in NASA’s Human Integration Design Handbook, and support determining whether AR is a viable tool for evaluating astronauts’ vestibular performance throughout mission timelines.

This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

Crew extravehicular activity (EVA) is limited on spaceflight missions. Multiple, small robotic spacecraft with varying levels of autonomy are needed to perform tasks that might have been completed by an astronaut (e.g., an exterior surface inspection or repair). Crews on long duration exploration missions (LDEM) will have less access to ground support during task operations. As a result, they will need to process more information and communicate with autonomous robots effectively to ensure tasks are progressing safely and on schedule.

The objective of these studies is to investigate the use of augmented reality (AR) multimodal interface displays and communication pathways for improving human-robot communication, situation awareness (SA), trust, and task performance. This will lead to developing guidelines for designing human-robot system interactions that enable operational performance for crews on spaceflight missions.

The specific aims are to:

1) Develop a simulation testbed for examining communication between human-robot teams.

2) Develop a hardware testbed for examining communication between human-robot teams.

3) Evaluate human SA, trust, and task performance within a short duration and long-duration ground-based study (simulation and/or hardware) through testing various interface communication modalities and information displays.

4) (Option) Perform additional studies for alternate parameters of interest that could be tested using the study testbeds. Additional parameters include timing and persistence of information, gesture command mapping, varying the levels of robot automation, evaluating precision enabled by each command mode.

During this reporting period, our team supported the NASA Human Exploration Research Analog (HERA) Campaign 6 missions in data collection and performed data analysis for the missions completed. This study used the Unity-based simulation testbed for evaluating the aims of assessing visual presentation modality across two tasks. The visual modalities considered were: (1) 2D camera images from fixed cameras placed externally on a simulated space station, (2) 3D reconstruction shown on a 2D projection, and (3) 3D reconstruction shown in an Augmented Reality environment using the HoloLens v2 platform. The reconstruction simulates using 2D camera images from the inspector to create the 3D object. The tasks evaluated are Synchronous and Asynchronous inspection tasks. In the Synchronous task, participants fly an inspector robot around the spacecraft to identify any surface anomalies that require closer inspection. The inspector can be flown in automatic mode along a predetermined path or manual mode to move off the assigned path. The robot is controlled with a 3-DOF joystick. In the Asynchronous task, participants analyze the imagery from a previous inspection flight to identify the potential anomalies on the spacecraft exterior. In both tasks, detected anomalies are captured by taking a picture of the anomaly within the viewpoint. Initial HERA results support that detection accuracy was the highest for the 2D display for the Synchronous Inspection task. Additional interactive 3D viewpoints decreased detection accuracy and increased task completion time. Augmented Reality provided no significant improvement to local navigation, i.e., minimum distance to station or portion of time within two meters, suggesting that the technology did not enhance the perception level of situation awareness. Based on these findings, mission planning operations, when applicable, should include synchronous human-in-the-loop presence for telerobotic inspection of spacecraft. Additional details on the single session study are available in Weiss et al. (2021) and on the initial HERA findings in Liu et al. (2022).

During the reporting period, our team also continued to develop a hardware-based implementation of the anomaly-detection task. We previously built a scaled physical mock-up of a space station and configured a quadcopter for a hardware-based ground evaluation. During this report period, pilot testing was performed to examine the hardware testbed and communication between human-robot teams. Additional details on these tests are reported in Weiss et al. (2022). Localization of the quadcopter for the automated mode is performed with motion capture, while the manual mode is performed using a handheld controller. Effort continues on the automation algorithms to support coverage mapping and real-time object reconstruction using this testbed.

Finally, our team began our effort to compare augmented reality task performance with physical task performance to support new knowledge for the NASA Human Integration Design Handbook (HIDH). Tasks selected align with sensorimotor and neurovestibular assessment and include a Fitts’ Law task examining precision and accuracy, as well as dynamic and operational balance assessments that make use of the virtual environment. This reporting period included software development and initial user studies. Human studies testing has begun and will continue into the next reporting period.

This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

Crew extravehicular activity (EVA) is limited on spaceflight missions. Multiple, small robotic spacecraft with varying levels of autonomy are needed to perform tasks that might have been completed by an astronaut (e.g., an exterior surface inspection or repair). Crews on long duration exploration missions (LDEM) will have less access to ground support during task operations. As a result, they will need to process more information and communicate with autonomous robots effectively to ensure tasks are progressing safely and on schedule.

The objective of these studies is to investigate the use of augmented reality (AR) multimodal interface displays and communication pathways for improving human-robot communication, situation awareness (SA), trust, and task performance. This will lead to developing guidelines for designing human-robot system interactions that enable operational performance for crews on spaceflight missions.

The specific aims are to:

1) Develop a simulation testbed for examining communication between human-robot teams.

2) Develop a hardware testbed for examining communication between human-robot teams.

3) Evaluate human SA, trust, and task performance within a short duration and long-duration ground-based study (simulation and/or hardware) through testing various interface communication modalities and information displays.

4) (Option) Perform additional studies for alternate parameters of interest that could be tested using the study testbeds. Additional parameters include timing and persistence of information, gesture command mapping, varying the levels of robot automation, evaluating precision enabled by each command mode.

During this reporting period, our team used the Unity-based simulation testbed for evaluating the aims of assessing visual presentation modality across two tasks. The visual modalities considered were (1) 2D camera images from fixed cameras placed externally on a simulated space station, (2) 3D reconstruction shown on a 2D projection, and (3) 3D reconstruction shown in an Augmented Reality environment using the HoloLens v2 platform. The reconstruction simulates using 2D camera images from the inspector to create the 3D object. The tasks evaluated are Synchronous and Asynchronous inspection tasks. In the Synchronous task, participants fly an inspector robot around the spacecraft to identify any surface anomalies that require closer inspection. The inspector can be flown in automatic mode along a predetermined path or manual mode to move off the assigned path. The robot is controlled with a 3-DOF joystick. In the Asynchronous task, participants analyze the imagery from a previous inspection flight to identify the potential anomalies on the spacecraft exterior. In both tasks, detected anomalies are captured by taking a picture of the anomaly within the viewpoint. We found that detection accuracy was the highest for the 2D display for the Synchronous Inspection task. Additional interactive 3D viewpoints decreased detection accuracy and increased task completion time. Augmented Reality provided no significant improvement to local navigation, i.e., minimum distance to station or portion of time within two meters, suggesting that the technology did not enhance the perception level of situation awareness. Based on these findings, mission planning operations, when applicable, should include synchronous human-in-the-loop presence for telerobotic inspection of spacecraft. Additional details on this study are available in our Human Factors journal article, Weiss et al. (2021). [Ed. Note: see Bibliography].

During the reporting period, our team also designed and built a scaled physical mock-up of a space station and configured a quadcopter for a hardware-based ground evaluation. Localization of the quadcopter for the automated mode is performed with motion capture, while the manual mode is performed using a handheld controller. Initial pilot testing with the integrated hardware system is ongoing and will continue into the next reporting period.

Finally, our team prepared for NASA Human Exploration Research Analog (HERA) testing, which will perform the same simulated study with the Unity-based environment, but will be able to explore the effects of isolation and time duration on use of the visual modalities to support robotic-assisted inspections. As the reporting period is concluding, initial training on HERA-study participants is beginning. Data collection will continue into the next reporting period.

This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

Crew extravehicular activity (EVA) is limited on spaceflight missions. Multiple, small robotic spacecraft with varying levels of autonomy are needed to perform tasks that might have been completed by an astronaut (e.g., an exterior surface inspection or repair). Crews on long duration exploration missions (LDEM) will have less access to ground support during task operations. As a result, they will need to process more information and communicate with autonomous robots effectively to ensure tasks are progressing safely and on schedule.

The objective of these studies is to investigate the use of augmented reality (AR) multimodal interface displays and communication pathways for improving human-robot communication, situation awareness (SA), trust, and task performance. This will lead to developing guidelines for designing human-robot system interactions that enable operational performance for crews on spaceflight missions.

The specific aims are to:

1) Develop a simulation testbed for examining communication between human-robot teams.

2) Develop a hardware testbed for examining communication between human-robot teams.

3) Evaluate human SA, trust, and task performance within a short duration and long-duration ground-based study (simulation and/or hardware) through testing various interface communication modalities and information displays.

4) (Option) Perform additional studies for alternate parameters of interest that could be tested using the study testbeds. Additional parameters include timing and persistence of information, gesture command mapping, varying the levels of robot automation, evaluating precision enabled by each command mode.

During this reporting period, our team developed a Unity-based simulation testbed for evaluating the aims of assessing visual presentation modality across two tasks. The visual modalities considered are (1) 2D camera images from fixed cameras placed external on a simulated space station, (2) 3D reconstruction shown on a 2D projection, and (3) 3D reconstruction shown in an Augmented Reality environment using the HoloLens v2 platform. The reconstruction simulates using 2D camera images from the inspector to create the 3D object. The tasks evaluated are Telerobotic Control and Inspection Task (TCIT) and the Analysis Task (AT). In TCIT, participants fly an inspector robot around the spacecraft to identify any surface anomalies that require closer inspection. The inspector can be flown in automatic mode along a predetermined path or manual mode to move off the assigned path. The robot is controlled with a 3-DOF (degree of freedom) joystick. In AT, participants analyze the imagery from a previous inspection flight to identify the potential anomalies on the spacecraft exterior. In both tasks, detected anomalies are captured by taking a picture of the anomaly within the viewpoint. The 2D and 3D interfaces are shown on a 24" computer monitor, while the AR image is presented on the Microsoft HoloLens. A second 24" computer monitor is used to display additional task information.

For the development of the simulation environment, a mock-up of a space station and an inspector satellite were created. Different anomaly sets were designed of equivalent difficulty to enable randomization of anomaly placement across trials. The automated mode was designed and implemented enabling the inspector to have edge tracking of the space station. The joystick was enabled to permit control of the inspector, picture taking tools, and mode switching.

Due to the pandemic, user tests planned for the summer months were not able to be performed. However, the simulation environment was configured in the home of three research team members to perform initial user testing within the team. A pipeline for data analysis was also developed and tested. We recently received approval to restart human studies and are preparing to begin the study to evaluate visual presentation modality. We were also able to virtually support integration of the hardware and software into the Human Exploration Research Analog (HERA) for the upcoming campaign.

his task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

Crew extravehicular activity (EVA) is limited on spaceflight missions. Multiple, small robotic spacecraft with varying levels of autonomy are needed to perform tasks that might have been completed by an astronaut (e.g., an exterior surface inspection or repair). Crews on long duration exploration missions (LDEM) will have less access to ground support during task operations. As a result, they will need to process more information and communicate with autonomous robots effectively to ensure tasks are progressing safely and on schedule.

The objective of these studies is to investigate the use of augmented reality (AR) multimodal interface displays and communication pathways for improving human-robot communication, situation awareness (SA), trust, and task performance. This will lead to developing guidelines for designing human-robot system interactions that enable operational performance for crews on spaceflight missions.

The specific aims are to:

1) Develop a simulation testbed for examining communication between human-robot teams.

2) Develop a hardware testbed for examining communication between human-robot teams.

3) Evaluate human SA, trust, and task performance within a short duration and long-duration ground-based study (simulation and/or hardware) through testing various interface communication modalities and information displays.

4) (Option) Perform additional studies for alternate parameters of interest that could be tested using the study testbeds. Additional parameters include timing and persistence of information, gesture command mapping, varying the levels of robot automation, evaluating precision enabled by each command mode.

NOTE (Ed., 4/15/2020) this is a continuation of "HCAAM VNSCOR: Responsive Multimodal Human-Automation Communication for Augmenting Human Situation Awareness in Nominal and Off-Nominal Scenarios," grant 80NSSC19K0703, due to PI Dr. Leia Stirling's move to University of Michigan from Massachusetts Institute of Technology in fall 2019. See that project for previous reporting.