Crew extravehicular activity (EVA) is limited on spaceflight missions. Multiple, small robotic spacecraft with varying levels of autonomy are needed to perform tasks that might have been completed by an astronaut (e.g., an exterior surface inspection or repair). Crews on long duration exploration missions (LDEM) will have less access to ground support during task operations. As a result, they will need to process more information and communicate with autonomous robots effectively to ensure tasks are progressing safely and on schedule.

The objective of these studies is to investigate the use of augmented reality (AR) multimodal interface displays and communication pathways for improving human-robot communication, situation awareness (SA), trust, and task performance. This will lead to developing guidelines for designing human-robot system interactions that enable operational performance for crews on spaceflight missions.

The specific aims are to:

1) Develop a simulation testbed for examining communication between human-robot teams.

2) Develop a hardware testbed for examining communication between human-robot teams.

3) Evaluate human SA, trust, and task performance within a short duration and long-duration ground-based study (simulation and/or hardware) through testing various interface communication modalities and information displays.

4) (Option) Perform additional studies for alternate parameters of interest that could be tested using the study testbeds. Additional parameters include timing and persistence of information, gesture command mapping, varying the levels of robot automation, evaluating precision enabled by each command mode.

Subjects performed the inspection in both fixed command mode (in each of the three operation modes, order randomized) and in free mode (with free choice of when and how often they utilized different operation modes). Performance measures included percentage of station inspected, number of collisions, and accuracy in anomaly detection. Workload was evaluated using the NASA Task Load Index (TLX) method. Interactions with the simulated environment were logged to characterize subject strategies for utilizing the interface (i.e., moving the viewpoint, changing scale and position of virtual models).

Manually controlling the satellite in the global and local frames maximized the percentage of the station that was inspected, while using waypoints led to fewer collisions between the satellite and the station. There was no significant difference in anomaly detection across the various command modes for the anomaly frequency and type selected. When operating with free choice of command mode, subjects preferred to remain in a single mode, typically either the global or local control. Subject feedback and NASA TLX scores suggest the global and local modes required less workload and were more usable than waypoint control as currently implemented. One subject was observed to select strategies that did not comply with our instructed task priorities as they appeared to prioritize speed over anomaly detection. The AR interface enabled users to manipulate the virtual models to alter their viewpoint and move around the virtual model in the real-world space throughout the inspection task. The design of waypoint navigation enabled better collision avoidance as it encouraged planning, but was more suited to single-point inspection tasks as currently implemented. The global and local modes were more suited to the patrol task in this study, enabling easier navigation at close proximity.

Based on the initial pilot evaluation, we are in the process of updating the simulation environment. While the initial pilot study focused on robotic system control and inspection, the follow-on study simulation incorporates two different tasks across the visual modalities, an inspection-only (IO) task, where the images are pre-recorded, and a robotic control and inspection (RCI), where the operator must both obtain the images and inspect within the same trial. We have also enabled an automated mode where the robot performs an edge following task. In the manual modes, the operator can still control in local or global mode. While waypoint navigation provides a mid-level form of automation, it was determined that this control mode was not appropriate for the perimeter inspection task. When the operator manually stops the edge following, they would switch into manual control mode. During the next year of the program, the simulation study comparing the visual modalities will be performed.

[EDITOR'S NOTE 4/15/2020: Due to Principal Investigator Stirling's move to University of Michigan, the project was transferred there with a new grant number 80NSSC20K0409 and a new period of performance, 12/4/2019-12/3/2023. See the continuation project for subsequent reporting.]

The objective of this research is to provide recommendations for augmenting human situation awareness (SA) and task performance through multimodal displays and communication pathways based on empirical evidence. Specifically, we will evaluate the effectiveness of several multimodal Virtual Reality (VR) techniques in providing spatial and temporal SA to a human operator controlling multiple semi-autonomous agents. Our testbed will simulate a Long-Duration Exploration Mission (LDEM) inspection task using the ground-based Massachusetts Institute of Technology (MIT) Synchronized Position Hold, Engage, Reorient, Experimental Satellites (SPHERES) platform enhanced with NASA Jet Propulsion Laboratory (NASA-JPL) automatic scene reconstruction capability. A human study will be conducted with the human supervisor providing commands to the SPHERES using images rendered in a virtual environment. The results of this project will provide empirical evidence for revising portions of NASA-STD-3001 and the NASA Human Integration Design Handbook (HIDH) that guide interface design for effective SA and task performance. There is a need to expand current guidance on responsive displays, especially when integrated with VR technologies, to enable SA for relevant operational tasks.

The proposed project will integrate current NASA-JPL technology within a small robotic satellite testbed to examine the bi-directional communication between the human-robot team to enable improved SA. We propose the following specific aims: (1) Integrate and extend existing capabilities at JPL and MIT into a testbed for examining information communication between human-autonomy teams and (2) Evaluate SA, trust, and task performance within a ground-based study with selected communication modalities and information displays.