The proposed research addresses three main research questions: First, how should roles and responsibilities be optimally assigned to robots and humans based on a combination of task demands, robotic capabilities, and available crew resources, with special attention to the capabilities inherent to classes of robots? Second, what is the human-robot system performance trade-space that serves as the basis for the allocation? Third, how can this function allocation method be validated as creating appropriate function allocation for both nominal and off-nominal operations?

We propose a three year effort to address these questions. In the first year we propose to model the function allocation design space that exists between humans and robots in deep space exploration missions. We will use a computational framework called Work Models that Compute (WMC), which allows us to model dynamical systems (such as space vehicles and robots), automated systems (such as the automated rendezvous and docking system), and human agents working together to achieve common goals. WMC was custom designed to model function allocation and to measure eight metrics of function allocation previously established by the proposers. In the second year we will explore the design space, deeply investigating each metric such as taskload, authority-responsibility mismatch, coherency, etc., while beginning the validation process through the use of human-in-the-loop experiments with simulated robots. In the final year we will move from exploring each metric individually to looking at their combined effects as we vary the design space constraints, the tasks, crew stress levels, and function allocation options. We will continue our validation efforts using human-in-the-loop experiments using a combination of simulated robots and/or real robots. These experiments will systematically explore a large number of conditions such that they serve not only to demonstrate the function allocation chosen by the method, but also to validate the method.

First, this project uses current-day computational methods to model and simulate the human-robot teams at work. We are expanding on existing methods used in aeronautics to advance the field of computational simulation of function allocation for the improvement of crewed space exploration where we encounter additional challenges of agents with differential capabilities, time delay of communication, and the need to represent limitations in resources which might be both physical (say a wrench or oxygen) as well as informational (say the current CO2 levels). The capability to simulate how human-robot teams work together will help systems designers understand the interaction between humans and space robotics to allow for robust and effective as well as efficient teamwork across missions and different crew-robot complements. In turn, human-robot teams not only become better at doing their taskwork, but also expand the capacity of what human-robot teams can accomplish. Human-robot teams may then go on to accomplish the numerous tasks that will expand humanity's knowledge of space exploration.

Second, our research also impacts the growing field of human-robot teaming, as robots continue to advance technically and become less like tools for humans and more like peers and teammates. The computational framework and capabilities we are creating and demonstrating advance the field of cognitive engineering to investigate robot-human teaming, which is a research area applicable to domains beyond space exploration including manufacturing, healthcare, transportation, and agriculture.

Finally, we introduced stochasticity into the simulation to identify how different inputs to our model affect various teamwork metrics. Throughout this work thus far, our various case studies have only used a few predetermined sets of inputs to analyze different scenarios. We expand upon our modeling by introducing Monte-Carlo simulations to randomly sample our various inputs and observe the output trends produced. The input variables that we introduced stochasticity to were action durations, responsibilities, and work strategies. The output metrics that we measured were mission metrics, team co-ordinations, physical coherency, informational coherency, and collaboration. We present four different case studies, the first three of which are based upon the in-orbit maintenance scenario and the fourth based upon the rover scenario.

A. Action Duration Case Study – On-orbit Maintenance. This case study focuses on the impact of action durations on the output metrics. We created five hundred different work allocations by randomly sampling five out of the thirty-nine actions to complete the work. The specific output metrics we observed were total mission durations, and human-robot teaming fluency metrics.

We first examine how action durations impact the overall mission duration and teamwork. We observe three different kinds of patterns in agent busy time when the action they are conducting varies in action duration. All three cases have linear relationships where increasing the action duration increases the total agent busytime; however, each has a different number of linear trends where the slope is the same but contain different intercepts. The Free flying robot has three clear trends, while the humanoid and EV (extra-vehicular) Astronaut each have two and one, respectively. We found these differences occurred from the way actions were scheduled in that small variations in action durations resulted in complex patterns where various agents traveled to different locations in different orders. This resulted in redundancies in locations traveled which is where the clear differences in intercepts lie. Therefore, we confirm part of our Hypothesis 1 that longer action durations will cause fluency metrics to be higher but with the caveat that how the robots traverse is a confounding factor.

B. Human and Robot Responsibilities Case Studies – On-orbit Maintenance. In these case studies, each of the actions that was authorized (conducted) by a robotic agent was varied between being responsible by the intra-vehicular (IV) astronaut or the EV astronaut. Out of the 39 total number of actions, 11 were allocated to the free flying robot, 3 to the Humanoid robot, and 8 to the Remote Manipulator System (RMS). In total, 600 different variations of work allocations were created by randomly selecting either the IV astronaut or EV astronaut to be responsible for every robot action.

Our second and third case studies took a deeper dive into how responsibility is related to human-robot teaming metrics. We found that giving robots responsibility of their own actions generally reduces the amount of communication that occurs between robot and human agents. However, this relationship was not completely linear with many variations on how much work the human IV astronaut conducted for each number of actions the robots were responsible for. This was due to the variety of action durations for which the robots were responsible.

We also showed that changing the responsibility of actions can change how team members communicate with each other. This tradeoff between agents can be used by team designers to ensure that team members are not overloaded or underloaded with work. We similarly show that goals, perceptions, intents, and evaluations can be traded between the IV astronaut and EV astronaut.

C. Work Strategies Case Study – Rover. In our fourth and last case study, we observed how work strategies impacted the total mission duration in a scenario where a rover and astronaut worked together in tandem. Unlike the action durations or responsibility results, the various work strategies changed the configuration of the mission into four discrete categories rather than a continuous distribution. This is because ordering the rover activities in different orders caused cases where interdependent activities had to be conducted in series rather than parallel. We show that Work Models that Compute (WMC) can survey a large state-space of work strategies and provide insight to mission designers on how interdependent activities interact with each other in complex scenarios.

X. DISCUSSION

Computational simulation of function allocation can provide objective insight in the teamwork that is required for human-robot interaction. We have created models suitable for such computational simulation, representing key aspects of human-robot interaction relative to the allocation of tasks between them.

We present two human-robot teaming scenarios: one on the International Space Station (ISS) with multiple robots and the other on the surface of the moon with a lunar rover. These scenarios demonstrate our capability to both model complex teams and also introduce kinematics and dynamics to our models. We also present various metrics and measures and how they relate to teaming fluency and human perceptions of the robot. An experimental confirmation was conducted to test how similar our models performed to real scenarios.

The benefit of using computational simulations to evaluate function allocation is that they can be used to identify potential pros and cons of various function allocations in the earliest design stages, without a need to conduct costly human-in-the-loop experiments. When potential problems are identified early in design, changes to the supporting technology and operations can still be made to alleviate the negative effects of a selected function allocation.

The proposed research addresses three main research questions: First, how should roles and responsibilities be optimally assigned to robots and humans based on a combination of task demands, robotic capabilities, and available crew resources, with special attention to the capabilities inherent to classes of robots? Second, what is the human-robot system performance trade-space that serves as the basis for the allocation? Third, how can this function allocation method be validated as creating appropriate function allocation for both nominal and off-nominal operations?

We propose a three year effort to address these questions. In the first year we propose to model the function allocation design space that exists between humans and robots in deep space exploration missions. We will use a computational framework called Work Models that Compute (WMC), which allows us to model dynamical systems (such as space vehicles and robots), automated systems (such as the automated rendezvous and docking system), and human agents working together to achieve common goals. WMC was custom designed to model function allocation and to measure eight metrics of function allocation previously established by the proposers. In the second year we will explore the design space, deeply investigating each metric such as taskload, authority-responsibility mismatch, coherency, etc., while beginning the validation process through the use of human-in-the-loop experiments with simulated robots. In the final year we will move from exploring each metric individually to looking at their combined effects as we vary the design space constraints, the tasks, crew stress levels, and function allocation options. We will continue our validation efforts using human-in-the-loop experiments using a combination of simulated robots and/or real robots. These experiments will systematically explore a large number of conditions such that they serve not only to demonstrate the function allocation chosen by the method, but also to validate the method.

First, this project uses current-day computational methods to model and simulate the human-robot teams at work. We are expanding on existing methods used in aeronautics to advance the field of computational simulation of function allocation for the improvement of crewed space exploration where we encounter additional challenges of agents with differential capabilities, time delay of communication, and the need to represent limitations in resources which might be both physical (say a wrench or oxygen) as well as informational (say the current CO2 levels). The capability to simulate how human-robot teams work together will help systems designers understand the interaction between humans and space robotics to allow for robust and effective as well as efficient teamwork across missions and different crew-robot complements. In turn, human-robot teams not only become better at doing their taskwork, but also expand the capacity of what human-robot teams can accomplish. Human-robot teams may then go on to accomplish the numerous tasks that will expand humanity's knowledge of space exploration.

Second, our research also impacts the growing field of human-robot teaming, as robots continue to advance technically and become less like tools for humans and more like peers and teammates. The computational framework and capabilities we are creating and demonstrating advance the field of cognitive engineering to investigate robot-human teaming, which is a research area applicable to domains beyond space exploration including manufacturing, healthcare, transportation, and agriculture.

In the second year of performance we created models of representative multi-human/multi-robot function allocations for prototypical EVA (extravehicular activity) missions. We then applied these models to demonstrate key implications in how various modes of human-robot interaction, including the implicit requirements for monitoring inherent to leaving human agents responsible for the outcome of tasks performed by robots, the implications of different human-robot control modes, and the idling time resulting from different distribution of tasks.

In this past year, we validated our simulation results by conducting a human-in-the-loop experiment of an on-orbit maintenance scenario. We then further modeled an additional case study involving an autonomous lunar rover with kinematic and dynamic motion. Finally, we applied work strategies to both these scenarios to identify how to select function allocations.

The proposed research addresses three main research questions: First, how should roles and responsibilities be optimally assigned to robots and humans based on a combination of task demands, robotic capabilities, and available crew resources, with special attention to the capabilities inherent to classes of robots? Second, what is the human-robot system performance trade-space that serves as the basis for the allocation? Third, how can this function allocation method be validated as creating appropriate function allocation for both nominal and off-nominal operations?

We propose a three year effort to address these questions. In the first year we propose to model the function allocation design space that exists between humans and robots in deep space exploration missions. We will use a computational framework called Work Models that Compute (WMC), which allows us to model dynamical systems (such as space vehicles and robots), automated systems (such as the automated rendezvous and docking system), and human agents working together to achieve common goals. WMC was custom designed to model function allocation and to measure eight metrics of function allocation previously established by the proposers. In the second year we will explore the design space, deeply investigating each metric such as taskload, authority-responsibility mismatch, coherency, etc., while beginning the validation process through the use of human-in-the-loop experiments with simulated robots. In the final year we will move from exploring each metric individually to looking at their combined effects as we vary the design space constraints, the tasks, crew stress levels, and function allocation options. We will continue our validation efforts using human-in-the-loop experiments using a combination of simulated robots and/or real robots. These experiments will systematically explore a large number of conditions such that they serve not only to demonstrate the function allocation chosen by the method, but also to validate the method.

First, this project uses current-day computational methods to model and simulate the human-robot teams at work. We are expanding on existing methods used in aeronautics to advance the field of computational simulation of function allocation for the improvement of crewed space exploration where we encounter additional challenges of agents with differential capabilities, time delay of communication, and the need to represent limitations in resources which might be both physical (say a wrench or oxygen) as well as informational (say the current CO2 levels). The capability to simulate how human-robot teams work together will help systems designers understand the interaction between humans and space robotics to allow for robust and effective as well as efficient teamwork across missions and different crew-robot complements. In turn, human-robot teams not only become better at doing their taskwork, but also expand the capacity of what human-robot teams can accomplish. Human-robot teams may then go on to accomplish the numerous tasks that will expand humanities knowledge of space exploration.

Second, our research also impacts the growing field of human-robot teaming, as robots continue to advance technically and become less like tools for humans and more like peers and teammates. The computational framework and capabilities we are creating and demonstrating advance the field of cognitive engineering to investigate robot-human teaming, which is a research area applicable to domains beyond space exploration including manufacturing, healthcare, transportation, and agriculture.

This second year of performance we continued our exploration of the function allocation design space, and more deeply investigated metrics of effective function allocation and human-robot teaming. Our first research effort was an elaborate study of taskload in relation to function allocation, where we studied function allocations under different imposed taskload constraints for each agent. The results highlighted how the taskload associated with teamwork requirements impacts idle time and mission duration, and it can be a driving factor in a function allocation’s effectiveness.

Our second research effort this year was the modeling and analysis of robotic failures. We have modeled robotic failures and the required response from human agents to resolve these failures. For different function allocations, we studied how the effects of failures ripple through the rest of a scenario. The results from this study showed that the effects of failures are different depending on whether the robot is aware of the failure. When the robot detects the failure, and notifies a responsible (human) agent to take action, the failed action could quickly be re-allocated to a nearby and capable agent and the resulting impact on the mission was minimal. On the other hand, if the failure is only discovered when the human operator confirms the robot’s action (thus the robot was not aware), the action needs to be re-done completely, and the impact on the mission are more apparent.

It was also clear the impact of the failure depends on how the failed action relates to others and how the actions were allocated across the agents. For example, if a failed action has many follow-up actions, they need to be delayed until the failure is resolved. These delays then have a particularly large impact if they are allocated to different agents, because all of these agents need to wait (idle time). If a failed action has no follow-up actions, other processes and actions can continue while the failure is resolved, thus resulting in minimal impact on the mission.

The third research effort was focused on incorporating human-robot teaming metrics in our methodology and simulation environment. This resulted in several new function allocation metrics associated with teamwork, including the spatial proximity of agents as they are working together and the coherency of function allocations (measured through required interaction and communication). We took an interaction model (the Scholtz model) that is widely used in the robotics community as a basis for modeling joint activity, in which two agents work together on a single task. This model allows us to evaluate communication requirements at different levels of cognition (goals, intent, perception, and evaluation). These new metrics, together with our existing metrics, have been incorporated in a postprocessing analysis script that creates reports for cross-comparing different function allocations.

Our fourth research effort is the validation of the simulation framework, which is currently ongoing. We have created an experiment plan for conducting a human-in-the-loop study, wherein we will have participants working on a simulated extra-vehicular mission. The participants will be working together with a simple robot, the Turtlebot2, which is able to perform inspections and fetch required tools. We have a total of four conditions, which vary in the function allocation between participant and robot, as well as the types of control input and monitoring that are supported by the robot. For each condition, we run a nominal case and a case wherein the robot fails one of the actions. The data gathered from this experiment will be compared to the output of our simulation framework to test whether it accurately captures the different aspects of function allocation and human-robot teaming.

Throughout these research efforts we have conducted several case studies, each extending our simulated extra-vehicular maintenance scenario. In this scenario, several panels exterior to the spacecraft need to be inspected and – if in bad condition – replaced. The aims of these case studies are to illustrate the use of our simulation framework for modeling and analyzing human-robot interaction in function allocation, including the metrics it can assess and the resulting insights that it can provide. The scenario includes six agents that can be deployed to execute the work: two human astronauts, one extra-vehicular (EV) and one intra-vehicular (IV), an RMS (remote manipulator system), a humanoid robot (Hum) that can operate inside and – at a more notional level – outside the spacecraft, a free-flying robot that can fetch tools, and the Mission Control Center (MCC). The robotic agents might need to be controlled by a human operator, depending on their capabilities per action, as well as the desired specifications of the function allocation. The scenario has been adapted for our validation experiment to include only one robot (the Turtlebot), working with the extra-vehicular and intra-vehicular astronauts.

A model of the work for this mission was created in the computational simulation framework Work Models That Compute. The work model consists of high-level function that each contain detailed descriptions of the work in the form of actions. These actions can be assigned to any agent involved in the scenario. The taskwork for the scenario includes actions associated with overhead tasks, locomotion outside the vehicle, inspection, replacement of a broken panel, and tool handling.

The proposed research addresses three main research questions: First, how should roles and responsibilities be optimally assigned to robots and humans based on a combination of task demands, robotic capabilities and available crew resources, with special attention to the capabilities inherent to classes of robots? Second, what is the human-robot system performance trade-space that serves as the basis for the allocation? Third, how can this function allocation method be validated as creating appropriate function allocation for both nominal and off-nominal operations?

We propose a three year effort to address these questions. In the first year we propose to model the function allocation design space that exists between humans and robots in deep space exploration missions. We will use a computational framework called Work Models that Compute (WMC), which allows us to model dynamical systems (such as space vehicles and robots), automated systems (such as the automated rendezvous and docking system), and human agents working together to achieve common goals. WMC was custom designed to model function allocation and to measure eight metrics of function allocation previously established by the proposers. In the second year we will explore the design space, deeply investigating each metric such as taskload, authority-responsibility mismatch, coherency, etc., while beginning the validation process through the use of human-in-the-loop experiments with simulated robots. In the final year we will move from exploring each metric individually to looking at their combined effects as we vary the design space constraints, the tasks, crew stress levels, and function allocation options. We will continue our validation efforts using human-in-the-loop experiments using a combination of simulated robots and/or real robots. These experiments will systematically explore a large number of conditions such that they serve not only to demonstrate the function allocation chosen by the method, but also to validate the method.

First, this project uses current-day computational methods to model and simulate the human-robot teams at work. We are expanding on existing methods used in aeronautics to advance the field of computational simulation of function allocation for the improvement of manned space exploration where we encounter additional challenges of agents with differential capabilities, time delay of communication, and the need to represent limitations in resources which might be both physical (say a wrench or oxygen) as well as informational (say the current CO2 levels). The capability to simulate how human-robot teams work together will help systems designers understand the interaction between humans and space robotics to allow for robust and effective as well as efficient teamwork across missions and different crew-robot complements. In turn, human-robot teams to not only become better at doing their taskwork, but also expand the capacity of what human-robot teams can accomplish. Human-robot teams may then go on to accomplish the numerous tasks that will expand humanities knowledge of space exploration.

Second, our research also impacts the growing field of human-robot teaming, as robots continue to advance technically and become less like tools for humans and more like peers and teammates. The computational framework and capabilities we are creating and demonstrating advance the field of cognitive engineering to investigate robot-human teaming, which is a research area applicable to domains beyond space exploration including manufacturing, healthcare, transportation, and agriculture.

As a case study, consider an on-orbit maintenance scenario in which three panels exterior to the spacecraft need to be inspected and – if in bad condition – replaced. The aim of this case study is to illustrate the use of our simulation framework for modeling and analyzing human-robot interaction in function allocation, including the metrics it can assess and the resulting insights that it can provide. The scenario includes six agents that can be deployed to execute the work: two human astronauts, one extra-vehicular (EV) and one intra-vehicular (IV), an RMS, two humanoid robots (Hum) that can operate inside and – at a more notional level – outside the spacecraft, and the Mission Control Center (MCC). The robotic agents might need to be controlled by a human operator, depending on their capabilities per action, as well as the desired specifications of the function allocation.

The IV astronaut has access to a datalink with the humanoid robot, a datalink with RMS, and a radio-connection with the EV astronaut and the MCC. The EV astronaut has access to the same radio-link, and can additionally directly interact with the humanoid when they are working shoulder-to-shoulder. The RMS is in connection with the IV astronaut, as well as with the humanoid. Finally, MCC can talk directly to the both astronauts over the radio link. Depending on which agents are involved in the function allocation and the required information transfer, communication needs to occur over these channels.

The taskwork for the scenario includes actions associated with overhead tasks, locomotion outside the vehicle, inspection, replacement of a broken panel, and tool handling.

We tested three potential function allocations, each with different distributions of the work and different requirements for control modes and monitoring and confirmation needs. FA1 is a function allocation in which the EV astronaut directly controls a humanoid robot, the two of them working shoulder-to-shoulder to conduct the inspection and replacement of broken panels. This human-robot team is assisted by the RMS, which is being manually controlled by the IV astronaut.

For FA2, the humanoid takes over the tasks of the EV astronaut. Most actions need humans to tele-operate the robots, either through direct tele-operation or command sequencing. MCC directs the humanoid, whereas the IV astronaut can directly operate the RMS. Furthermore, we have denoted five critical actions that need to be confirmed instead of monitored: the “apply inspection tools” and the four actions associated with the replacement of a broken panel.

Finally, FA3 is a more notional function allocation, in which it is assumed that the humanoid is capable of executing tasks more independently from human operators. Thus, two humanoids, one performing the inspection and one doing the replacement of bad panels, are intermittently commanded by the IV astronaut and MCC, respectively. Humanoid I is continuously being monitored by the IV astronaut. MCC is responsible for Humanoid II, but because there might not be real-time data available for MCC, all actions of the Humanoid II are confirmed as opposed to monitored.

Results from FA1: Here we have an EV astronaut who is working shoulder-to-shoulder with a humanoid. The astronaut at times needs to directly control this humanoid to execute inspection tasks. Replacement of broken panels is conducted by the astronaut, in collaboration with the RMS, which is being controlled and monitored by an IV astronaut. The time traces show that the RMS, IV astronaut, and humanoid often need to wait for the EV astronaut to complete his/her task before they can continue their own operations.

Results from FA2: It shows MCC has a high taskload in controlling, monitoring, and confirming the operations of the humanoid. Furthermore, the IV astronaut has long periods of idling time and only occasionally needs to control and confirm the operations of the RMS. We additionally observe that the humanoid needs to occasionally wait for the MCC to provide commands. Likewise, the RMS is sometimes idling while the intra-vehicular astronaut is confirming the correct execution of the previous action.

Results from FA3: Here, two humanoids are together performing the mission, and are assumed to only occasionally need commands from human operators. The IV astronaut is responsible for the actions of Humanoid I and thus has a continuous monitoring load. MCC is confirming every action of Humanoid II, and together with the waiting for commands, this causes long idling times. Additionally, because the replacement of panels is now only executed by a single agent, the mission duration is extended.

The total number of information transfer requirements increases when moving from FA1 to FA3. The use of different communication channels is seen for the direct interaction between the EV astronaut and the humanoid, the Datalink1 for the humanoid, and Datalink2 for RMS. For FA1 we see some of the interaction associated with control and monitoring takes place through direct interaction between EV astronaut and the humanoid. For FA3, all communication takes place over the datalink channel with the two humanoids on it.

The proposed research addresses three main research questions: First, how should roles and responsibilities be optimally assigned to robots and humans based on a combination of task demands, robotic capabilities and available crew resources, with special attention to the capabilities inherent to classes of robots? Second, what is the human-robot system performance trade-space that serves as the basis for the allocation? Third, how can this function allocation method be validated as creating appropriate function allocation for both nominal and off-nominal operations?

We propose a three year effort to address these questions. In the first year we propose to model the function allocation design space that exists between humans and robots in deep space exploration missions. We will use a computational framework called Work Models that Compute (WMC), which allows us to model dynamical systems (such as space vehicles and robots), automated systems (such as the automated rendezvous and docking system) and human agents working together to achieve common goals. WMC was custom designed to model function allocation and to measure eight metrics of function allocation previously established by the proposers. In the second year we will explore the design space, deeply investigating each metric such as taskload, authority- responsibility mismatch, coherency, etc., while beginning the validation process through the use of human-in-the-loop experiments with simulated robots. In the final year we will move from exploring each metric individually to looking at their combined effects as we vary the design space constraints, the tasks, crew stress levels, and function allocation options. We will continue our validation efforts using human-in-the-loop experiments using a combination of simulated robots and/or real robots. These experiments will systematically explore a large number of conditions such that they serve not only to demonstrate the function allocation chosen by the method, but also to validate the method.