NOTE: Period of performance changed to 6/1/2015-5/31/2017 per NSBRI (original period of performance was 5/31/15-5/30/17)--Ed., 6/25/15

During the second year, we conducted another experiment investigating the effects of SD on human performance during robotic supervisory control. We used a similar inpatient protocol in Experiment 2 as used in Experiment 1. We reused two task types developed in the first experiment: i) Efficiency Tasks - discover and use more efficient task order; and ii) Constraint Tasks - recall and use equipment status changes. Additionally, two countermeasures were developed to mitigate the effects of SD on human performance when supervising robots, and evaluated in the second experiment. The task planning countermeasure software was developed to aid short-term memory when sleep-deprived by allowing users to build and execute new sequences of automation tasks on-the-fly. This countermeasure should improve task accuracy and timing. The adaptive alerting countermeasure was developed to remind users to perform infrequent but important supervisory tasks when sleep-deprived. This countermeasure should reduce the time automation waits for human intervention and thus reduce overall task timing.

During Experiment 1 we experienced data loss due to unexpected variation in the way subjects interpreted and responded to instructions. Tasks needed to be sufficiently constrained that successful performance would likely generate a predictable stream of events, thus making scoring of the behavior streams feasible and results across subjects comparable. We redesigned core tasks from Experiment 1 for Experiment 2 and were overall successful in providing the intended, complex task sequences. The core tasks were executed as planned in Experiment 1 with essentially very low data loss. Data were collected from 8 participants in Experiment 2. We had expected 15 participants for this experiment. The number of participants was less than expected because a delayed project start reduced the amount of time we had to perform Experiment 2 and analyze data from it. We also experienced difficulty in finding subjects with the necessary education and computer literacy to serve as rough astronaut surrogates.

We report on results from the core tasks and from the tasks assessing use of the task planning countermeasure. The factors for core tasks were sleep status (rested versus sleep deprived) and repetition (first versus second exposure to the type of task), with assignment of rested-first or rested-second counterbalanced across users. We have an n of 8 with large individual differences, hence low sensitivity, and we have done no inferential statics. Rather, we looked for suggestive patterns in performance. For Core tasks, we looked for patterns suggesting impact of SD and repetition on flexible reasoning. For task planning countermeasure assessment, we looked for patterns concerning how planner use influenced performance. We did not analyze data from use of the adaptive alerting countermeasure, because of reduced time available to conduct Experiment 2 and analyze data from it.

Our ability to detect effects of SD or Trial is limited by small n and large individual differences. We found some evidence of effects of SD on performance for one core task. The most suggestive pattern was found on the accuracy of performing a Constraint Task. For this task, the subject should devise a sequence of tasks that connect batteries to solar panels while complying with a new safety constraint designating how many batteries can be connected to a solar panel. Four of the eight users showed sensitivity to the constraint at some point, two of these four respected the constraint on both trials. Two users, however, did well on the first trial (which was also rested) but did not respect the constraint later when they were sleep deprived. That is, they showed a decrement in performance when sleep deprived, even though that was their second exposure to this constraint task. We consider this pattern of performance to be consistent with the hypothesis that SD reduces the chance a user will correctly integrate information that requires modifying the way a user is doing the task. This is an intriguing pattern though our n is small, suggesting that further investigation may be warranted.

We assessed use of the task planning countermeasure by requiring users to work on two tasks at once, the context where we expected a benefit from being able to automatically run a sequence of procedures. The primary task was executing a series of robot procedures automatically and the secondary task was manually identifying the most efficient paths for a Rover using a diagram of possible paths. The primary dependent variables were how much work could be done on these tasks in 12 minutes. We looked at data from three occurrences of this task. Summarizing across these occurrences, users who were re-using their plans, or had planned outside the time window had some advantage in number of tasks completed over users who planned within the time window. Those in the Plan re-use task also had the lowest number of errors on the secondary task. For all tasks considered, slips are low, with little impact of plan use. Slips were defined as errors due to insertion of extra tasks, skipped tasks, unnecessary task repetition, or mis-ordered tasks. There may be a trend for slips to increase over the course of a session. A large amount of data was collected in this project, but very limited time was available to analyze these data. Further analyses of these data are merited.

Task 2. Developed adaptive automation countermeasure software: We developed technology for automation countermeasures are intended to mitigate SD effects on human performance when supervising robots. The adaptive alerting countermeasure intends to aid recall to perform supervisory tasks when sleep-deprived by alerting the user when the robot needs human intervention. The adaptive planning countermeasure intends to aid short-term memory when executing complex sequences of procedures by supporting users in planning robot activity sequences that can be built on the fly and executed automatically.

Task 3. Designed supervisory control tasks for experiment 2: We revised the core tasks from Experiment 1 intended to make it difficult for the sleep-deprived participant i) to self-regulate switching among tasks in accord with allowed choices, i.e., perseveration; and ii) to change strategy to suit conditions or to inhibit a dominant strategy, i.e., mental set rigidity. We added new tasks to investigate how the automation countermeasures affected performance under sleep deprived conditions. Each participant used the countermeasure tools one of two sessions, half in the earlier and half in the later session.

Task 4. Conduct Experiment 2 at Brigham and Women's Hospital (BWH): Participants were selected based on education or work experience in engineering or science. They received training one week prior to the experiment. The inpatient protocol lasts four calendar days, with five 2-hour sessions using the robotic testbed. Additionally, the participant takes Psychomotor Vigilance Test (PVT), and saliva and urine are collected.

Task 5. Analyzed data from Experiment 2: We analyzed performance data collected for the core tasks requiring constraint awareness and compliance or task efficiency, and for the tasks assessing use of the automation countermeasure tools.

We have an n of 8 with large individual differences, hence low sensitivity, and we did no inferential statics. Rather, we looked for suggestive patterns in performance. For core tasks, we looked for patterns suggesting impact of sleep deprivation (and also repetition) on flexible reasoning. For task countermeasure assessment, we looked for patterns concerning how use of the planner influenced performance. Overall, we found very little evidence of effects from sleep status. The most suggestive pattern was found on one of the core constraint tasks.

NOTE: Period of performance changed to 6/1/2015-5/31/2017 per NSBRI (original period of performance was 5/31/15-5/30/17)--Ed., 6/25/15

We aim to develop and evaluate adaptive automation countermeasures to mitigate human performance decrements caused by the stress of sleep deprivation (SD) when supervising robots. We focus on problem solving and decision making tasks that are likely to require active human intervention and be impacted by SD. We also aim to develop methods to measure trust in automation (particularly, modeling the pattern of reliance on automation), to foster appropriate use of automation notices and suggestions, and to track how trust is related to performance and performance degradation.

2. Key findings

Our aim is to mitigate effects of SD on crewmembers managing complex operations such as supervisory control of robots and automation. To do this, we need to understand what cognitive processes are highly impacted by SD and what cognitive processes are likely to limit performance in this type of operational context. A large and consistent body of evidence shows that tasks primarily drawing on sustained attention and processing speed in routine tasks are highly vulnerable to SD. However, we expect that higher-level cognitive processes, rather than basic attention, are and should be primarily implicated in future exploration operations. We expect, crewmember cognition may more usefully be spent on decision making, problem solving, or managing the unexpected. Consequently, Dr. Billman performed a literature review during the first year of this project on the effects of SD on higher-level cognitive processes and tasks. The results of this literature review were used to define the decision-making tasks performed by subjects using the robotic supervisory control testbed during the first experiment at Brigham and Women's Hospital (BWH).

The first experiment uses the supervisory control testbed to evaluate human-robot interaction in a human exploration mission scenario. In this scenario, resources have been pre-positioned on the planetary surface prior to the arrival of human astronauts. These resources include a habitat with systems powered from batteries. These batteries are charged from solar panels. A humanoid robot R2, developed at NASA Johnson Space Center (JSC), was deployed with these resources. R2 connects batteries to solar panels or to power distribution grids for the habitat. Astronauts in transit to the planet plan and direct the activities of R2 and the smart habitat systems to make the habitat habitable by the time they arrive. Ms. Schreckenghost leads the TRACLabs' development of a testbed for supervisory control of robots to support this scenario. We use the NASA Gazebo simulation of the R2 robot, provided by the Co-Investigator (CoI) Dr. Hambuchen. The Gazebo simulation was modified to include a power generation system based on solar panels that charge batteries, and a power distribution system that takes battery power and distributes it to systems in the habitat. The robot turns knobs on a control panel outside the habitat to connect batteries to solar panels or to power distribution systems. The Gazebo simulation was integrated with TRACLabs's PRIDE procedure automation software to provide supervisory control of the robot. We developed procedures for autonomous execution by the R2 robot that i) connect batteries to solar panels to recharge them, ii) connect batteries to power distribution modules to provide power to the habitat, or iii) disconnect batteries to reserve power. The participant's role is to decide which batteries, solar panels, and power distribution modules should be connected and to plan the sequence in which these components are connected.

Dr. Billman leads the San Jose State University (SJSU) effort to develop and test the tasks for the participant to perform during Experiment 1 using the testbed for supervisory control. These tasks were designed to create situations where the expected cognitive deficits associated with SD are likely to occur, based on the literature review. Experiment 1 will assess the impact of SD on higher-level cognitive processes and human trust when directing the R2 robot in the supervisory control testbed. This experiment is being conducted at BWH under the supervision of Dr. Klerman. Experimentation begins in May 2016.

3. Impact of key findings

The literature review performed during this year found that the impact of SD on higher-level cognitive processes and tasks is less studied, less consistently found, and harder to compare across studies. Consequently, an expected contribution of our research is improved knowledge about the nature of performance and performance degradation on supervisory tasks when an individual is sleep-deprived. Based on this review we also expect sleep-deprived participants to experience impairments in divergent or flexible thought, which can affect the ability to build and adapt task plans for the robot. Therefore, we have broadened our investigation of adaptive automation countermeasures to include both adaptive alerting identified in the original proposal and technology for adaptive planning of robot tasks.

4. Proposed research plan for next year

During Year 1 Ms. Schreckenghost developed initial designs for two types of automation technology countermeasure to these effects – technology for adaptive alerting and technology for adaptive task planning. We will use the results of the first experiment available in September 2016 to inform which of these countermeasures are most likely to mitigate cognitive effects of SD during supervisory control. We then will implement and evaluate technology for the selected countermeasure design in the second experiment to be conducted in Year 2. The Year 2 experiment also will develop objective methods for measuring human trust in these automation countermeasures and investigate the effects of automation trust on task outcome when humans are sleep-deprived. We will conduct Experiment 2 at BWH in early 2017. All work will be complete by May 31, 2017.

Task 2. Developed a testbed for supervisory control of robots (Ms. Schreckenghost). We modified the NASA R2 robot simulation, from Co-I Dr. Kimberly Hambuchen, to include a power system with solar panels to charge batteries, and power units called DDCUs to distribute battery power to habitat systems. The robot turns knobs on a control panel to connect batteries to solar panels or DDCUs. This simulation was integrated with TRACLabs's PRIDE procedure automation software to provide procedures for autonomous execution by R2 to i) connect batteries to solar panels to recharge them, ii) connect batteries to power distribution, or iii) disconnect batteries. The participant decides which components should be connected and plans the connection sequences.

Task 3. Designed supervisory control tasks for experiment 1 (Dr. Billman and Ms. Schreckenghost). These tasks were designed to make it difficult for the sleep-deprived participant i) to self-regulate switching among tasks in accord with allowed choices i.e., perseveration; and ii) to change strategy to suit conditions or to inhibit a dominant strategy, i.e., mental set rigidity.

Task 4. Conduct Experiment 1 at BWH (Dr. Klerman). Experiment 1 is ongoing at the time of this report. Participants are selected based on education or work experience in engineering or science. They receive training one week prior to the experiment. The inpatient protocol lasts four calendar days, with five 2-hour sessions using the robotic testbed. Additionally, the participant takes PVT, and saliva and urine are collected.

Task 5. Develop designs for technology countermeasures (Ms. Schreckenghost). We identify adaptive alerting as a promising automation countermeasure to the cognitive effects of SD. Adaptive alerting software detects situations with increased potential for errors around attentional lapses and notifies users about them to mitigate their impact. The literature review suggests a second candidate automation countermeasure that seeks to mitigate the anticipated effect of SD on divergent or novel thinking, particularly associated with task planning. For this research, automated task planning refers to selecting and ordering robotic tasks to be efficient while complying with operational constraints.

Sleep deprivation degrades performance in several ways. Of particular concern is degradation of tasks that cannot be automated--specifically requiring higher-level reasoning and particularly for novel situations requiring divergent, or creative thinking, and situations when the usual solution should not be applied. While research on these tasks is limited, evidence suggests it is precisely these types of higher-level cognitive tasks that may be seriously impacted by SD. Our proposed research will quantify SD-associated impairments in human regulatory processes that provide flexibility, intentional search for relevant information, weighing and integrating information, and assessing merit of response.

Sleep deprivation also may cause users to become complacent and inappropriately rely on automation. Thus it is important to assess the effects of SD on user trust in automation and what effect proposed countermeasures have on automation trust. Appropriate level of trust by an operator relies on assessing the relative competence of automation versus self in carrying out the specific task including indirect factors such as operator workload and demands of other tasks. Multiple factors therefore influence what appropriate trust would be for a specific context.

Recent research on trust of robotic autonomy identifies a range of contributors to trust, including robot reliability and performance, robot competency, perceived risk, and user self-assessment of competency. Our proposed research will extend and contribute to this research by investigating the relationship between trust of robot autonomy and patterns of reliance on autonomy under stressed conditions due to insufficient sleep.

We propose to quantify the effect of sleep deprivation on multiple tasks involving human-automation-robotic (HAR) systems. We will develop measures and interventions for sleep deprived users when supervising robots. We will design and prototype appropriate adaptive technology countermeasures. We will conduct experiments with users performing supervisory control of simulated robots in space exploration--similar scenarios to quantify sleep-deprivation-related impairment, trust in automation, and effectiveness of adaptive countermeasure technology.

The result of this project will be guidance and technology prototypes representing effective human-automation-robotic systems in operational environments where humans and robots perform distributed, concurrent teamwork under situational stressors like sleep deprivation. These guidelines will be relevant to future human space flight missions that take astronauts deeper into space and require increased crew independence from Earth (crew autonomy). They enable greater reliance on robotic and spacecraft automation by improving understanding of the effects of stress from sleep deprivation on human performance during supervisory control of robots and countermeasures for these effects.

These guidelines address Human Research Program (HRP) Risk of Inadequate Design of Human and Automation/Robotic Integration, Gap SHFE-HARI-02--'We need to develop design guidelines for effective human-automation-robotic systems in operational environments that may include distributed, non-colocated adaptive mixed-agent teams with variable transmission latencies.'