FINAL REPORTING--FEBRUARY 2015
1. Project aims
This project investigates the hypothesis that selecting units of work to automate based on human procedures will improve human-automation designs. The structured actions sets within procedures, such as steps, can be used as the basis for meaningful, sharable organization of what is automated and how information should be passed between humans and automation. Investigators hypothesize that human-automation interaction organized this way will provide several benefits, relative both to manual operations and to less user-centric automation, including effective execution and more adaptable use with flexible levels of automation. To test this hypothesis, we are (i) developing strategies for using procedures to identify units of work for adjustable automation, (ii) building a test environment with software for manual or partially automated execution of procedures, and (iii) using this environment to evaluate the effectiveness of alternative strategies for automating procedures.
2. Key findings
The analysis was completed on data from the Year 2 study investigating whether procedure automation could be used effectively in situations where procedure preconditions do not hold (a type of problem-solving). Recovering from or preventing failures threatened by these unmet conditions can be difficult, and software designed for fluent execution of nominal tasks may not facilitate problem solving. However, all 12 participants solved the problem posed by an unmet procedure precondition in two sessions. Specifically, they found a procedure that could make the required condition true then resumed the original procedure. They did this with no prior encounter or training for problem solving when conditions assumed in a procedure are not met. By the second session 11 of 12 participants were able to prevent rather than recover from the failure [Billman, et al., 2015].
A study was completed in the Human Exploration Research Analog (HERA) Campaign 2 evaluating PRIDE electronic procedure software. 16 subjects participated, 4 per mission. Each subject was trained for 2 hours prior to the 2-week mission, then performed seven 1-hour sessions during the mission. We embedded two study-designs within the larger 7-session activities. The Step-List Study tested the hypothesis that performance using procedures with actions grouped into functional units of work called steps (Step Format) would be better than using procedures without this grouping (List Format). The Problem-Solving Study investigated 1) the effects of repetition on problem solving, 2) the effects of a secondary manual task done in conjunction with the primary habitat task where problem occurs, and 3) the effects of encouragement to plan in advance on subsequent execution. Several exploratory tasks were distributed over the mission as well, including the Automation-Manual Investigation comparing human performance with an electronic procedure to the same procedure done using as much automation as possible.
HERA Step-List Study: We found significant effects of format on both markup errors and mode-execution errors given a correct markup. This suggests the Step Format made it somewhat easier to assign execution mode (Manual or Automated) and execute from those assignments; there was no influence of Format on time measures. These results suggest that the Step format reduces errors in setting function allocation relative to List format and they merit further investigation.
HERA Problem Solving Study. Behavior across users and days for the HERA experiment is both less successful in accomplishing the task-specified goals and more variable in the strategies adopted, than in the earlier Year 2 study of problem-solving behavior. The most striking result is the degree of variation in strategy and outcome across this user group. Some users were successful in using electronic procedure software to recover from an unexpected situation where the procedure could not be executed directly and normally. Other users were not, and their difficulties included an assumption that automation would prevent any harmful action, reluctance to do anything not directly specified in their daily tasks, and uncertainty about what might be causing the specific value or its importance. There are many possible contributors to the reduced likelihood of accomplishing the task goal in HERA over that in the Year 2 study. The task was structurally more complex in HERA, because the user needed to identify and execute the recovery procedures for two habitat systems instead of one. Additionally, the absence of experimenter available for the Year 2 Study, and differences in training may have increased the difficulty of the problem-solving tasks in HERA.
HERA Automation-Manual Investigation. The kinds of errors that users perform with electronic procedures are described. Our error analyses suggest that the mode differences (manual versus automated), between two low error rates, would be expected again in similar conditions. Thus executing with maximal automation may reduce errors relative to manual execution. Executing procedures with maximal automation also may reduce execution timing relative to executing the same procedures manual only. Small reductions in execution timing with automation were observed. Faster execution time with automation versus manual only modes should be investigated further [Schreckenghost, et al, 2015].
3. Impact of key findings: The results from the HERA and prior studies conducted for this project have informed the identification of candidate design guidance and factors for effective task allocation strategies within our procedure-automation approach. This design guidance directly addresses the Human Research Program (HRP) Risk of Inadequate Design of Human and Automation/Robotic Integration, Gap SHFE-HARI-01. The design guidance and factors are reported in the Main Findings section.
4. Research plan next year: None; this project ended in September 2015.
ANNUAL REPORTING IN OCTOBER 2014
1. Project aims
We investigate the hypothesis that selecting units of work to automate based on human procedures will improve human-automation designs. The structured actions sets within procedures, such as steps, can be used as the basis for meaningful, sharable organization of what is automated and how information should be passed between humans and automation. We hypothesize that human-automation interaction organized this way will provide several benefits, relative both to manual operations and to less user-centric automation, including effective execution and more adaptable use with flexible levels of automation. To test this hypothesis we will 1) develop strategies for using procedures to identify units of work for adjustable automation, 2) build a test environment with software for manual or partially automated execution of procedures, and 3) use this environment to evaluate the effectiveness of alternative strategies for automating procedures.
2. Key findings
We analyzed data collected in Year 1 comparing manual execution of the new procedure system (PRIDE Interface) to a system analogous to procedures for ISS (Legacy Interface). We assessed whether manual performance with PRIDE would be as good or better than with the legacy system. This lays the foundation for integrating automated execution into the flow of procedures designed for humans. We found speed and accuracy of manual procedure execution was better using PRIDE interface over Legacy interface. When using PRIDE interface, less than 3% of procedures had errors. For Legacy Interface, 33% of procedures had errors. Using PRIDE interface took less time than Legacy interface for all trials. We also analyzed Year 1 data where participants used PRIDE procedure automation. These results informed a redesign of the PRIDE software for procedure automation that is being evaluated in Year 2 experiments. Using the original PRIDE interface it was hard to predict what can be automated, what should be automated, and the consequences of attempting to automate elements that cannot or should not be automated. Methods to identify what cannot or should not be automated were an important direction for development. Support for constructing and checking an advance automation plan also was identified as valuable. Automation plans may be more complex than was easily supported by the original approach of automating between the focus bar and a breakpoint. Users may find it useful to specify spans of procedure lines to automate. Since steps are a central unit of work in procedures, supporting step-level automation control may be valuable. Procedure technology was demonstrated to 4 astronauts on the User Panel. Crew Factors scores were all high. Systems score were high or medium.
3. Impact of key findings
Our identification of strategies for allocating tasks to automation and techniques for assessing strategy effectiveness directly addresses the HRP Risk of Inadequate Design of Human and Automation/Robotic Integration, Gap SHFE-HARI-01 (We need to evaluate, develop, and validate methods and guidelines for identifying human-automation/robot task information needs, function allocation, and team composition for future long duration, long distance space missions). We observed 3 models of automation use when performing procedures, which varied according to the style and effort spent planning what actions to automate.
1. Minimal planners spent no time planning automation and relied on the software to stop automatic execution when it reached a non-automatable instruction. This model incurs no cost to plan automation, but is the least operationally flexible of these strategies.
2. Incremental planners automated 1 span of actions at a time, interleaving automation planning with procedure execution. This model is suited to simple automation plans with few handovers, or situations where the decision of which tasks to automate can be altered by the effects of intervening procedure actions.
3. Predictive planners built a plan for automating the entire procedure prior to taking any action in the procedure. This model is suited to situations where deciding what to automate is based on information known prior to execution. This predictability means that it may be useful to save automation plans for reuse when executing the procedure later. Observation suggested that human performance is impacted by the frequency and number of handovers between automation and manual execution, and the time between these handovers. We are investigating techniques to characterize use of handovers in a task allocation strategy to provide insight into its effectiveness. Partial automation of procedures can make it possible for the user to perform secondary tasks during periods of automation. We are investigating techniques to assess how well an allocation strategy supports multitasking, including measuring supervision costs to assess how independent automation is from its supervisor. For an allocation strategy to support effective multitasking, these costs should be low relative to time made available for a secondary task.
4. Research plan next year
Our project is being considered for 2 HERA FY15 missions. These experiments will evaluate partial automation strategies developed in Year 2 during longer duration experiments under more flight-like conditions with participants more similar to astronauts. Each participant will perform procedures both manually and partially automated during 7 one-hour sessions per mission. Performance measures include completion time, errors, and workload. Performance under both conditions will be compared, within subject. We will update the procedure software for these experiments, based on Year 2 findings. We will document our findings about using units of work in procedures as the basis of automation, including strategies for allocating tasks to automation and techniques for evaluating strategy effectiveness. |