Menu

 

Task Book: Biological & Physical Sciences Division and Human Research Program
Advanced Search     

Project Title:  Automation in Procedures: Guidelines for Allocating Tasks for Performance Reduce
Fiscal Year: FY 2015 
Division: Human Research 
Research Discipline/Element:
HRP SHFH:Space Human Factors & Habitability (archival in 2017)
Start Date: 10/01/2012  
End Date: 09/30/2015  
Task Last Updated: 02/24/2016 
Download report in PDF pdf
Principal Investigator/Affiliation:   Schreckenghost, Debra  M.E.E. / TRACLabs, Inc. 
Address:  16969 N Texas Ave 
Suite 300 
Webster , TX 77598-4085 
Email: ghost@ieee.org 
Phone: 281-461-7886  
Congressional District: 22 
Web:  
Organization Type: INDUSTRY 
Organization Name: TRACLabs, Inc. 
Joint Agency:  
Comments:  
Co-Investigator(s)
Affiliation: 
Billman, Dorrit  San Diego State University 
Project Information: Grant/Contract No. NCC 9-58-HFP02803 
Responsible Center: NSBRI 
Grant Monitor:  
Center Contact:   
Solicitation / Funding Source: 2011 Crew Health NNJ11ZSA002NA 
Grant/Contract No.: NCC 9-58-HFP02803 
Project Type: GROUND 
Flight Program:  
TechPort: No 
No. of Post Docs:
No. of PhD Candidates:
No. of Master's Candidates:
No. of Bachelor's Candidates:
No. of PhD Degrees:
No. of Master's Degrees:
No. of Bachelor's Degrees:
Human Research Program Elements: (1) SHFH:Space Human Factors & Habitability (archival in 2017)
Human Research Program Risks: (1) HARI:Risk of Inadequate Design of Human and Automation/Robotic Integration
Human Research Program Gaps: (1) SHFE-HARI-01:We need to evaluate, develop, and validate methods and guidelines for identifying human-automation/robot task information needs, function allocation, and team composition for future long duration, long distance space missions (IRP Rev G name change to HARI-01 from previous designation SHFE-HARI-01, which was IRP Rev F) (Previously: What guidelines and tools can we develop to enable system designers and mission planners to conduct systematic task/needs analyses at the appropriate level of detail to allocate work among appropriate agents (human and automation)?
Task Description: FINAL REPORTING--FEBRUARY 2015

1. Project aims

This project investigates the hypothesis that selecting units of work to automate based on human procedures will improve human-automation designs. The structured actions sets within procedures, such as steps, can be used as the basis for meaningful, sharable organization of what is automated and how information should be passed between humans and automation. Investigators hypothesize that human-automation interaction organized this way will provide several benefits, relative both to manual operations and to less user-centric automation, including effective execution and more adaptable use with flexible levels of automation. To test this hypothesis, we are (i) developing strategies for using procedures to identify units of work for adjustable automation, (ii) building a test environment with software for manual or partially automated execution of procedures, and (iii) using this environment to evaluate the effectiveness of alternative strategies for automating procedures.

2. Key findings

The analysis was completed on data from the Year 2 study investigating whether procedure automation could be used effectively in situations where procedure preconditions do not hold (a type of problem-solving). Recovering from or preventing failures threatened by these unmet conditions can be difficult, and software designed for fluent execution of nominal tasks may not facilitate problem solving. However, all 12 participants solved the problem posed by an unmet procedure precondition in two sessions. Specifically, they found a procedure that could make the required condition true then resumed the original procedure. They did this with no prior encounter or training for problem solving when conditions assumed in a procedure are not met. By the second session 11 of 12 participants were able to prevent rather than recover from the failure [Billman, et al., 2015].

A study was completed in the Human Exploration Research Analog (HERA) Campaign 2 evaluating PRIDE electronic procedure software. 16 subjects participated, 4 per mission. Each subject was trained for 2 hours prior to the 2-week mission, then performed seven 1-hour sessions during the mission. We embedded two study-designs within the larger 7-session activities. The Step-List Study tested the hypothesis that performance using procedures with actions grouped into functional units of work called steps (Step Format) would be better than using procedures without this grouping (List Format). The Problem-Solving Study investigated 1) the effects of repetition on problem solving, 2) the effects of a secondary manual task done in conjunction with the primary habitat task where problem occurs, and 3) the effects of encouragement to plan in advance on subsequent execution. Several exploratory tasks were distributed over the mission as well, including the Automation-Manual Investigation comparing human performance with an electronic procedure to the same procedure done using as much automation as possible.

HERA Step-List Study: We found significant effects of format on both markup errors and mode-execution errors given a correct markup. This suggests the Step Format made it somewhat easier to assign execution mode (Manual or Automated) and execute from those assignments; there was no influence of Format on time measures. These results suggest that the Step format reduces errors in setting function allocation relative to List format and they merit further investigation.

HERA Problem Solving Study. Behavior across users and days for the HERA experiment is both less successful in accomplishing the task-specified goals and more variable in the strategies adopted, than in the earlier Year 2 study of problem-solving behavior. The most striking result is the degree of variation in strategy and outcome across this user group. Some users were successful in using electronic procedure software to recover from an unexpected situation where the procedure could not be executed directly and normally. Other users were not, and their difficulties included an assumption that automation would prevent any harmful action, reluctance to do anything not directly specified in their daily tasks, and uncertainty about what might be causing the specific value or its importance. There are many possible contributors to the reduced likelihood of accomplishing the task goal in HERA over that in the Year 2 study. The task was structurally more complex in HERA, because the user needed to identify and execute the recovery procedures for two habitat systems instead of one. Additionally, the absence of experimenter available for the Year 2 Study, and differences in training may have increased the difficulty of the problem-solving tasks in HERA.

HERA Automation-Manual Investigation. The kinds of errors that users perform with electronic procedures are described. Our error analyses suggest that the mode differences (manual versus automated), between two low error rates, would be expected again in similar conditions. Thus executing with maximal automation may reduce errors relative to manual execution. Executing procedures with maximal automation also may reduce execution timing relative to executing the same procedures manual only. Small reductions in execution timing with automation were observed. Faster execution time with automation versus manual only modes should be investigated further [Schreckenghost, et al, 2015].

3. Impact of key findings: The results from the HERA and prior studies conducted for this project have informed the identification of candidate design guidance and factors for effective task allocation strategies within our procedure-automation approach. This design guidance directly addresses the Human Research Program (HRP) Risk of Inadequate Design of Human and Automation/Robotic Integration, Gap SHFE-HARI-01. The design guidance and factors are reported in the Main Findings section.

4. Research plan next year: None; this project ended in September 2015.

ANNUAL REPORTING IN OCTOBER 2014

1. Project aims

We investigate the hypothesis that selecting units of work to automate based on human procedures will improve human-automation designs. The structured actions sets within procedures, such as steps, can be used as the basis for meaningful, sharable organization of what is automated and how information should be passed between humans and automation. We hypothesize that human-automation interaction organized this way will provide several benefits, relative both to manual operations and to less user-centric automation, including effective execution and more adaptable use with flexible levels of automation. To test this hypothesis we will 1) develop strategies for using procedures to identify units of work for adjustable automation, 2) build a test environment with software for manual or partially automated execution of procedures, and 3) use this environment to evaluate the effectiveness of alternative strategies for automating procedures.

2. Key findings

We analyzed data collected in Year 1 comparing manual execution of the new procedure system (PRIDE Interface) to a system analogous to procedures for ISS (Legacy Interface). We assessed whether manual performance with PRIDE would be as good or better than with the legacy system. This lays the foundation for integrating automated execution into the flow of procedures designed for humans. We found speed and accuracy of manual procedure execution was better using PRIDE interface over Legacy interface. When using PRIDE interface, less than 3% of procedures had errors. For Legacy Interface, 33% of procedures had errors. Using PRIDE interface took less time than Legacy interface for all trials. We also analyzed Year 1 data where participants used PRIDE procedure automation. These results informed a redesign of the PRIDE software for procedure automation that is being evaluated in Year 2 experiments. Using the original PRIDE interface it was hard to predict what can be automated, what should be automated, and the consequences of attempting to automate elements that cannot or should not be automated. Methods to identify what cannot or should not be automated were an important direction for development. Support for constructing and checking an advance automation plan also was identified as valuable. Automation plans may be more complex than was easily supported by the original approach of automating between the focus bar and a breakpoint. Users may find it useful to specify spans of procedure lines to automate. Since steps are a central unit of work in procedures, supporting step-level automation control may be valuable. Procedure technology was demonstrated to 4 astronauts on the User Panel. Crew Factors scores were all high. Systems score were high or medium.

3. Impact of key findings

Our identification of strategies for allocating tasks to automation and techniques for assessing strategy effectiveness directly addresses the HRP Risk of Inadequate Design of Human and Automation/Robotic Integration, Gap SHFE-HARI-01 (We need to evaluate, develop, and validate methods and guidelines for identifying human-automation/robot task information needs, function allocation, and team composition for future long duration, long distance space missions). We observed 3 models of automation use when performing procedures, which varied according to the style and effort spent planning what actions to automate.

1. Minimal planners spent no time planning automation and relied on the software to stop automatic execution when it reached a non-automatable instruction. This model incurs no cost to plan automation, but is the least operationally flexible of these strategies.

2. Incremental planners automated 1 span of actions at a time, interleaving automation planning with procedure execution. This model is suited to simple automation plans with few handovers, or situations where the decision of which tasks to automate can be altered by the effects of intervening procedure actions.

3. Predictive planners built a plan for automating the entire procedure prior to taking any action in the procedure. This model is suited to situations where deciding what to automate is based on information known prior to execution. This predictability means that it may be useful to save automation plans for reuse when executing the procedure later. Observation suggested that human performance is impacted by the frequency and number of handovers between automation and manual execution, and the time between these handovers. We are investigating techniques to characterize use of handovers in a task allocation strategy to provide insight into its effectiveness. Partial automation of procedures can make it possible for the user to perform secondary tasks during periods of automation. We are investigating techniques to assess how well an allocation strategy supports multitasking, including measuring supervision costs to assess how independent automation is from its supervisor. For an allocation strategy to support effective multitasking, these costs should be low relative to time made available for a secondary task.

4. Research plan next year

Our project is being considered for 2 HERA FY15 missions. These experiments will evaluate partial automation strategies developed in Year 2 during longer duration experiments under more flight-like conditions with participants more similar to astronauts. Each participant will perform procedures both manually and partially automated during 7 one-hour sessions per mission. Performance measures include completion time, errors, and workload. Performance under both conditions will be compared, within subject. We will update the procedure software for these experiments, based on Year 2 findings. We will document our findings about using units of work in procedures as the basis of automation, including strategies for allocating tasks to automation and techniques for evaluating strategy effectiveness.

Research Impact/Earth Benefits: FINAL REPORTING--FEBRUARY 2015

As crewed missions move deeper into space and communication latency increases, astronauts will be unable to depend on real-time support from flight controllers, such as performing or advising on procedures as done for the International Space Station (ISS). Astronauts will have a longer time lag between task training and execution while also needing to perform more, and more diverse, tasks. These can increase astronaut workload, decrease efficiency, and increase the risk of sub-optimal task execution. A critical resource for meeting these challenges is greater reliance on spacecraft automation. Today astronauts and flight controllers carry out tasks by following written procedures. As a result approaches that integrate automation with procedures are a good fit to NASA's concept of operations. If human-automation integration (HAI) for such procedure automation is not appropriately designed, however, increased automation may contribute to rather than alleviate these challenges. The strategies for allocating tasks to automation resulting from this project is of direct benefit to current and future NASA human space exploration missions. Since the Deep Water Horizon accident, the oil and gas industry has become increasingly interested in technologies to improve the safety and traceability of drilling operations. Similar to NASA, these industries use Standard Operating Procedures (SOPs) as the basis of their operations. TRACLabs is currently developing and testing procedure automation technology for well-site automation to be used by an oil and gas service company. This software incorporates the automation functions and interface design modified and evaluated under the NSBRI project. This software will be deployed for testing at a drilling site. The successful completion of this test should result in the gradual phasing of the procedure automation technology into operational use by the service company. The Department of Defense (DoD) also uses procedures for operations. TRACLabs has two Department of Defense (DoD) contracts that use the PRIDE procedure technology for human-in-the-loop automation. The AMP project (Phase II Navy SBIR) is using procedure automation to control multiple coordinated unmanned vehicles for the Navy. The COACT project (Phase I Air Force SBIR) completed in June 2015 used procedure automation technology for remote control and coordination of smart weapons systems for the Air Force. The automation functions and, in some cases, the interface design developed under the NSBRI project have been adopted for use in future PRIDE applications, such as these. It is anticipated that task allocation strategies demonstrated effective for NASA and the oil and gas industry would have potential use for any industry operating in complex dynamic environments with a high cost of failure. Such industries include chemical and nuclear process control, and power management and distribution.

ANNUAL REPORTING IN OCTOBER 2014

As crewed missions move deeper into space and communication latency increases, astronauts will be unable to depend on real-time support from flight controllers, such as performing or advising on procedures as done for the International Space Station (ISS). Astronauts will have a longer time lag between task training and execution while also needing to perform more, and more diverse, tasks. These can increase astronaut workload, decrease efficiency, and increase the risk of suboptimal task execution. A critical resource for meeting these challenges is greater reliance on spacecraft automation. Today astronauts and flight controllers carry out tasks by following written procedures. As a result approaches that integrate automation with procedures are a good fit to NASA's concept of operations. If human-automation integration (HAI) for such procedure automation is not appropriately designed, however, increased automation may contribute to rather than alleviate these challenges. The strategies for allocating tasks to automation resulting from this project is of direct benefit to current and future NASA human space exploration missions. The PRIDE procedure software being evaluated and revised under this project has been used to flight follow two International Space Station (ISS) Extra Vehicular Activities (EVA), and is currently being modified to support manual execution of multiple coordinated procedures during EVA. PRIDE has been used to author test procedures for Morpheus, and is being considered for authoring and execution mission procedures by Sierra Nevada. Since the Deep Water Horizon accident, the oil and gas industry has become increasingly interested in technologies to improve the safety and traceability of drilling operations. Like NASA, these industries use Standard Operating Procedures (SOPs) as the basis of their operations. TRACLabs is currently developing procedure automation technology for well-site automation to be used by an oil and gas service company. This software will incorporate the automation planning functions and interface design developed and evaluated under the NSBRI project. This software will be deployed for testing at a drilling site in Oklahoma in 2015. The successful completion of this test should result in the gradual phasing of the procedure automation technology into operational use by the service company. The Department of Defense (DoD) also uses procedures for operations. TRACLabs has three DoD contracts that plan to use PRIDE procedure technology for human-in-the-loop automation. The AMP project will use procedure automation to control multiple coordinated unmanned vehicles for the Navy. The COACT project will use procedure automation technology for remote control and coordination of smart weapons systems for the Air Force. And the CHESS project will use procedure technology for missile defense. The automation functions and, in some cases, the interface design developed under the NSBRI project has been adopted for use in future PRIDE applications, such as these. We anticipate that task allocation strategies demonstrated effective for NASA and the oil and gas industry would have potential use for any industry operating in complex dynamic environments with a high cost of failure. Such industries include chemical and nuclear process control, and power management and distribution.

Task Progress & Bibliography Information FY2015 
Task Progress: FINAL REPORTING--FEBRUARY 2016

Task 1. Define scenarios and metrics for HERA Study: Three habitat systems were simulated - Carbon Dioxide Removal System (CDRS), Active Thermal Control System (ATCS), and Electrical Power System (EPS). The EPS provides power to the CDRS and ATCS, which provides cooling water for the CDRS. Procedures were developed for these systems, and for a secondary manual task to assess images collected by a robot. Performance was measured for impact on the controlled system (e.g., entry into unsafe states) and correctness of user actions (e.g., adherence to specified method to achieve goal).

Task 2. Complete HERA software development: Software activities for HERA experiments included 1) developing daily tasks that could be performed without an experimenter present, 2) simplifying startup, shutdown, and transfer of data for the PRIDE procedure software, and 3) connecting PRIDE software to Webmirage to retrieve HERA data. Software development supporting data analysis included reading PRIDE database, computing performance measures, and saving these measures to Excel spreadsheet.

Task 3. Conduct HERA Experiment: Experiments were conducted in all missions of HERA Campaign 2. The HERA investigation embedded two study-designs (n=16). The Step-List Study tested the hypothesis that performance using Step formatted procedures would be better than using List formatted procedures. The Problem-Solving Study investigated the effects of repetition on problem solving, and the effects of a secondary manual task done in conjunction with the primary habitat task where a problem occurs. Additional exploratory tasks were performed. Participants received two hours of training prior to the mission and performed procedure during 7 one-hour sessions.

Task 4. Analyze and document experimental results: Analysis of data collected in Year 2 was completed. This study investigates strategies participants used to problem-solve when the preconditions of procedure actions do not match operational conditions. Study results are reported in a paper presented at Human Factors and Ergonimcs Society (HFES 2015) meeting in October 2015. Analysis of data collected in the HERA experiment was completed. Because HERA Campaign 2 ended in late August 2015, results of the Step-List and the Problem-Solving Studies have not yet been published; they are reported in the Main Findings section. An exploratory task compares manual performance of a single procedure to performance using maximal automation. Preliminary results are reported in a paper presented at the AI in Space Workshop in July 2015.

Tasks 5. Document design guidance for allocating tasks to procedure automation: Results from the HERA study, with results from prior project studies, informed the identification of candidate design guidance and factors for effective task allocation strategies for our procedure-automation approach. They are reported in the Main Findings section.

ANNUAL REPORTING IN OCTOBER 2014

Task 1. Define scenarios, allocation strategies, and metrics for Year 2 Experiment. Year 2 scenarios reuse the Year 1 Carbon Dioxide Removal System (CDRS) procedures and simulation, and add new Active Thermal Control System (ATCS) procedures and simulation. The ATCS provides cooling water for the CDRS. This system coupling is used to create problem-solving situations where participants must detect and resolve problems using procedures. User task allocation strategy and performance are compared when using procedures organized by units of work (Step) to procedures without this organization (Flat). Metrics include errors, timing, and workload. Error will be assessed at 3 levels: undesired device commanding, execution of non-standard action sequence, and action execution in a mode different than requested.

Task 2. Complete Year 2 software development. PRIDE software for procedure automation was redesigned using observations of automation use in Year 1. The PRIDE interface was modified to inform the user which actions are automatable. User functions were added to mark units of work (lines or steps) for automatic execution. Automation functions were modified to use these markings during procedure execution.

Task 3. Conduct Year 2 Experiment. A key purpose of the Year 2 Experiment is to identify user strategies for automation use, and how they vary in different situations. This will inform the identification of core principles affecting the quality of automation allocation strategies. This experiment also provides a first usability assessment of new PRIDE automation planning capability. Independent variables are Scenario (7 scenarios), Procedure Structure (Steps, Flat), and Order (Steps-First, Flat-First). The Scenario and Structure factors are within-subject and Order is between-subject. Data collection from a planned 12 subjects is in progress at the time of this report.

Task 4. Analyze and document experimental results. Analysis of manual performance data collected in Year 1 was completed. Speed and accuracy of manual procedure execution was better using the PRIDE interface than the ISS-like interface. Results are documented in a paper at HFES 2014. Analysis of Year 1 data also was completed on how participants used PRIDE procedure software to allocate tasks to automation. Results are documented in a paper at IEEE Aerospace 2014. Results also informed the redesign of procedure automation software being evaluated in Experiment 2.

Task 5. Begin development of Year 3 software. Our project is being considered for HERA FY15 missions. These experiments will evaluate partial automation strategies during long duration experiments under in flight-similar conditions with crew-like participants. To improve our ability to collect data during HERA missions, we are developing software for automatic collection and scoring of performance data. Procedure automation software will be revised for usability once analysis of Year 2 data is complete.

Bibliography Type: Description: (Last Updated: 03/18/2021)  Show Cumulative Bibliography Listing
 
Papers from Meeting Proceedings Billman D, Schreckenghost D, Miri P. "Assessment of Alternative Interfaces for Manual Commanding of Spacecraft Systems: Compatibility with Flexible Allocation Policies." 58th Annual Meeting of the Human Factors and Ergonomics Society, Chicago, IL, October 27-31, 2014.

Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2014 Sep;58(1):365-9. (58th Annual Meeting of the Human Factors and Ergonomics Society, Chicago, IL, October 27-31, 2014.) http://dx.doi.org/10.1177/1541931214581075 , Sep-2014

Papers from Meeting Proceedings Schreckenghost D, Milam T, Billman D. "Human performance with procedure automation to manage spacecraft systems." 35th International Conference for Aerospace Experts, Academics, Military Personnel, and Industry Leaders, Big Sky, MT, March 1-8, 2014.

In: 2014 IEEE Aerospace Conference, Big Sky, MT, March 1-8, 2014. http://dx.doi.org/10.1109/AERO.2014.6836403 ; accessed 10/14/2014. , Mar-2014

Papers from Meeting Proceedings Schreckenghost D, Billman D, Milam T. "Effectiveness of Strategies for Partial Automation of Electronic Procedures during NASA HERA Analog Missions." 5th International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina, July 25-27, 2015.

5th International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina, July 25-27, 2015. See https://www.researchgate.net/publication/289539879 ; accessed 2/24/16. , Jul-2015

Papers from Meeting Proceedings Billman D, Schreckenghost D, Billinghurst S. "Problem Solving when Procedure Conditions Are Not Met: Using Procedure-Automation Software for Support." 59th Annual Meeting of the Human Factors and Ergonomics Society, Los Angeles, CA, October 26–30, 2015.

Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2015 Sep;59(1):294-8. (59th Annual Meeting of the Human Factors and Ergonomics Society, Los Angeles, CA, October 26–30, 2015.) http://dx.doi.org/10.1177/1541931215591060 , Sep-2015

Significant Media Coverage Kortenkamp D, Schreckenghost D (interviewed). "Procedure-Authoring Tool Improves Safety on Oil Rigs. PRIDE procedure automation technology featured as NASA spin-off to oil and gas industry." NASA Spinoff 2013 online. http://spinoff.nasa.gov/Spinoff2013/ee_3.html ; accessed 10/14/2014., Apr-2014
Project Title:  Automation in Procedures: Guidelines for Allocating Tasks for Performance Reduce
Fiscal Year: FY 2014 
Division: Human Research 
Research Discipline/Element:
HRP SHFH:Space Human Factors & Habitability (archival in 2017)
Start Date: 10/01/2012  
End Date: 09/30/2015  
Task Last Updated: 11/08/2013 
Download report in PDF pdf
Principal Investigator/Affiliation:   Schreckenghost, Debra  M.E.E. / TRACLabs, Inc. 
Address:  16969 N Texas Ave 
Suite 300 
Webster , TX 77598-4085 
Email: ghost@ieee.org 
Phone: 281-461-7886  
Congressional District: 22 
Web:  
Organization Type: INDUSTRY 
Organization Name: TRACLabs, Inc. 
Joint Agency:  
Comments:  
Project Information: Grant/Contract No. NCC 9-58-HFP02803 
Responsible Center: NSBRI 
Grant Monitor:  
Center Contact:   
Solicitation / Funding Source: 2011 Crew Health NNJ11ZSA002NA 
Grant/Contract No.: NCC 9-58-HFP02803 
Project Type: GROUND 
Flight Program:  
TechPort: No 
No. of Post Docs:
No. of PhD Candidates:
No. of Master's Candidates:
No. of Bachelor's Candidates:
No. of PhD Degrees:
No. of Master's Degrees:
No. of Bachelor's Degrees:
Human Research Program Elements: (1) SHFH:Space Human Factors & Habitability (archival in 2017)
Human Research Program Risks: (1) HARI:Risk of Inadequate Design of Human and Automation/Robotic Integration
Human Research Program Gaps: (1) SHFE-HARI-01:We need to evaluate, develop, and validate methods and guidelines for identifying human-automation/robot task information needs, function allocation, and team composition for future long duration, long distance space missions (IRP Rev G name change to HARI-01 from previous designation SHFE-HARI-01, which was IRP Rev F) (Previously: What guidelines and tools can we develop to enable system designers and mission planners to conduct systematic task/needs analyses at the appropriate level of detail to allocate work among appropriate agents (human and automation)?
Task Description: 1. Original project aims

We investigate the hypothesis that selecting units of work to automate based on procedures for humans will improve human-automation designs. The structured actions sets within procedures, such as steps, can be used as the basis for meaningful, sharable organization of what is automated and how information should be passed between humans and automation. We hypothesize that human-automation interaction organized this way will provide several benefits, relative both to manual operations and to less user-centric automation, including effective execution and more adaptable use with flexible levels of automation. To test this hypothesis we will 1) develop strategies for using human procedures to identify units of work for adjustable automation, 2) build a test environment with software for manual or partially automated execution of procedures, and 3) use this environment to evaluate the effectiveness of alternative strategies for executing procedures.

2. Key findings

First, structured but informal piloting identified several improvements needed in the interaction design of the PRIDE procedure automation software. These improvements ensure that the formal pilot study can address the underlying human-system integration, without being defeated by poor design choice. Second, the pilot study is producing findings about human use and its measurement. Our performance metrics for this study are based on measuring correctness and timing when following a procedure. Our study of metrics revealed two types of correctness to be measured - human action and behavior of the human-automation system. Performance with respect to human action assesses how well a person follows the procedure; performance with respect to the human-automation system assesses if human operations, whether following the procedure or not, produce the desired system effects safely. Thus a key finding is the need for new performance measures to characterize the behavior of the human-automation system. We assessed what combination of document-based instruction and experience-based practice is needed to gain competence quickly in the study. Results from ongoing testing indicate that subjects can perform procedures using PRIDE manually (PRIDE-manual condition) without initial, structured instruction. The experimenter provided modest instruction on how components work during a run, if needed. Training is required for running procedures in the ISS condition. The formal pilot is assessing PRIDE operation and comparing its performance to an interface very similar to that used on ISS. The study also investigates use of PRIDE's partial automation modes. Human subject testing in the formal pilot study is in progress at the time of this report, and will yield additional findings once test data are analyzed.

3. Impact of key findings

We assess how well task allocation strategies improve performance by comparing performance with partial automation to manual performance of procedures. We predict that human performance of procedures in the PRIDE-manual condition will be comparable to or better than human performance of the same procedures in the ISS condition. To test this hypothesis we added the ISS condition to the pilot study and are comparing performance for these two conditions. If this hypothesis is proven true, manual performance of procedures will be measured in Years 2 and 3 only using PRIDE-manual. Further, ongoing PRIDE development will be guided by improvements suggested when comparing these very different systems. New measures of performance were identified to assess the effectiveness of strategies for allocating tasks to automation. They also provide a means of assessing how well these strategies address safe use of procedures in complex dynamic environments, where degraded systems and novel use can result in procedure actions with unexpected effects. We predict that a shift from measuring how well a person follows procedure steps to measuring how well a person achieves desired system states safely will provide a more accurate and flexible measure of performance. A preliminary finding of the PRIDE evaluation is that the very ease of use of the PRIDE interface can mask the need for the user to pause and consider before taking action; this is especially important for actions with potential safety risks. The current PRIDE interface provides a minimum of information needed to send commands and observe command effects. We predict that displaying additional context information about intended system effects and potential risks of procedure actions will improve situation awareness when deciding whether to automate.

4. Proposed research plan for next year

Year 2 research will evaluate the initial strategy for allocating tasks to automation around procedural units of work. Experiments will compare two task allocation strategies (manual vs partially automated) for their effect on performance accuracy and speed, and on how easily users generalize from one procedure to another. We predict improvement of partially automated over manual execution on these measures. We expect to modify and develop new procedures, software, and simulations for use in performing these evaluations. We will develop new measures of human performance that characterize how well human actions achieve the desired system states in procedures, and avoid unsafe states. We then will evaluate the hypothesis that a shift from defining error by measuring how well a person follows procedure steps to measuring how well the person achieves desired system states safely provides a more accurate and flexible measure of human performance with procedures. We will develop new procedures and PRIDE displays that provide context information about the intended effects and potential risks of procedure actions, and evaluate whether these displays improve situation awareness when deciding whether to automate.

Research Impact/Earth Benefits: As crewed missions move deeper into space and communication latency increases, astronauts will be unable to depend on real-time support from flight controllers, such as performing or advising on procedures as done for the International Space Station (ISS). Astronauts will have a longer time lag between task training and execution while also needing to perform more, and more diverse, tasks. These can increase astronaut workload, decrease efficiency, and increase the risk of suboptimal task execution. A critical resource for meeting these challenges is greater reliance on spacecraft automation. Today astronauts and flight controllers carry out tasks by following written procedures. As a result approaches that integrate automation with procedures are a good fit to NASA's concept of operations. If human-automation integration (HAI) for such procedure automation is not appropriately designed, however, increased automation may contribute to rather than alleviate these challenges. The strategies for allocating tasks to automation resulting from this project is of direct benefit to current and future NASA human space exploration missions. There is significant recent interest in remote operations and automation for the oil and natural gas drilling, extraction, and processing. Similar to NASA, oil and gas drilling operations use Standard Operating Procedures (SOPs) that would benefit from procedure automation software like PRIDE. Whether monitoring and controlling an off-shore oil rig from an on-shore location using both local and remote experts or controlling robots that monitor and maintain off-shore rigs during an evacuation, or controlling robots that perform disaster response tasks in large refinery, the need for procedure-based automation in the oil and gas industry is growing. TRACLabs is currently participating in a field test of our PRIDE procedure automation technology with two oil and gas companies, and in discussion with multiple other companies about deploying our procedure automation technology into drilling operations. The improved support for human interaction with automation resulting from this project has direct application to the oil and gas drilling. The goal of developing strategies for automation that consider safety risks is particularly relevant to this industry. We also anticipate that task allocation strategies demonstrated effective for NASA and the oil and gas industry would have potential use for any industry operating in complex dynamic environments with a high cost of failure. Such industries include chemical and nuclear process control, power management and distribution, and military operations.

Task Progress & Bibliography Information FY2014 
Task Progress: Year 1 research aims to prepare for evaluating strategies to allocate tasks to the PRIDE procedure automation software in Years 2 and 3. We are developing software and conducting a pilot study to develop metrics and methods for these experiments and to evaluate PRIDE. Task progress is described below.

Task 1: Define scenarios, task allocation strategies, and metrics for pilot study. Scenarios developed for the pilot include executing procedures to start up, change modes, and shutdown a simulation of the CO2 Removal System (CDRS). Initial strategies were developed for allocating tasks to automation according to units of work. Performance metrics were defined to measure the accuracy and timing in following procedures during the pilot. Additional metrics were identified but not implemented to characterize how well and how safely human actions achieve the desired system states implied by the procedure.

Task 2: Develop software for pilot study. The PRIDE software was modified for the formal pilot study, including changes to improve ease of use and consistency of interaction. Software was developed to monitor and log to database the user's interaction with PRIDE. New procedures were developed to give the subjects more options for controlling the CDRS. New software was developed to emulate ISS displays and log user interactions with these displays to database. This software was integrated with the CDRS simulation for subjects to use to monitor and control the CDRS. PRIDE, ISS software, and the CDRS simulation were deployed at a website for early evaluation at ARC. Subsequently this software was installed on computers at ARC for the pilot.

Task 3: Conduct formative evaluation of procedure software. A formative evaluation of PRIDE is being conducted at ARC in 2 phases. During the first informal phase, human subject tests were performed to develop a training approach, and to identify changes to PRIDE prior to experiments comparing it to the ISS. Six subjects were tested with the PRIDE software. During the second formal phase (now ongoing), human subject tests are being conducted to compare the PRIDE conditions (manual and partially automated) to the ISS condition. Data collection from an expected 12 subjects is in progress at the time of this report.

Task 4: Analyze study results. Data from the first phase of the pilot study were analyzed. Data from the second phase of the pilot study is being collected. Metrics for accuracy and timing will be computed from these data. A preliminary analysis of these data will be done by the end of Year 1.

Task 5: Begin software development for Year 2. Software development to author and execute malfunction procedures was begun, to enable evaluating a more diverse set of procedures in Year 2. Malfunction procedures are used to safe systems after an anomaly and diagnose the anomaly cause. Software development for automatic computation of metrics also was begun to prove out our approach for Year 2.

Bibliography Type: Description: (Last Updated: 03/18/2021)  Show Cumulative Bibliography Listing
 
 None in FY 2014
Project Title:  Automation in Procedures: Guidelines for Allocating Tasks for Performance Reduce
Fiscal Year: FY 2013 
Division: Human Research 
Research Discipline/Element:
HRP SHFH:Space Human Factors & Habitability (archival in 2017)
Start Date: 10/01/2012  
End Date: 09/30/2015  
Task Last Updated: 10/24/2012 
Download report in PDF pdf
Principal Investigator/Affiliation:   Schreckenghost, Debra  M.E.E. / TRACLabs, Inc. 
Address:  16969 N Texas Ave 
Suite 300 
Webster , TX 77598-4085 
Email: ghost@ieee.org 
Phone: 281-461-7886  
Congressional District: 22 
Web:  
Organization Type: INDUSTRY 
Organization Name: TRACLabs, Inc. 
Joint Agency:  
Comments:  
Project Information: Grant/Contract No. NCC 9-58-HFP02803 
Responsible Center: NSBRI 
Grant Monitor:  
Center Contact:   
Solicitation / Funding Source: 2011 Crew Health NNJ11ZSA002NA 
Grant/Contract No.: NCC 9-58-HFP02803 
Project Type: GROUND 
Flight Program:  
TechPort: No 
No. of Post Docs:  
No. of PhD Candidates:  
No. of Master's Candidates:  
No. of Bachelor's Candidates:  
No. of PhD Degrees:  
No. of Master's Degrees:  
No. of Bachelor's Degrees:  
Human Research Program Elements: (1) SHFH:Space Human Factors & Habitability (archival in 2017)
Human Research Program Risks: (1) HARI:Risk of Inadequate Design of Human and Automation/Robotic Integration
Human Research Program Gaps: (1) SHFE-HARI-01:We need to evaluate, develop, and validate methods and guidelines for identifying human-automation/robot task information needs, function allocation, and team composition for future long duration, long distance space missions (IRP Rev G name change to HARI-01 from previous designation SHFE-HARI-01, which was IRP Rev F) (Previously: What guidelines and tools can we develop to enable system designers and mission planners to conduct systematic task/needs analyses at the appropriate level of detail to allocate work among appropriate agents (human and automation)?
Task Description: As crewed missions move deeper into space and communication latency increases, strategies for carrying out tasks must shift. Astronauts will be unable to depend on real-time support from flight controllers; controllers will not be able to perform procedures in the same way they do for Station, nor to advise on changed applicability of procedures in real-time. This change threatens to increase astronaut workload, decrease efficiency, and increase the risk of suboptimal task execution.

Automation is an important resource for adapting to this altered environment. To help rather than harm, however, automation must be effectively integrated with the humans it supports. The proposed research will 1) identify and refine candidate strategies for allocating tasks to automation and the factors when guidelines apply, 2) define effectiveness measures for these task allocation strategies, 3) conduct empirical assessment of the effectiveness of human-automation integration based on the proposed allocation strategies, and 4) integrate our findings as proposed task allocation strategies with automation.

Our approach to automation investigates the use of procedures as the basis of automation. By using a human-oriented procedure to organize automation, automation is designed to be more comprehensible to operators. For procedure automation, the actions in a procedure are enabled for automatic execution when instrumentation is available to perform the action. The degree of automation can be varied by changing which actions are designated for automatic execution. Strategies for determining which steps should be automated range from pre-defined allocations to flexible adjustment while the procedure is being executed. It is not clear how these strategies affect human-automation performance. Research is needed to determine how these different allocation strategies affect task performance and learnability. Additionally it is not clear how robust these strategies are to changes in situation that invalidate the procedure as written.

Our aim is identification of several candidate guidelines and factors that define effective task allocation strategies, within our procedure-automation approach. These guidelines consider where benefits may be greatest and how automation can be structured to realize the potential benefits. However, the ability to automate will depend upon reliably determining whether a procedure should be applied exactly in the circumstances and, if not, how to provide human skills to ensure appropriate application. The way in which human and automated actions are coordinated also needs to make the work organization meaningful to humans, as well as reliably executable by the automation. Specifying how to identify meaningful units of work that serve as the foundation for coordination between human and automated actions is a core part of the research.

TRACLabs and its partners from NASA and San Jose State University propose to evaluate human performance for different task allocation strategies for procedure automation and use the results to articulate a set of allocation strategies. We will define operational scenarios for the evaluation, including a set of multi-step procedures and simulation that works with these procedures. We will provide procedure automation software for executing these procedures on the simulation. We will perform ground-based human subject testing where subjects use the procedure automation to perform the procedures. We will select the task allocation strategies for evaluation that include both predefined and flexible allocation of tasks. We will measure and analyze human and automation performance for each of these strategies under both nominal and off-nominal circumstances. We will use experimental results to derive strategies for task allocation that are an important step toward developing technology to guide the allocation for tasks among humans and automation.

Research Impact/Earth Benefits: 0

Task Progress & Bibliography Information FY2013 
Task Progress: New project for FY2013.

Bibliography Type: Description: (Last Updated: 03/18/2021)  Show Cumulative Bibliography Listing
 
 None in FY 2013