Menu

 

The NASA Task Book
Advanced Search     

Project Title:  Design and Evaluation of Automated Electronic Checklists for Robotics Operations Reduce
Fiscal Year: FY 2018 
Division: Human Research 
Research Discipline/Element:
HRP HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Start Date: 10/01/2015  
End Date: 01/31/2018  
Task Last Updated: 04/13/2018 
Download report in PDF pdf
Principal Investigator/Affiliation:   Oman, Charles M. Ph.D. / Massachusetts Institute of Technology 
Address:  Department of Aeronautics and Astronautics 
77 Massachusetts Avenue 37-219 
Cambridge , MA 02139-4301 
Email: coman@mit.edu 
Phone: 617-253-7508  
Congressional District:
Web:  
Organization Type: UNIVERSITY 
Organization Name: Massachusetts Institute of Technology 
Joint Agency:  
Comments:  
Co-Investigator(s)
Affiliation: 
Liu, Andrew  Ph.D. Massachusetts Institute of Technology 
Key Personnel Changes / Previous PI: August 2016: No changes
Project Information: Grant/Contract No. NNX15AW35G 
Responsible Center: NASA JSC 
Grant Monitor: Williams, Thomas  
Center Contact: 281-483-8773 
thomas.j.will1@nasa.gov 
Solicitation / Funding Source: 2014-15 HERO NNJ14ZSA001N-Crew Health (FLAGSHIP & NSBRI) 
Grant/Contract No.: NNX15AW35G 
Project Type: GROUND 
Flight Program:  
TechPort: No 
No. of Post Docs:
No. of PhD Candidates:
No. of Master's Candidates:
No. of Bachelor's Candidates:
No. of PhD Degrees:
No. of Master's Degrees:
No. of Bachelor's Degrees:
Human Research Program Elements: (1) HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Human Research Program Risks: (1) HCI:Risk of Inadequate Human-Computer Interaction
Human Research Program Gaps: (1) SHFE-HCI-06:We need guidelines to ensure crewmembers receive all of the information required to accomplish necessary tasks in a timely fashion, even when operating autonomously (IRP Rev D)
Flight Assignment/Project Notes: NOTE: Had been extended to 1/31/2018 per NSSC information (Ed., 4/16/18)

NOTE: Element change to Human Factors & Behavioral Performance; previously Space Human Factors & Habitability (Ed., 1/19/17)

Task Description: The primary objective of this project was to provide empirical evidence supporting design guidelines for automated electronic checklists for robotics operations. We used International Space Station (ISS) robotic arm operations as our proxy for general space operations and developed scenarios to test a crew’s ability to interact with more system automation than available in current ISS operations.

This project had the following two specific aims:

1. Develop a prototype display for robotics operations that integrated the electronic procedures with the displays for performing robotic tasks. The design process began with a hierarchical task analysis approach to drive out functional and information requirements for the display. Additionally, critical system states were identified using “Object-Process Methodology,” a task modeling approach that extends the task analysis to a computable level. We augmented our prototype design with the capability for automated execution of procedural steps. This prototype was integrated into the MIT (Massachusetts Institute of Technology) ISS robotic workstation simulation that has previously been used in several National Space Biomedical Research Institute (NSBRI) projects.

2. Perform a human-in-the-loop study that provided empirical evidence about the effects of the prototype system architecture, including varying levels of automation, on task performance, operator situation awareness, and mental workload. The study investigated how the allocation of procedural step execution between human operator and automation affects situation awareness (SA), mental workload, and task performance. Shifting more procedure steps to the automation may ease workload and speed task completion, but may also degrade the operator’s SA and ability to detect and resolve errant conditions. The experiment scenario included interrupting events that required switching between different procedures.

The project results provide a potential design method, implementation guidelines, and supporting empirical evidence for designing electronic checklists for other tasks.

Research Impact/Earth Benefits: Although we used robotic arm operations on the International Space Station as our study platform, the results of this research could be applied to many Earth-based systems. Aviation, for example, has been a domain that has long used checklists to complete complex procedures safely. However, the use of a checklist does not automatically rule out the possibility of human operator errors. Errors of commission or omission of checklist steps, miscommunication between pilots, stress and time pressure in an emergency situation, among other factors, can lead to mistakes. To mitigate these mistakes, the aviation industry has increased the role of automation in the cockpit, including recent integration of electronic checklists, allowing the computers to do some of the work so that the pilots aren’t overloaded and are less likely to make an error. Our study aims to investigate the effect of varying levels of automation on human performance when using electronic checklists and could provide valuable insight not only for future space robotic operations, but also to the aviation industry, and other domains in which the human and automation must work together to accomplish complex procedures.

Task Progress & Bibliography Information FY2018 
Task Progress: The project has successfully completed both of the project Aims. Here we review the progress made over the duration of the two-year project and provide a brief summary of the results of the human-in-the-loop study.

Our initial efforts focused on developing a prototype electronic procedure system with new automated capabilities and integrating this capapbility into our robotic workstation simulator. First, informed by actual robotic arm procedures conducted on the International Space Station, we defined and developed a set of robotic task scenarios that includes system setup, support of a series of simulated spacewalk activities, and system shutdown. Within these scenarios, the possibility for a variety of failures and associated failure recovery steps were designed and built into the simulation so that we can test responses to non-normal situations. We used tools including Hierarchical Task Analysis (HTA) and Object-Process Methodology (OPM) to break down each task goal into its components and necessary actions and to define system states throughout the flow of the simulated robotic arm operation.

Secondly, we used the HTA and OPM analyses to design the computer code that creates the simulated integrated control panel and procedure checklist system. In doing so, we created a new type of human-machine electronic procedure interface that integrates automation and system information into an electronic display. By integrating a functional electronic control panel, procedure, and checklist into our simulator, we created a system that presents information equivalent to two separate systems and laptops that are currently used on the International Space Station during robotic operations. This, in turn, reduces the required number of displays and equipment, an important feature for future space missions that will have strict mass and power constraints.

Following the development of our prototype automated electronic procedure system, we completed a human-in-the-loop experiment to investigate how the allocation of procedural step execution between the human operator and automation, or level of automation, would affect task performance, situation awareness, and mental workload. The key independent variable in our experiment was the level of automation used during task completion. The simulator was capable of allocating the automation to perform all or a subset of the procedural steps including state selection, identification verification, and confirmation. Three levels of automation were tested--Full Auto, Full Manual, and Auto Set/ Manual Verify. If the automation was tasked with doing all of the procedure steps, then the system is in Full Auto mode in which we expected low subject mental workload and high performance but also low situation awareness. In Full Manual mode, all steps were allocated to the human operator, and we expected high situation awareness but also high workload and possibly lower task performance. In the intermediate automation allocation, Auto Set/ Manual Verify, the automation was tasked with setting the states while the human operator verified the state change. In this intermediate automation level, we expected high task performance and situation awareness, and intermediate workload.

Our results are largely consistent with our initial hypotheses, that Full Automation mode results in faster task completion times with practically no errors and low subject mental workload, measured objectively with a visual secondary task and subjectively using the NASA Task Load Index (NASA-TLX). The reduction in completion time comes primarily from the automatic selection of cameras and entry of arm parameters for Space Station Remote Manipulator System (SSRMS) movement performed by the automation, rather than manually. The elimination of manual control tasks reduces the visual demands of the task and frees attentional resources for monitoring system state and task progress. This behavior was observed in the improvement of the operator’s ability to detect system events and states in the Full Manual condition. However, the use of automation also degraded the operator’s ability to comprehend of current system state and predict future states compared to Full Manual mode. The intermediate automation mode, Auto Set/Manual Verify, was generally preferred by the subjects as the best compromise to speed up task completion while maintain their situation awareness of the task and also resulted in the best overall performance across time, workload, and awareness. We believe that the findings of this research will be beneficial in the design and use of future automatable systems to be used in long duration, limited resource, and time-delayed spaceflight missions.

Bibliography Type: Description: (Last Updated: 01/17/2020) 

Show Cumulative Bibliography Listing
 
Abstracts for Journals and Proceedings Yang YE, Liu A, Dori D, Galvan-Garza R, Oman C. "Task Analysis and Interface Design Using Object-Oriented Methodology." Presented at 88th Aerospace Medical Association Annual Meeting, Denver, CO, April 30-May 4, 2017.

Aerospace Medicine and Human Performance. 2017 Mar;88:234. , Mar-2017

Abstracts for Journals and Proceedings Liu A, Galvan-Garza R, Yang YK, Oman CM. "Design and Automation of Electronic Checklists for Robotic Operations." Presented at the 22018 NASA Human Research Program Investigators’ Workshop, Galveston, TX, January 22-25, 2018.

2018 NASA Human Research Program Investigators’ Workshop, Galveston, TX, January 22-25, 2018. , Jan-2018

Abstracts for Journals and Proceedings Liu AM, Galvan-Garza R, Yang YK, Oman CM. "Design and Automation of Electronic Checklists for Robotic Operations." 2017 NASA Human Research Program Investigators’ Workshop, Galveston, TX, January 23-26, 2017.

2017 NASA Human Research Program Investigators’ Workshop, Galveston, TX, January 23-26, 2017. , Jan-2017

Dissertations and Theses Yang YE. (Yongkai Eugene Yang) "Using Object Process Methodology to Develop Interfaces and Smart Electronic Procedures for Simulated Telerobotic Operations." MS Thesis, Massachusetts Institute of Technology, Cambridge, MA, February 2017. , Feb-2017
Project Title:  Design and Evaluation of Automated Electronic Checklists for Robotics Operations Reduce
Fiscal Year: FY 2017 
Division: Human Research 
Research Discipline/Element:
HRP HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Start Date: 10/01/2015  
End Date: 09/30/2017  
Task Last Updated: 08/02/2016 
Download report in PDF pdf
Principal Investigator/Affiliation:   Oman, Charles M. Ph.D. / Massachusetts Institute of Technology 
Address:  Department of Aeronautics and Astronautics 
77 Massachusetts Avenue 37-219 
Cambridge , MA 02139-4301 
Email: coman@mit.edu 
Phone: 617-253-7508  
Congressional District:
Web:  
Organization Type: UNIVERSITY 
Organization Name: Massachusetts Institute of Technology 
Joint Agency:  
Comments:  
Co-Investigator(s)
Affiliation: 
Liu, Andrew  Ph.D. Massachusetts Institute of Technology 
Key Personnel Changes / Previous PI: August 2016: No changes
Project Information: Grant/Contract No. NNX15AW35G 
Responsible Center: NASA JSC 
Grant Monitor: Williams, Thomas  
Center Contact: 281-483-8773 
thomas.j.will1@nasa.gov 
Solicitation / Funding Source: 2014-15 HERO NNJ14ZSA001N-Crew Health (FLAGSHIP & NSBRI) 
Grant/Contract No.: NNX15AW35G 
Project Type: GROUND 
Flight Program:  
TechPort: No 
No. of Post Docs:
No. of PhD Candidates:
No. of Master's Candidates:
No. of Bachelor's Candidates:
No. of PhD Degrees:
No. of Master's Degrees:
No. of Bachelor's Degrees:
Human Research Program Elements: (1) HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Human Research Program Risks: (1) HCI:Risk of Inadequate Human-Computer Interaction
Human Research Program Gaps: (1) SHFE-HCI-06:We need guidelines to ensure crewmembers receive all of the information required to accomplish necessary tasks in a timely fashion, even when operating autonomously (IRP Rev D)
Flight Assignment/Project Notes: NOTE: Element change to Human Factors & Behavioral Performance; previously Space Human Factors & Habitability (Ed., 1/19/17)

Task Description: The primary objective of this project is to provide empirical evidence supporting design guidelines for automated electronic checklists for robotics operations. We use International Space Station (ISS) robotic arm operations as our proxy for general space operations and are developing scenarios to test a crew’s ability to interact with more system automation than available in current ISS operations.

This project has the following two specific aims:

1. Develop a prototype display for robotics operations that integrates the electronic procedures with the displays for performing robotic tasks. The design process will begin with a hierarchical task analysis approach to drive out functional and information requirements for the display. Additionally, critical system states will be identified using a “Object-Process Methodology,” a task modeling approach that extends the task analysis to a computable level. Lessons learned from the development of aviation electronic checklists will also be considered in the design. We will augment our prototype design with the capability for automated execution of procedural steps. This prototype will be built on the MIT (Massachusetts Institute of Technology) ISS robotic workstation simulation that has previously been used in several National Space Biomedical Research Institute (NSBRI) projects.

2. Perform a human-in-the-loop study that investigates the following questions concerning design choices for the integrated display:

a. Does the prototype electronic checklist enable the same or better situation awareness during task execution while minimizing mental workload when compared to current practice?

b. What procedural steps should be allocated to human operators or to the automation for both nominal operations and off-nominal, time-critical operations? How does the reliability of the automation affect the ideal allocation of steps?

c. Does the use of automated procedural step execution increase or decrease the human operator’s information requirements when executing multiple procedures?

The project results will provide a design method, implementation guidelines, and supporting empirical evidence for designing electronic checklists for other tasks.

Research Impact/Earth Benefits: Although we are using robotic arm operations on the International Space Station as our study platform, the results of this research could be applied to many Earth-based systems. Aviation, for example, has been a domain that has long used checklists to complete complex procedures safely. However, the use of a checklist does not automatically rule out the possibility of human operator errors. Errors of commission or omission of checklist steps, miscommunication between pilots, stress and time pressure in an emergency situation, among other factors, can lead to mistakes. To mitigate these mistakes, the aviation industry has increased the role of automation in the cockpit, including recent integration of electronic checklists, allowing the computers to do some of the work so that the pilots aren’t overloaded and are less likely to make an error. Our study aims to investigate the effect of varying levels of automation on human performance when using electronic checklists and could provide valuable insight not only for future space robotic operations, but also to the aviation industry, and other domains in which the human and automation must work together to accomplish complex procedures.

Task Progress & Bibliography Information FY2017 
Task Progress: During the first year of this award, we focused our efforts on achieving Specific Aim 1, developing a prototype electronic procedure system with new automated capabilities, into our robotic workstation simulator. First, informed by actual robotic arm procedures conducted on the International Space Station, we defined and developed a set of robotic task scenarios that includes system setup, support of a series of simulated spacewalk activities, and system shutdown. Within these scenarios, the possibility for a variety of failures and associated failure recovery steps were designed and built into the simulation so that we can test responses to non-normal situations. We used tools including Hierarchical Task Analysis (HTA) and Object-Process Methodology (OPM) to break down each task goal into its components and necessary actions and to define system states throughout the flow of the simulated robotic arm operation.

Secondly, we used the HTA and OPM analyses to design the computer code that creates the simulated integrated control panel and procedure checklist system. In doing so, we created a new type of human-machine electronic procedure interface that integrates automation and system information into an electronic display. By integrating a functional electronic control panel, procedure, and checklist into our simulator, we created a system that presents information equivalent to two separate systems and laptops that are currently used on the International Space Station during robotic operations. This, in turn, reduces the required number of displays and equipment, an important feature for future space missions that will have strict mass and power constraints.

Bibliography Type: Description: (Last Updated: 01/17/2020) 

Show Cumulative Bibliography Listing
 
 None in FY 2017
Project Title:  Design and Evaluation of Automated Electronic Checklists for Robotics Operations Reduce
Fiscal Year: FY 2016 
Division: Human Research 
Research Discipline/Element:
HRP HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Start Date: 10/01/2015  
End Date: 09/30/2017  
Task Last Updated: 10/06/2015 
Download report in PDF pdf
Principal Investigator/Affiliation:   Oman, Charles M. Ph.D. / Massachusetts Institute of Technology 
Address:  Department of Aeronautics and Astronautics 
77 Massachusetts Avenue 37-219 
Cambridge , MA 02139-4301 
Email: coman@mit.edu 
Phone: 617-253-7508  
Congressional District:
Web:  
Organization Type: UNIVERSITY 
Organization Name: Massachusetts Institute of Technology 
Joint Agency:  
Comments:  
Co-Investigator(s)
Affiliation: 
Liu, Andrew  Ph.D. Massachusetts Institute of Technology 
Project Information: Grant/Contract No. NNX15AW35G 
Responsible Center: NASA JSC 
Grant Monitor: Whitmore, Mihriban  
Center Contact: 281-244-1004 
mihriban.whitmore-1@nasa.gov 
Solicitation / Funding Source: 2014-15 HERO NNJ14ZSA001N-Crew Health (FLAGSHIP & NSBRI) 
Grant/Contract No.: NNX15AW35G 
Project Type: GROUND 
Flight Program:  
TechPort: No 
No. of Post Docs:  
No. of PhD Candidates:  
No. of Master's Candidates:  
No. of Bachelor's Candidates:  
No. of PhD Degrees:  
No. of Master's Degrees:  
No. of Bachelor's Degrees:  
Human Research Program Elements: (1) HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Human Research Program Risks: (1) HCI:Risk of Inadequate Human-Computer Interaction
Human Research Program Gaps: (1) SHFE-HCI-06:We need guidelines to ensure crewmembers receive all of the information required to accomplish necessary tasks in a timely fashion, even when operating autonomously (IRP Rev D)
Task Description: The primary objective of this project is to provide empirical evidence supporting design guidelines for automated electronic checklists for robotics operations. We use International Space Station (ISS) robotic arm operations as our proxy for general arm or rover operations but we will develop scenarios that test a crew’s ability to operate more autonomously than in ISS operations.

The proposed project will have the following two specific aims:

1. Develop a prototype display for supporting robotics operations that integrates the electronic procedures with the displays for performing robotics tasks. The design process will begin with a hierarchical task analysis approach to drive out functional and information requirements for the display. Lessons learned from the development of aviation electronic checklists will also be considered in the design. We will also augmented our prototype design with the capability for automated execution of the procedural steps. This prototype will be built on the MIT (Massachusetts Institute of Technology) ISS robotics simulation that has previously been used in several National Space Biomedical Research Institute (NSBRI) projects.

2. Complete human-in-the-loop studies that investigate the following questions concerning design choices for the integrated display:

a. Does the prototype electronic checklist enable the same or better situation awareness during task execution while minimizing mental workload when compared to current practice?

b. What is an appropriate allocation for procedural step execution between human operator and automation for both nominal operations and off-nominal time-critical operations? How does the reliability of the automation affect the ideal allocation of steps?

c. Does the use of automated procedural step execution increase or decrease the information requirements when executing multiple procedures?

The project results will provide a design method, implementation guidelines, and supporting empirical evidence for designing electronic checklists for other tasks.

Research Impact/Earth Benefits:

Task Progress & Bibliography Information FY2016 
Task Progress: New project for FY2016.

Bibliography Type: Description: (Last Updated: 01/17/2020) 

Show Cumulative Bibliography Listing
 
 None in FY 2016