Menu

 


Download in PDF pdf     
Fiscal Year: FY 2016  Task Last Updated:  01/26/2017 
PI Name: Duda, Kevin R Ph.D. 
Project Title: Metrics and Methods for Real-Time Task Performance Assessment 
   
Division Name: Human Research 
Program/Discipline--
Element/Subdiscipline:
NSBRI--Human Factors and Performance Team 
 
Joint Agency Name:   TechPort:  No 
Human Research Program Elements: (1) SHFH:Space Human Factors & Habitability
Human Research Program Risks: (1) HARI:Risk of Inadequate Design of Human and Automation/Robotic Integration
Human Research Program Gaps: (1) HARI-03:We do not know how to quantify overall human-automation-robotic system performance to inform and evaluate system designs to ensure safe and efficient space mission operations (IRP Rev G name change to HARI-03 from previous designation SHFE-HARI-03 (which was IRP Rev F) (Previously: How can performance, efficiency and safety guidelines be developed for appropriate task automation and the effective allocation of tasks between humans and automation?)
Space Biology Element: None
Space Biology Cross-Element Discipline: None
Space Biology Special Category: None
PI Email: kduda@draper.com  Fax:  617-258-2772 
PI Organization Type: NON-PROFIT  Phone: 617-258-4385  
Organization Name: The Charles Stark Draper Laboratory, Inc. 
PI Address 1: 555 Technology Sq 
PI Address 2: MS 27 
PI Web Page:  
City: Cambridge  State: MA 
Zip Code: 02139-3539  Congressional District: 
Comments:  
Project Type: GROUND  Solicitation:  2012 Crew Health NNJ12ZSA002N 
Start Date: 07/01/2013  End Date:  09/30/2016 
No. of Post Docs: No. of PhD Degrees: 
No. of PhD Candidates: No. of Master' Degrees: 
No. of Master's Candidates: No. of Bachelor's Degrees: 
No. of Bachelor's Candidates: Monitoring Center:  NSBRI 
Contact Monitor:   Contact Phone:   
Contact Email:  
Flight Program:  
Flight Assignment: NOTE: End date changed to 9/30/2016 per NSBRI (Ed., 4/5/16)

 

Key Personnel Changes/Previous PI:  
COI Name (Institution): Robinson, Stephen   ( University of California, Davis ) 
Grant/Contract No.: NCC 9-58-HFP03401 
Performance Goal No.:  
Performance Goal Text:

 

Task Description: 1. Original Project Aims/Objective: The project objective is to produce a configurable and portable simulation capability for developing and validating real-time metrics for assessing flight performance, workload, and situational awareness. There are three integrated specific aims: (1) Define the system architecture for integrating vehicle and environmental models with the simulation environment. (2) Perform a critical analysis of four piloted tasks: MPCV/Orion docking, MPCV/Orion entry, Lunar Landing, and EVA (extravehicular activity) SAFER self-rescue. Simulator data will be analyzed to identify candidate metrics for performance, workload, and situational awareness as well as operationally relevant options for presenting feedback to the operator. (3) Conduct a series of experiments using the simulated spaceflight tasks and real-time metrics engine to baseline performance, workload, and situational awareness in each task in order to develop algorithms and methods for alerting the operator to deviations from nominal.

2. Key Findings: In project year 3, our final project year, we completed the development of the simulation capability by integrating the piloted lunar lander, EVA SAFER return to the International Space Station (ISS), EVA SAFER ISS solar array inspection, and MPCV/Orion docking with the ISS vehicle models. The simulation software, along with an operations manual, was made freely available to those who agree to the Draper software licensing agreement. A copy of the Draper simulation station was delivered to NSBRI (National Space Biomedical Research Institute), and is fully capable of executing all the experimentation and demonstration use cases that were developed. In this final project year, an automated flight instructor was designed by our collaborators at University of California (UC) Davis to investigate the effects of three variations of an instructor-model performance-feedback strategy on human performance in a novel SAFER inspection task. Thirty subjects flew SAFER to perform an inspection of the ISS solar arrays. Subjects were initially placed 40 ft away from the array, and were asked to close to 30 ft and hold this distance while inspecting 4 damage" points using a guidance display for navigation to the waypoints. In order of priority, the task included: 1) maintaining the 30 ft distance, 2) minimizing their roll angle to a relative angle of 0 degrees, and 3) navigating the waypoints as quickly and accurately possible. Subjects were also asked to respond to a secondary task in the form of a comm light" as a proxy for workload. Finally, the subjects also made verbal callouts about their fuel level every 5% as a measure of situational awareness.

Subjects were placed into one of three groups. Group 1 acted as a control group, and had an analog distance error display that read from -10 to 10 ft, and digital displays with one significant figure. Group 2 had an analog distance error display that read from -5 to 5 ft, and digital display with two significant figures. Group 3 had the same analog and digital distance error displays as Group 1, and experienced real-time feedback in the form of displays that changed color. Results suggest Groups 2 and 3 both perform better than the control group on the primary and secondary flight tasks. The data also suggests that the real-time performance feedback immediately provided a performance improving effect, even with novice operators. While both Groups 2 and 3 showed performance improvements over the control group, Group 2's workload was increased, as was their time to complete the task.

3. Impact of Key Findings on hypotheses, technology requirements, objectives and specific aims of the original proposal: The development of the integrated simulation platform for running vehicle models, logging data, unobtrusively estimating workload and situation awareness, and providing visualizations and feedback to the pilot has significantly enhanced the capabilities for developing real-time performance metrics. By using typical spacecraft command and control tasks, such as piloted lunar landing, SAFER self-rescue, and Orion/MPCV docking, we have several operational scenarios to test our metrics. The Human Research Program (HRP) Integrated Research Plan gap (SHFE-TASK-01) states, in part, that, …The successful management or evaluation of workload must include a consideration of the nature of individual tasks that operators must perform, the combinations of tasks that are performed during a work period, priorities among tasks, and individual differences among operators. The development and evaluation of real-time performance metrics in representative operational settings—which include task performance, workload, and situational awareness, and are measured objectively as well as subjectively—will provide valuable data for the validity assessment. In project year 3, through conducting the experiment investigating the effects of an instructor-model performance-feedback strategy on human performance in a novel Simplified Aid for EVA Rescue (SAFER) inspection task revealed many interesting aspects. Not surprisingly, providing real-time performance feedback immediately provided a performance improving effect, even with novice operators. However, as the sensitivity of the feedback was increased (i.e., tighter performance criteria), the time to complete the task increased as well as the reported operator workload. The significance of this is that there is likely an optimum set of feedback sensitivity parameters that balance the operators task completion performance with their workload. The installation of a copy of our simulation station at NSBRI Headquarters is a valuable tool for collaborative use by researchers. Our team's continual pursuit of scientific investigation into the dynamic interaction between a pilot and their spacecraft led to a workshop on Piloted Spacecraft Guidance & Control Systems and Human Performance.

4. Proposed research plan for the coming year: This was the final year of the project. There is no proposed research plan for the coming year.

 

Rationale for HRP Directed Research:

 

Research Impact/Earth Benefits: This project delivered a research capability for evaluating the applicability and robustness of metrics for quantifying operator performance in real-time. Although our case studies are specific to piloted spacecraft, the innovations and implementation approach are generally applicable to any vehicle that requires a human in the loop. This re-configurable, portable simulation and test station provides a capability for integrating and testing real-time performance metrics for assessing operator effectiveness continually throughout a trial, as opposed to a single mission effectiveness metric. In addition, temporal operator performance can then be assessed against system-level metrics such as fuel consumption vs. time. Regardless of the domain, the interaction between vehicle/operation performance, operator workload, and operator situation awareness is complicated. Prior approaches to quantify these metrics have relied on post-hoc analyses or measurement approaches that affect the parameter of interest. This project aims to reduce to practice in-situ real-time performance, workload, and situation awareness metrics that can be objectively and unobtrusively collected. We are doing this through a flexible and module architecture that allows researchers to develop their own modules (either vehicle/system models or metrics modules) that can be integrated with our simulation framework. Through rigorous testing and integration with operationally-relevant tasks and scenarios, our goal is that this platform be adopted by the human-system integration and research community as the gold standard in crew performance benchmarking through open-source integration of algorithms for metrics development and validation.

 

Task Progress: In project year 3, our final project year, we completed the development of the simulation capability by integrating the piloted lunar lander, EVA SAFER return to ISS, EVA SAFER ISS solar array inspection, and MPCV/Orion docking with the ISS vehicle models. The simulation software, along with an operations manual, was made freely available to those who agree to the Draper software licensing agreement.

In this final project year, an automated flight instructor was designed by our collaborators at UC Davis to investigate the effects of three variations of an instructor-model performance-feedback strategy on human performance in a novel SAFER inspection task. Thirty subjects flew SAFER to perform an inspection of the ISS solar arrays. Subjects were tasked with: 1) maintaining the 30 ft distance, 2) minimizing their roll angle to a relative angle of 0 degrees, and 3) navigating the waypoints as quickly and accurately possible. Subjects were also asked to respond to a secondary task as a proxy for workload, and made verbal fuel percentage callouts as a measure of situational awareness. Group 1 had an analog distance error display that read from -10 to 10 ft, and digital displays with one significant figure. Group 2 had an analog distance error display that read from -5 to 5 ft, and digital display with two significant figures.

Results suggest Groups 2 and 3 both perform better than Group 1 on the primary and secondary flight tasks. The data also suggests that the real-time performance feedback immediately provided a performance improving effect, even with novice operators. While both Groups 2 and 3 showed performance improvements over Group 1, Group 2's workload was increased, as was their time to complete the task. The development of the integrated simulation platform for running vehicle models, logging data, unobtrusively estimating workload and situation awareness, and providing visualizations and feedback to the pilot has significantly enhanced the capabilities for developing real-time performance metrics and addressing the HRP TASK-01 Gap. By using typical spacecraft command and control tasks, such as piloted lunar landing, SAFER self-rescue, and Orion/MPCV docking, we have several operational scenarios to test our metrics.

The HRP Integrated Research Plan gap (SHFE-TASK-01) states, in part, that, …The successful management or evaluation of workload must include a consideration of the nature of individual tasks that operators must perform, the combinations of tasks that are performed during a work period, priorities among tasks, and individual differences among operators. The installation of a copy of our simulation station at NSBRI Headquarters is a valuable tool for collaborative use by researchers. Our team's continual pursuit of scientific investigation into the dynamic interaction between a pilot and their spacecraft led to a workshop on Piloted Spacecraft Guidance & Control Systems and Human Performance.

 

Bibliography Type: Description: (Last Updated: 04/05/2019) Show Cumulative Bibliography Listing
 
Abstracts for Journals and Proceedings Duda KR, Robinson SK, Prasov Z, York SP, Handley P, Karasinski J, Paddock E, West JJ. "Metrics and Methods for Real-Time Task Performance Assessment." 2016 NASA Human Research Program Investigators’ Workshop, Galveston, TX, February 8-11, 2016.

2016 NASA Human Research Program Investigators’ Workshop, Galveston, TX, February 8-11, 2016. , Feb-2016

Abstracts for Journals and Proceedings Duda KR, Handley PM, "Metrics and Methods for Real-Time Task Performance Assessment," Invited presentation to the NSBRI External Advisory Council Meeting. Houston, TX, 13 April 2016.

NSBRI External Advisory Council Meeting. Houston, TX, 13 April 2016. , Apr-2016

Articles in Peer-reviewed Journals Johnson AW, Duda KR, Sheridan TB, Oman CM. "A closed-loop model of operator visual attention, situation awareness, and performance across automation mode transitions." Human Factors. 2017 Mar;59(2):229-41. Epub 2016 Sep 2. http://dx.doi.org/10.1177/0018720816665759 ; PubMed PMID: 27591207 [Note: reported originally in Jan 2017 as 2016 Sep 2. (Epub ahead of print)] , Mar-2017
Awards Duda KR. "Elected to Associate Fellow of the American Institute of Aeronautics and Astronautics (AIAA), November 2016." Nov-2016
Awards Duda KR. "Elected to the Steering Committee for the International Conference on Environmental Systems (ICES), August 2016." Aug-2016
Awards Duda KR. "Promoted to Group Lead, Human Systems Integration, at The Charles Stark Draper Laboratory, Inc., April 2016." Apr-2016
Awards Duda KR. "Promoted to Principal Member of the Technical Staff at The Charles Stark Draper Laboratory, Inc., January 2016." Jan-2016
Awards Robinson SK. "Elected to Chair of the Mechanical and Aerospace Engineering Department at University of California Davis, May 2016." May-2016
Papers from Meeting Proceedings Karasinski JK, Robinson SK, Prasov Z, Duda KR. "Development of Real-Time Performance Metrics for Manually-Guided Spacecraft Operations." 2016 IEEE Aerospace Conference, Big Sky, MT, March 5-12, 2016.

In: 2016 IEEE Aerospace Conference Proceedings, 2016. p. 1-9. http://dx.doi.org/10.1109/AERO.2016.7500734 , Mar-2016

Download in PDF pdf     
Fiscal Year: FY 2015  Task Last Updated:  07/16/2015 
PI Name: Duda, Kevin R Ph.D. 
Project Title: Metrics and Methods for Real-Time Task Performance Assessment 
   
Division Name: Human Research 
Program/Discipline--
Element/Subdiscipline:
NSBRI--Human Factors and Performance Team 
 
Joint Agency Name:   TechPort:  No 
Human Research Program Elements: (1) SHFH:Space Human Factors & Habitability
Human Research Program Risks: (1) HARI:Risk of Inadequate Design of Human and Automation/Robotic Integration
Human Research Program Gaps: (1) HARI-03:We do not know how to quantify overall human-automation-robotic system performance to inform and evaluate system designs to ensure safe and efficient space mission operations (IRP Rev G name change to HARI-03 from previous designation SHFE-HARI-03 (which was IRP Rev F) (Previously: How can performance, efficiency and safety guidelines be developed for appropriate task automation and the effective allocation of tasks between humans and automation?)
Space Biology Element: None
Space Biology Cross-Element Discipline: None
Space Biology Special Category: None
PI Email: kduda@draper.com  Fax:  617-258-2772 
PI Organization Type: NON-PROFIT  Phone: 617-258-4385  
Organization Name: The Charles Stark Draper Laboratory, Inc. 
PI Address 1: 555 Technology Sq 
PI Address 2: MS 27 
PI Web Page:  
City: Cambridge  State: MA 
Zip Code: 02139-3539  Congressional District: 
Comments:  
Project Type: GROUND  Solicitation:  2012 Crew Health NNJ12ZSA002N 
Start Date: 07/01/2013  End Date:  09/30/2016 
No. of Post Docs: No. of PhD Degrees: 
No. of PhD Candidates: No. of Master' Degrees: 
No. of Master's Candidates: No. of Bachelor's Degrees: 
No. of Bachelor's Candidates: Monitoring Center:  NSBRI 
Contact Monitor:   Contact Phone:   
Contact Email:  
Flight Program:  
Flight Assignment: NOTE: End date changed to 9/30/2016 per NSBRI (Ed., 4/5/16)

 

Key Personnel Changes/Previous PI:  
COI Name (Institution): Robinson, Stephen   ( University of California, Davis ) 
Grant/Contract No.: NCC 9-58-HFP03401 
Performance Goal No.:  
Performance Goal Text:

 

Task Description: 1. Original Project Aims/Objective: The project objective is to produce a configurable and portable simulation capability for developing and validating real-time metrics for assessing flight performance, workload, and situational awareness. There are three integrated specific aims: (1) Define the system architecture for integrating vehicle and environmental models with the simulation environment. (2) Perform a critical analysis of four piloted tasks: MPCV/Orion docking, MPCV/Orion entry, Lunar Landing, and EVA SAFER self-rescue. Simulator data will be analyzed to identify candidate metrics for performance, workload, and situational awareness as well as operationally relevant options for presenting feedback to the operator. (3) Conduct a series of experiments using the simulated spaceflight tasks and real-time metrics engine to baseline performance, workload, and situational awareness in each task in order to develop algorithms and methods for alerting the operator to deviations from nominal.

2. Key Findings: In project year 2, we furthered the development of the simulation capability through integration of additional vehicle models, and testing the calculation of pilot flight performance, workload, and situation awareness metrics in real-time, without adding additional hardware to the simulator or requiring interrupts to the flight task. The baseline piloted lunar landing simulation was modified to include real-time estimation and plotting of flight performance (entropy of the hand controller inputs, root mean square error of the difference between the actual and guidance recommended pitch attitude), response time to the two-choice secondary task for workload estimation, and an estimate of situation awareness via the accuracy of required key system state callouts through an automatic speech recognition engine. These metrics were calculated in real-time throughout the trial, and are displayed through an engineering-level view. A human subject experiment (n = 26) was conducted using the lunar landing simulation in an effort to validate the performance of the automatic speech recognition (ASR) engine, as well as to collect data to develop unobtrusive workload estimation metrics that do not require a two-choice secondary task. The ASR analysis found that the system performed well (the ASR algorithm correctly recognized 838 of the 1035 valid callouts, for a precision of 0.81), and also identified several areas for further development to improve performance. We also developed a new real-time statistical approach to estimating flight performance by computing the percentage of pitch axis error in relation to pre-set bounds (comparison of actual vs. guidance recommended). The real-time performance metrics (flight, workload, and situation awareness) infrastructure was integrated with an extravehicular activity (EVA) simplified aid for EVA rescue (SAFER) jetpack International Space Station (ISS) self-rescue simulation in our portable ground station for future experimentation. The baseline Orion/MPCV piloted simulation for docking with the ISS was also implemented and tested in the portable ground station. It is these simulations and the data that we collect from the experiment that will enable the development of robust metrics that can be presented to the pilot for making operations more safe and efficient.

3. Impact of Key Findings on hypotheses, technology requirements, objectives and specific aims of the original proposal: The development of the integrated simulation platform for running the vehicle models, recording/logging data, unobtrusively estimating workload and situation awareness, and providing visualizations and feedback to the pilot has significantly enhanced the capabilities for developing real-time performance metrics. By using typical spacecraft command and control tasks, such as piloted lunar landing, SAFER self-rescue, and Orion/MPCV docking, we have several operational scenarios to test our metrics. The Human Research Program (HRP) Integrated Research Plan gap (SHFE-TASK-01) states, in part, that, …The successful management or evaluation of workload must include a consideration of the nature of individual tasks that operators must perform, the combinations of tasks that are performed during a work period, priorities among tasks, and individual differences among operators. The development and evaluation of real-time performance metrics in representative operational settings—which include task performance, workload, and situational awareness, and are measured objectively as well as subjectively—will provide valuable data for the validity assessment.

4. Proposed research plan for the coming year: In project year 3, we aim to conclude the analysis of the piloted lunar landing study. This analysis will produce a quantitative assessment of the ASR engine performance as a real-time situation awareness metric, as well as an unobtrusive workload estimation metric based on flight performance analysis (compared against the two-choice secondary task a as the gold standard). The presentation of the results to affect flight and mission performance will be prototyped in the simulator in a manner that seamlessly integrated with the existing displays. In collaboration with our team members at the University of California (UC) – Davis and the NASA Johnson Space Center (JSC) Virtual Reality Laboratory, we aim to conduct an experiment using the SAFER self-rescue simulation to develop and test our metrics engine, and to compare its robustness to novel scenarios through data collected during JSC VR Lab simulations. We will also complete the development of the Orion/MPCV docking simulation for human subject testing that is planned for later in project year 3. Lastly, we will deliver and install a copy of our simulation station at National Space Biomedical Research Institute (NSBRI) Headquarters to provide a foundational capability for subsequent test and evaluation with operators and subject matter experts.

 

Rationale for HRP Directed Research:

 

Research Impact/Earth Benefits: This project aims to deliver a research capability for evaluating the applicability and robustness of metrics for quantifying operator performance in real-time. Although our case studies are specific to piloted spacecraft, the innovations and implementation approach are generally applicable to any vehicle that requires a human in the loop. This re-configurable, portable simulation and test station provides a capability for integrating and testing real-time performance metrics for assessing operator effectiveness continually throughout a trial, as opposed to a single mission effectiveness metric. In addition, temporal operator performance can then be assessed against system-level metrics such as fuel consumption vs. time. Regardless of the domain, the interaction between vehicle/operation performance, operator workload, and operator situation awareness is complicated. Prior approaches to quantify these metrics have relied on post-hoc analyses or measurement approaches that affect the parameter of interest. This project aims to reduce to practice in-situ real-time performance, workload, and situation awareness metrics that can be objectively and unobtrusively collected. We are doing this through a flexible and module architecture that allows researchers to develop their own modules (either vehicle/system models or metrics modules) that can be integrated with our simulation framework. Through rigorous testing and integration with operationally relevant tasks and scenarios, our goal is that this platform be adopted by the human-system integration and research community as the gold standard in crew performance benchmarking through open-source integration of algorithms for metrics development and validation.

 

Task Progress: In project year 2, we furthered the development of the simulation capability through integration of additional vehicle models, and testing the calculation of pilot flight performance, workload, and situation awareness metrics in real-time, without adding additional hardware to the simulator or requiring interrupts to the flight task. The baseline piloted lunar landing simulation was modified to include real-time estimation and plotting of flight performance (entropy of the hand controller inputs, root mean square error of the difference between the actual and guidance recommended pitch attitude), response time to the two-choice secondary task for workload estimation, and an estimate of situation awareness via the accuracy of required key system state callouts through an automatic speech recognition engine. These metrics were calculated in real-time throughout the trial, and are displayed through an engineering-level view. A human subject experiment (n = 26) was conducted at UC Davis using the lunar landing simulation in an effort to validate the performance of the automatic speech recognition (ASR) engine, as well as to collect data to develop unobtrusive workload estimation metrics that do not require a two-choice secondary task. The ASR analysis found that the system performed well (the ASR algorithm correctly recognized 838 of the 1035 valid callouts, for a precision of 0.81), and also identified several areas for further development to improve performance. We also developed a new real-time statistical approach to estimating flight performance by computing the percentage of pitch axis error in relation to pre-set bounds (comparison of actual vs. guidance recommended). The real-time performance metrics (flight, workload, and situation awareness) infrastructure was integrated with an extravehicular activity (EVA) simplified aid for EVA rescue (SAFER) jetpack International Space Station (ISS) self-rescue simulation in our portable ground station for future experimentation. The baseline Orion/MPCV piloted simulation for docking with the ISS was also implemented and tested in the portable ground station. It is these simulations and the data that we collect from the experiment that will enable the development of robust metrics that can be presented to the pilot for making operations more safe and efficient. A copy of the portable ground control station was assembled. In year 3, we aim to complete the integration and test of the components, transfer the software to the new system, and develop the operational procedures in advance of delivering it to NSBRI Headquarters.

 

Bibliography Type: Description: (Last Updated: 04/05/2019) Show Cumulative Bibliography Listing
 
Abstracts for Journals and Proceedings Duda KR. "Metrics and Methods for Real-Time Task Performance Assessment." Invited presentation to the University of Central Florida Workshop, Developing Best Practices for Measuring Safety and Efficiency in Human-Automation Systems, 58th Annual Meeting of the Human Factors and Ergonomics Society, Chicago, IL, October 27-31, 2014

58th Annual Meeting of the Human Factors and Ergonomics Society, Chicago, IL, October 27-31, 2014. , Oct-2014

Abstracts for Journals and Proceedings Duda KR, Prasov Z, Robinson SK, York SP, Handley PM, West JJ. "Methods and Metrics for Real-Time Task Performance Assessment in Crewed Spacecraft." 2015 National Defense Industrial Association (NDIA) Human Systems Conference, Alexandria, VA, February 10-11, 2015.

2015 National Defense Industrial Association (NDIA) Human Systems Conference, Alexandria, VA, February 10-11, 2015. , Feb-2015

Abstracts for Journals and Proceedings Duda KR, Robinson SK, Prasov Z, York SP, Handley PM, Karasinski J, Tinch JD, West JJ. "Metrics and Methods for Real-Time Task Performance Assessment." Abstract and poster. 2015 NASA Human Research Program Investigators’ Workshop, Galveston, TX, January 13-15, 2015.

2015 NASA Human Research Program Investigators’ Workshop, Galveston, TX, January 13-15, 2015. , Jan-2015

Abstracts for Journals and Proceedings Duda KR, Robinson SK, Prasov Z, York SP, Handley PM, Karasinski J, Tinch JD, West JJ. "Metrics and Methods for Real-Time Task Performance Assessment." 86th Scientific Meeting of the Aerospace Medical Association, Lake Buena Vista, Florida, May 10-14, 2015.

Aerospace Medicine and Human Performance. 2015 Mar;86(3):207-8. See http://www.ingentaconnect.com/content/asma/amhp/2015/00000086/00000003/art00009 ;jsessionid=22uxwaslsgs2x.alice for searching table of contents; accessed 10/15/15. , Mar-2015

Awards Duda KR. "Chair of the AIAA Life Sciences & Systems Technical Committee, May 2015." May-2015
Awards Duda KR. "The Charles Stark Draper Laboratory, Inc. President's Outstanding Mentor Award, March 2015." Mar-2015
Dissertations and Theses Handley PM. "A Pilot Model for the NASA Simplified Aid for EVA Rescue (SAFER) (Single-Axis Pitch Task)." M.S. Thesis, University of California Davis, June 2014. , Jun-2014
Papers from Meeting Proceedings Duda KR, Prasov Z, York SP, West JJ, Robinson SK, Handley PM. "Development of an integrated simulation platform for real-time task performance assessment." 2015 IEEE Aerospace Conference, Big Sky, MT, March 7-14, 2015.

In: 2015 IEEE Aerospace Conference Proceedings, 2015. http://dx.doi.org/10.1109/AERO.2015.7118974 , Mar-2015

Download in PDF pdf     
Fiscal Year: FY 2014  Task Last Updated:  08/21/2014 
PI Name: Duda, Kevin R Ph.D. 
Project Title: Metrics and Methods for Real-Time Task Performance Assessment 
   
Division Name: Human Research 
Program/Discipline--
Element/Subdiscipline:
NSBRI--Human Factors and Performance Team 
 
Joint Agency Name:   TechPort:  No 
Human Research Program Elements: (1) SHFH:Space Human Factors & Habitability
Human Research Program Risks: (1) HARI:Risk of Inadequate Design of Human and Automation/Robotic Integration
Human Research Program Gaps: (1) HARI-03:We do not know how to quantify overall human-automation-robotic system performance to inform and evaluate system designs to ensure safe and efficient space mission operations (IRP Rev G name change to HARI-03 from previous designation SHFE-HARI-03 (which was IRP Rev F) (Previously: How can performance, efficiency and safety guidelines be developed for appropriate task automation and the effective allocation of tasks between humans and automation?)
Space Biology Element: None
Space Biology Cross-Element Discipline: None
Space Biology Special Category: None
PI Email: kduda@draper.com  Fax:  617-258-2772 
PI Organization Type: NON-PROFIT  Phone: 617-258-4385  
Organization Name: The Charles Stark Draper Laboratory, Inc. 
PI Address 1: 555 Technology Sq 
PI Address 2: MS 27 
PI Web Page:  
City: Cambridge  State: MA 
Zip Code: 02139-3539  Congressional District: 
Comments:  
Project Type: GROUND  Solicitation:  2012 Crew Health NNJ12ZSA002N 
Start Date: 07/01/2013  End Date:  06/30/2016 
No. of Post Docs: No. of PhD Degrees: 
No. of PhD Candidates: No. of Master' Degrees: 
No. of Master's Candidates: No. of Bachelor's Degrees: 
No. of Bachelor's Candidates: Monitoring Center:  NSBRI 
Contact Monitor:   Contact Phone:   
Contact Email:  
Flight Program:  
Flight Assignment:

 

Key Personnel Changes/Previous PI:  
COI Name (Institution): Robinson, Stephen   ( University of California, Davis ) 
Grant/Contract No.: NCC 9-58-HFP03401 
Performance Goal No.:  
Performance Goal Text:

 

Task Description: 1. Original Project Aims/Objective: The project objective is to produce a configurable and portable simulation capability for developing and validating real-time metrics for assessing flight performance, workload, and situational awareness. There are three integrated specific aims: (1) Define the system architecture for integrating vehicle and environmental models with the simulation environment. (2) Perform a critical analysis of four piloted tasks: MPCV/Orion docking, MPCV/Orion entry, Lunar Landing, and EVA SAFER self-rescue. Simulator data will be analyzed to identify candidate metrics for performance, workload, and situational awareness as well as operationally relevant options for presenting feedback to the operator. (3) Conduct a series of experiments using the simulated spaceflight tasks and real-time metrics engine to baseline performance, workload, and situational awareness in each task in order to develop algorithms and methods for alerting the operator to deviations from nominal.

2. Key Findings: In project year 1, we completed the development of the system architecture for integrating the vehicle models with our simulation framework, calculating objective workload metrics from flight performance data, performing speech recognition for situation awareness estimation, and rendering an out-the-window view using NASA's Engineering Dynamic On-board Ubiquitous Graphics (DOUG) Graphics for Exploration (EDGE) package. We developed a portable ground station, which includes a single PC, two joysticks (a rotational hand controller (RHC) and a translational hand controller (THC)), five monitors, speakers, and a microphone. Three prior lunar landing simulations – all developed under prior NSBRI projects (Project SA01604 Fuel Contour Display, HFP02001 Mode Transition and Failure Detection models) – and the SAFER EVA self-rescue scenario were integrated in the simulation. These simulations will provide the basis for the planned year 2 experimentation for developing and validating the implemented real-time performance metrics. Three real-time objective workload metrics were prototyped for implementation with the Mode Transition simulation (Hainley et al., 2013). These included the baseline implementation of a two-chose secondary task response time, an analysis of the entropy of the RHC inputs, and the root mean square error (RMSE) of the difference between the actual and guidance recommended attitude during piloted flight. The RHC entropy and attitude RMSE metrics were chosen based on post-hoc analysis of prior experiment data. In addition, we implemented a real-time speech recognition engine to analyze the accuracy of required key system state callouts as a measure of situation awareness (see Hainley et al., 2013). In this implementation, the utterances spoken by the user are recognized by the speech engine and then compared against the actual simulated vehicle state and are scored correct based on temporal (e.g., did they make the callout within 2 seconds?) and spatial (e.g., did they make the callout within 10 feet of the actual altitude?) boundaries. It is this feedback – both from the secondary task response time, mental workload, and situation awareness metrics – that we plan to further develop in the out years of the project to refine the presentation to the pilot for making operations more safe and efficient.

3. Impact of Key Findings on hypotheses, technology requirements, objectives, and specific aims of the original proposal: The development of the integrated simulation platform for running the vehicle models, recording/logging data, unobtrusively estimating workload and situation awareness, and providing visualizations and feedback to the pilot has significantly enhanced the capabilities for developing real-time performance metrics. By using typical spacecraft command and control tasks, such as piloted lunar landing, we have an initial operational scenario to test our metrics. The HRP Integrated Research Plan gap (SHFE-TASK-01) states, in part, that, …The successful management or evaluation of workload must include a consideration of the nature of individual tasks that operators must perform, the combinations of tasks that are performed during a work period, priorities among tasks, and individual differences among operators. The development and evaluation of real-time performance metrics in representative operational settings—which include task performance, workload, and situational awareness, and are measured objectively as well as subjectively—will provide valuable data for the validity assessment.

4. Proposed research plan for the coming year: In project year 2, we aim to begin conducting human-in-the-loop experiments with our lunar landing Mode Transition experiment to quantitatively compare the real-time performance metrics calculations with our baseline post-hoc measures of flight technical error, workload, and situation awareness. The real-time metrics that are analyzed and collected on-the-fly will be quantitatively compared against the prior approach to analyze the data after-the-fact. This will be done in close collaboration with our team mates at the University of California, Davis. In addition, an operationally relevant approach for fusing the vehicle flight data, workload, and situation awareness data will be developed, as well as an approach for providing feedback will be prototyped and evaluated. In project year 2, we also aim to develop and integrate Orion/MPCV rendezvous and docking and atmospheric entry models with our simulation. Real-time performance metrics will also be developed and integrated with this simulation, with the goal of having consistent metrics between classes of simulations. Lastly, we propose to deliver and install a copy of our simulation station at NSBRI Headquarters to provide a foundational capability for subsequent test and evaluation with operators and subject matter experts.

 

Rationale for HRP Directed Research:

 

Research Impact/Earth Benefits: This proposed project aims to deliver a research capability for evaluating the applicability and robustness of metrics for quantifying operator performance in real-time. Although our case studies are specific to piloted spacecraft, the innovations and implementation approach are generally applicable to any vehicle that requires a human in the loop. This re-configurable, portable simulation and test station provides a capability for integrating and testing real-time performance metrics for assessing operator effectiveness continually throughout a trial, as opposed to a single mission effectiveness metric. In addition, temporal operator performance can then be assessed against system-level metrics such as fuel consumption vs. time. Regardless of the domain, the interaction between vehicle/operation performance, operator workload, and operator situation awareness is complicated. Prior approaches to quantify these metrics have relied on post-hoc analyses or measurement approaches that affect the parameter of interest. This project aims to reduce to practice in-situ real-time performance, workload, and situation awareness metrics that can be objectively and unobtrusively collected. We are doing this through a flexible and module architecture that allows researchers to develop their own modules (either vehicle/system models or metrics modules) that can be integrated with our simulation framework. Through rigorous testing and integration with operationally-relevant tasks and scenarios, our goal is that this platform be adopted by the human-system integration and research community as the gold standard in crew performance benchmarking through open-source integration of algorithms for metrics development and validation.

 

Task Progress: In project year 1, we completed the development of the system architecture for integrating the vehicle models with our simulation framework, calculating objective workload metrics from flight performance data, performing speech recognition for situation awareness estimation, and rendering an out-the-window view using NASA's Engineering Dynamic On-board Ubiquitous Graphics (DOUG) Graphics for Exploration (EDGE) package. We developed a portable ground station, which includes a single PC, two joysticks (a rotational hand controller (RHC) and a translational hand controller (THC)), five monitors, speakers, and a microphone. (This ground control station leverages development that was performed for NASA's Game Changing Development Program.) Three prior lunar landing simulations – all developed under prior NSBRI projects (Project SA01604 Fuel Contour Display, HFP02001 Mode Transition and Failure Detection models) – as well as the SAFER EVA rescue scenario were integrated in the simulation, which included the vehicle/system dynamics models, flight displays, landing/return metrics, and data logging capability.

Three real-time objective workload metrics were prototyped for implementation with the lunar landing Mode Transition simulation (Hainley et al., 2013). These included the baseline implementation of a two-chose secondary task response time, an analysis of the entropy of the RHC inputs, and the root mean square error (RMSE) of the difference between the actual and guidance recommended attitude during piloted flight. In addition, we implemented a real-time speech recognition engine with our simulation to analyze the accuracy of required key system state callouts as a measure of situation awareness (see Hainley et al., 2013). In this implementation, the utterances spoken by the user are recognized by the speech engine and then compared against the actual simulated vehicle state and are scored correct based on temporal and spatial boundaries. It is this feedback – both from the secondary task response time, mental workload, and situation awareness metrics – that we plan to further develop in the out years of the project to refine the presentation to the pilot for making operations more safe and efficient.

Planning for the start of human subject experimentation in year 2 was initiated. This included the submission and approval of an experimental protocol to the University of California, Davis IRB, the shipment of a copy of the simulation control station with integrated lunar landing simulations, and the identification of research objectives and testable hypotheses for the investigation and validation of in-situ real-time flight performance, workload, and situation awareness metrics. In addition, the framework for fusing these metrics for providing feedback to the pilot and/or control system to make operations more safe and efficient was also completed.

REFERENCE (reported in NSBRI "Human-Automation Interactions and Performance Analysis of Lunar Lander Supervisory Control" project):

Hainley CJ Jr, Duda KR, Oman CM, Natapoff A. "Pilot performance, workload, and situation awareness during lunar landing mode transitions." Journal of Spacecraft and Rockets. 2013 Jul;50(4):793-801. http://dx.doi.org/10.2514/1.A32267

 

Bibliography Type: Description: (Last Updated: 04/05/2019) Show Cumulative Bibliography Listing
 
Abstracts for Journals and Proceedings Duda KR, Robinson SK, Handley P, Tinch JD, West JJ. "Metrics and Methods for Real-Time Task Performance Assessment." 2014 NASA Human Research Program Investigators’ Workshop, Galveston, TX, February 12-13, 2014.

2014 NASA Human Research Program Investigators’ Workshop, Galveston, TX, February 12-13, 2014. http://www.hou.usra.edu/meetings/hrp2014/pdf/3053.pdf , Feb-2014

Download in PDF pdf     
Fiscal Year: FY 2013  Task Last Updated:  02/05/2014 
PI Name: Duda, Kevin R Ph.D. 
Project Title: Metrics and Methods for Real-Time Task Performance Assessment 
   
Division Name: Human Research 
Program/Discipline--
Element/Subdiscipline:
NSBRI--Human Factors and Performance Team 
 
Joint Agency Name:   TechPort:  No 
Human Research Program Elements: (1) SHFH:Space Human Factors & Habitability
Human Research Program Risks: (1) HARI:Risk of Inadequate Design of Human and Automation/Robotic Integration
Human Research Program Gaps: (1) HARI-03:We do not know how to quantify overall human-automation-robotic system performance to inform and evaluate system designs to ensure safe and efficient space mission operations (IRP Rev G name change to HARI-03 from previous designation SHFE-HARI-03 (which was IRP Rev F) (Previously: How can performance, efficiency and safety guidelines be developed for appropriate task automation and the effective allocation of tasks between humans and automation?)
Space Biology Element: None
Space Biology Cross-Element Discipline: None
Space Biology Special Category: None
PI Email: kduda@draper.com  Fax:  617-258-2772 
PI Organization Type: NON-PROFIT  Phone: 617-258-4385  
Organization Name: The Charles Stark Draper Laboratory, Inc. 
PI Address 1: 555 Technology Sq 
PI Address 2: MS 27 
PI Web Page:  
City: Cambridge  State: MA 
Zip Code: 02139-3539  Congressional District: 
Comments:  
Project Type: GROUND  Solicitation:  2012 Crew Health NNJ12ZSA002N 
Start Date: 07/01/2013  End Date:  06/30/2016 
No. of Post Docs:   No. of PhD Degrees:   
No. of PhD Candidates:   No. of Master' Degrees:   
No. of Master's Candidates:   No. of Bachelor's Degrees:   
No. of Bachelor's Candidates:   Monitoring Center:  NSBRI 
Contact Monitor:   Contact Phone:   
Contact Email:  
Flight Program:  
Flight Assignment:

 

Key Personnel Changes/Previous PI:  
COI Name (Institution): Robinson, Stephen   ( University of California, Davis ) 
Grant/Contract No.: NCC 9-58-HFP03401 
Performance Goal No.:  
Performance Goal Text:

 

Task Description: This proposal addresses the NSBRI Human Factors and Performance research area to “develop and validate methods to assess task performance in real-time, provide immediate feedback, and recommend appropriate changes in time to improve mission outcomes,” using “operationally relevant scenarios or tasks for the spaceflight environment” (p. NSBRI-4). Future human exploration missions designs will likely be of varying duration, and require the direct interaction with and/or teleoperation of onboard systems and equipment, to accomplish exploration, assembly, or maintenance tasks (Review of U.S. Human Spaceflight Plans Committee, October 2009). Quantifying human factors and performance issues during real-time interaction with spacecraft systems is critical for assessing the impact of current tasking on mission outcomes and performance. The proposed project has three specific aims to develop a set of objective metrics that can be quantified to assess task performance in real-time, and provide immediate feedback to the human using several operationally relevant scenarios for the spaceflight environment:

1) Define the system architecture, integrate vehicle, system and environment models, and perform a critical analysis of the operationally relevant tasks to identify the specifics of candidate metrics for performance, workload, and situation awareness,

2) Develop and integrate real-time performance analysis techniques with the vehicle/system models that can run in real-time and provide immediate feedback to the operator, and

3) Conduct a series of simulations and experiments to baseline performance, workload, and situation awareness in each of the tasks.

Vehicle, system, and environment models, as well as task-specific displays and controls will be available to the operator for the following selectable scenarios: a) piloted MPCV/Orion atmospheric entry, b) piloted MPCV/Orion rendezvous, proximity operation, and docking with the ISS, c) ISS EVA/SAFER operations, and d) piloted lunar landing. We intend to leverage extensively the performance assessment methods developed under NSBRI Project HFP02001 to quantify performance, workload, and situation awareness as temporal measures during complex system automation mode transitions (e.g., Hainley, 2011; Hainley, Duda, et. al, in review).

 

Rationale for HRP Directed Research:

 

Research Impact/Earth Benefits: 0

 

Task Progress: New project for FY2013.

 

Bibliography Type: Description: (Last Updated: 04/05/2019) Show Cumulative Bibliography Listing