Menu

 

The NASA Task Book
Advanced Search     

Project Title:  HCAAM VNSCOR: Virtual Assistant for Spacecraft Anomaly Treatment During Long Duration Exploration Missions Reduce
Images: icon  Fiscal Year: FY 2022 
Division: Human Research 
Research Discipline/Element:
HRP HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Start Date: 03/06/2019  
End Date: 03/05/2023  
Task Last Updated: 01/05/2022 
Download report in PDF pdf
Principal Investigator/Affiliation:   Selva, Daniel  Ph.D. / Texas A&M University 
Address:  Aerospace Engineering Department 
701 Ross St 3141 TAMU 
College Station , TX 77843-0001 
Email: dselva@tamu.edu 
Phone: 607-255-6351  
Congressional District: 17 
Web:  
Organization Type: UNIVERSITY 
Organization Name: Texas A&M University 
Joint Agency:  
Comments:  
Co-Investigator(s)
Affiliation: 
Diaz Artiles, Ana  Ph.D. Texas A&M Engineering Experiment Station 
Dunbar, Bonnie  Ph.D. Texas A&M Engineering Experiment Station 
Wong, Raymond  Ka Wai Ph.D. Texas A & M, College Station 
Project Information: Grant/Contract No. 80NSSC19K0656 
Responsible Center: NASA JSC 
Grant Monitor: Whitmire, Alexandra  
Center Contact:  
alexandra.m.whitmire@nasa.gov 
Solicitation / Funding Source: 2017-2018 HERO 80JSC017N0001-BPBA Topics in Biological, Physiological, and Behavioral Adaptations to Spaceflight. Appendix C 
Grant/Contract No.: 80NSSC19K0656 
Project Type: GROUND 
Flight Program:  
TechPort: Yes 
No. of Post Docs:
No. of PhD Candidates:
No. of Master's Candidates:
No. of Bachelor's Candidates:
No. of PhD Degrees:
No. of Master's Degrees:
No. of Bachelor's Degrees:
Human Research Program Elements: (1) HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Human Research Program Risks: (1) HSIA:Risk of Adverse Outcome Due to Inadequate Human Systems Integration Architecture (IRP Rev L)
Human Research Program Gaps: (1) HSIA-101:We need to identify the Human Systems Integration (HSI) – relevant crew health and performance outcomes, measures, and metrics, needed to characterize and mitigate risk, for future exploration missions (IRP Rev L)
(2) HSIA-201:We need to evaluate the demands of future exploration habitat/vehicle systems and mission scenarios (e.g. increased automation, multi-modal communication) on individuals and teams, and determine the risks these demands pose to crew health and performance (IRP Rev L)
(3) HSIA-401:We need to determine how HSI can be applied in the vehicle/habitat and computer interface Design Phase to mitigate potential decrements in operationally-relevant performance (e.g. problem-solving, execution procedures), during increasingly earth-independent, future exploration missions (including in-mission and at landing) (IRP Rev L)
(4) HSIA-701:We need to determine how human-automation-robotic systems can be optimized for effective enhancement and monitoring of crew capabilities, health, and performance, during increasingly earth-independent, future exploration missions (including in-mission and at landing) (IRP Rev L)
Task Description: This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

The research objective of this proposal is to investigate the impact of using Virtual Assistants (VA) to support crewmembers in the context of anomaly treatment during Long Duration Exploration Missions (LDEM), when ground support will be limited. A VA will be developed building upon the software architecture from existing VAs developed by the Principal Investigator (PI) for similar purposes. The VA will provide support for various aspects of anomaly treatment, including detecting and diagnosing the anomaly, as well as recommending a course of action. It will also have the ability to take initiative in the dialog with the user (mixed-initiative mode), and the ability to provide explanations for its actions. The impact of the VA on performance, cognitive workload, situational awareness, and trust, will be assessed through a set of three experiments with human subjects in a laboratory environment. The first experiment will establish the baseline impact (master-slave, no explanations), and subsequent experiments will study the effect of switching to the mixed-initiative mode and adding explanations. The system will also be deployed and tested in the Human Exploration Research Analog (HERA) analog environment.

Research Impact/Earth Benefits: This project will provide standards and guidelines that will help NASA design similar virtual assistants to support astronauts during future long duration exploration missions. Such standards and guidelines will concern both the functionality and the user interface of the virtual assistant.

Task Progress & Bibliography Information FY2022 
Task Progress: In Year 3 of this project, we have completed the first lab experiment, thus achieving Specific Aim 1, pending publication of the journal paper with the result. A total of 12 participants (mean age ± SD = 24.8 ± 1.9 years) were selected from an astronaut-like population. The participants were largely comprised of students at Texas A&M University, who had either already completed or were currently pursuing a graduate degree in a science, technology, engineering, mathematics (STEM) field. In addition, most participants had previous experience working with technical procedures. Each participant completed two experiment sessions: a control session without Daphne and a treatment session with Daphne. During both conditions, subjects were tasked with detecting, diagnosing, and resolving Environmental Control and Life Support Subsystem (ECLSS) anomaly scenarios. We selected two different anomaly scenario groups (5 anomaly scenarios per group) based on the perceived difficulty of resolving that particular anomaly and completing its corresponding resolution procedure. We counterbalanced participants with respect to which anomaly group they received first and in which experiment session they could use Daphne.

We measured performance metrics by evaluating the number of anomalies that participants could successfully resolve with and without the use of Daphne-AT. Additional performance metrics included the time it took for each subject to solve each anomaly, whether or not the subject detected the anomaly, diagnosed the root cause, and selected the correct anomaly resolution procedure. We also recorded the number of attempts for the latter three metrics. In addition, we measured cognitive workload by employing the NASA Task Load Index (TLX) survey, and situational awareness using the Situational Awareness Rating Technique (SART). Finally, the participants’ trust in autonomous systems was measured using Jian's trust scale.

The summary of the results is as follows. For performance, a significant difference (p=0.002) was found in the number of anomalies correctly resolved with Daphne (4.1±0.3) vs. without Daphne (2.7±0.4). This supports our hypothesis that a VA can improve performance in anomaly resolution. For cognitive workload, a significant difference (p=0.002) was found in the TLX score with Daphne (45.53±6.67) vs. without Daphne (65.97±4.81). This supports our hypothesis that a VA can reduce cognitive workload in anomaly resolution. For situational awareness, a marginally significant difference (p=0.052) was found in the SART score with Daphne (5.19±0.56) vs. without Daphne (4.41±0.71). This does not fully support our hypothesis that a VA can improve situational awareness in anomaly resolution. Finally, the responses to the Jian scale items showed generally high trust in automation. A much more detailed description of the experiment and the results is in the paper about to be submitted to the Human Factors and Ergonomics Society (HFES) journal. The draft paper will be provided to the NASA Human Research Program (HRP) as soon as it is submitted.

This year we also completed the first HERA mission Campaign 6, Mission XXII (C6M1). Campaign 6 / C6M2 will also be complete by the time the period of performance of this report is exhausted, which gets us about halfway in meeting Specific Aim 3. Our HERA study was divided into two phases. In Phase 1, each crewmember performed two experiment sessions per week. In each session, the crewmember worked on a single anomaly scenario, either with Daphne or without. Thus, at the end of Phase 1, each subject had worked individually on 3 anomalies with Daphne and 3 without. Because of a technical issue with Daphne, one of the crewmembers ended up with only 2 anomalies with Daphne and 4 without Daphne. In Phase 2, crew members worked together as a team. They conducted two sessions per week, and worked on one anomaly scenario per session. This resulted in a total of 3 anomalies with Daphne and 3 without. Preliminary results are as follows.

Concerning performance, all scenarios were successfully resolved and therefore this was not an interesting metric. We are currently looking at the time to diagnosis of the anomaly to see if there are any interesting trends there. Trends appear to indicate that in some cases the crew took more time to solve scenarios with Daphne than without, especially in Phase 2, but we attribute this to some logon issues we encountered towards the end of the mission. Our perception is that the diagnosis was very fast both with and without Daphne, suggesting that the anomaly scenarios were “too easy”. The subjects did spend considerable time conducting the procedures to resolve the anomalies, but that was not the focus of our study.

Concerning cognitive workload, we observed a small difference between the TLX scores with Daphne (23.42±5.17) and without Daphne (25.20±3.54) but this difference was not significant with only 4 subjects (p=0.42). It remains to be seen if the effect becomes significant with the full 16 subjects. It is also interesting to note that these scores are very low compared to the scores obtained in the laboratory experiments. This could be due to the shorter training and less experience of the participants of the lab experiments compared to the HERA crew. This is also consistent with the explanation that the scenarios were too “easy” for the HERA crew.

Concerning situational awareness, we observed the opposite trend compared to the lab experiments, although in both cases the effects are not significant. In this case, SART scores were slightly lower with Daphne (20.46±3.35) than without Daphne (21.21±3.50), for a p-value of 0.41. We note that the effects were not consistent among crewmembers, and the overall result is driven by one of the 4 crewmembers, who reported a significantly higher level of situational awareness without Daphne vs. with Daphne. We are looking in more detail into the components of the SART score (understanding, demand, supply) because the trends appear to be conflicting there too. While the literature has reported cases where using some kind of assistance results in a decrease of situational awareness (e.g., autopilots in commercial aircraft), this may be due to other factors (e.g., minor technical issues with the VA) as well. More research is needed to understand this – we hope that missions 2,3,4 will help us elucidate this issue. We also note that crewmembers mentioned in the exit debrief interviews that adding explanations would likely significantly increase situational awareness with Daphne, and this is something that we plan on testing in our lab experiments 2 and 3.

Concerning trust, the results were similar to the ones from the lab experiments, showing high trust in the VA across the board. In the exit interviews, crewmembers indicated that they trusted Daphne “right away”, after the first few of her recommendations were proven to be correct.

Some of the insights we got from the exit interviews are as follows. The crew generally exhibited strong interest in using VAs for anomaly resolution, and enjoyed using Daphne. Most of them showed an interest in the social aspects of VA and mentioned that they “attributed a personality to Daphne” and “talked about her as if she were another crewmember”. They all mentioned establishing trust very quickly, once and for all, thanks to Daphne “getting it right the first 3 times or so”. Almost none of them found the question answering capabilities essential because they were “going for speed” and didn’t feel like they needed to ask any questions. However, all of them mentioned that question answering would be very useful in cases where Daphne recommended more than one diagnosis with the same confidence level. Moreover, crewmembers expressed an interest in the more interactive diagnosis and advanced explanations capabilities we are currently developing as something that would “significantly increase the usefulness” of the tool. Finally, they all confirmed that the scenarios generally felt very easy to diagnose and adding some more complexity would make it more interesting and fun.

As for next steps, in Year 4 we will support the remaining missions of C6, thus finalizing Specific Aim 3. We plan on publishing a journal paper with the results. Otherwise, most of the activity will focus on achieving Specific Aim 2: we will finalize the development of the enhanced VA capabilities (explanations and mixed initiative) and conduct lab experiments 2 and 3.

Bibliography Type: Description: (Last Updated: 03/17/2022) 

Show Cumulative Bibliography Listing
 
Abstracts for Journals and Proceedings Selva D, Josan P, Dutta P, Abbott R, Viros i Martin A, York K, Dunbar B, Wong RKW, Diaz Artiles A. "Virtual assistant for anomaly resolution In long duration exploration missions: Baseline effects on performance, cognitive workload, and situational awareness." 2022 NASA Human Research Program Investigators' Workshop, Virtual. February 7-10, 2022.

Abstracts. 2022 NASA Human Research Program Investigators' Workshop, Virtual. February 7-10, 2022 (Abstract #1133-000353). , Feb-2022

Abstracts for Journals and Proceedings Josan P, Dutta P, Abbott R, Viros A, York K, Dunbar B, Wong R, Selva D, Diaz-Artiles A. "Results from first laboratory testing of virtual assistant Daphne-AT. " 2022 NASA Human Research Program Investigators' Workshop, Virtual. February 7-10, 2022.

Abstracts. 2022 NASA Human Research Program Investigators' Workshop, Virtual. February 7-10, 2022 (Abstract #1133-000351). , Feb-2022

Abstracts for Journals and Proceedings Dutta P, Kaur PK, Viros A, York K, Abbott R, Dunbar B, Wong R, Diaz-Artiles A, Selva D. "Virtual assistants for anomaly treatment – lessons learned and path forward." 2022 NASA Human Research Program Investigators' Workshop, Virtual. February 7-10, 2022.

Abstracts. 2022 NASA Human Research Program Investigators' Workshop, Virtual. February 7-10, 2022 (Abstract #1133-000410). , Feb-2022

Project Title:  HCAAM VNSCOR: Virtual Assistant for Spacecraft Anomaly Treatment During Long Duration Exploration Missions Reduce
Images: icon  Fiscal Year: FY 2021 
Division: Human Research 
Research Discipline/Element:
HRP HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Start Date: 03/06/2019  
End Date: 03/05/2023  
Task Last Updated: 01/06/2021 
Download report in PDF pdf
Principal Investigator/Affiliation:   Selva, Daniel  Ph.D. / Texas A&M University 
Address:  Aerospace Engineering Department 
701 Ross St 3141 TAMU 
College Station , TX 77843-0001 
Email: dselva@tamu.edu 
Phone: 607-255-6351  
Congressional District: 17 
Web:  
Organization Type: UNIVERSITY 
Organization Name: Texas A&M University 
Joint Agency:  
Comments:  
Co-Investigator(s)
Affiliation: 
Diaz Artiles, Ana  Ph.D. Texas A&M Engineering Experiment Station 
Dunbar, Bonnie  Ph.D. Texas A&M Engineering Experiment Station 
Wong, Raymond  Ka Wai Ph.D. Texas A & M, College Station 
Project Information: Grant/Contract No. 80NSSC19K0656 
Responsible Center: NASA JSC 
Grant Monitor: Whitmire, Alexandra  
Center Contact:  
alexandra.m.whitmire@nasa.gov 
Solicitation / Funding Source: 2017-2018 HERO 80JSC017N0001-BPBA Topics in Biological, Physiological, and Behavioral Adaptations to Spaceflight. Appendix C 
Grant/Contract No.: 80NSSC19K0656 
Project Type: GROUND 
Flight Program:  
TechPort: Yes 
No. of Post Docs:
No. of PhD Candidates:
No. of Master's Candidates:
No. of Bachelor's Candidates:
No. of PhD Degrees:
No. of Master's Degrees:
No. of Bachelor's Degrees:
Human Research Program Elements: (1) HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Human Research Program Risks: (1) HSIA:Risk of Adverse Outcome Due to Inadequate Human Systems Integration Architecture (IRP Rev L)
Human Research Program Gaps: (1) HSIA-101:We need to identify the Human Systems Integration (HSI) – relevant crew health and performance outcomes, measures, and metrics, needed to characterize and mitigate risk, for future exploration missions (IRP Rev L)
(2) HSIA-201:We need to evaluate the demands of future exploration habitat/vehicle systems and mission scenarios (e.g. increased automation, multi-modal communication) on individuals and teams, and determine the risks these demands pose to crew health and performance (IRP Rev L)
(3) HSIA-401:We need to determine how HSI can be applied in the vehicle/habitat and computer interface Design Phase to mitigate potential decrements in operationally-relevant performance (e.g. problem-solving, execution procedures), during increasingly earth-independent, future exploration missions (including in-mission and at landing) (IRP Rev L)
(4) HSIA-701:We need to determine how human-automation-robotic systems can be optimized for effective enhancement and monitoring of crew capabilities, health, and performance, during increasingly earth-independent, future exploration missions (including in-mission and at landing) (IRP Rev L)
Task Description: This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

The research objective of this proposal is to investigate the impact of using Virtual Assistants (VA) to support crew members in the context of anomaly treatment during Long Duration Exploration Missions (LDEM), when ground support will be limited. A VA will be developed building upon the software architecture from existing VAs developed by the Principal Investigator (PI) for similar purposes. The VA will provide support for various aspects of anomaly treatment, including detecting and diagnosing the anomaly, as well as recommending a course of action. It will also have the ability to take initiative in the dialog with the user (mixed-initiative mode), and the ability to provide explanations for its actions. The impact of the VA on performance, cognitive workload, situational awareness, and trust, will be assessed through a set of three experiments with human subjects in a laboratory environment. The first experiment will establish the baseline impact (master-slave, no explanations), and subsequent experiments will study the effect of switching to the mixed-initiative mode and adding explanations. The system will also be deployed and tested in the Human Exploration Research Analog (HERA) analog environment.

Research Impact/Earth Benefits: This project will provide standards and guidelines that will help NASA design similar virtual assistants to support astronauts during future long duration exploration missions. Such standards and guidelines will concern both the functionality and the user interface of the virtual assistant.

Task Progress & Bibliography Information FY2021 
Task Progress: In Year 2 of this project, we have completed the first round of testing of the Daphne-AT prototype and conducted a pilot study. We worked closely with HERA throughout the integration and testing of our study as part of the C6 campaign. We are now getting ready for a final training documentation delivery to NASA this month, and a first system-level test of our study with the entire team at Johnson Space Center (JSC), assuming the COVID situation permits it.

To be able to do experiments at Texas A&M, we developed an immersive 3d user interface to simulate the Human Exploration Research Analog (HERA) habitat during our experiments at TAMU. The goal was for the simulator to have reasonable fidelity in terms of the general dimensions and shape as well as the main components present in the habitat related to the ECLSS system. The simulator is basically a first-person game developed in Unreal Engine 4 (UE4). All of the major components of the HERA Environmental Control and Life Support Subsystem (ECLSS) system are present and the user can interact with them (e.g., press buttons, open/close panels) as required by the anomaly resolution procedures. More details about this simulator are provided in

Woodruff R, Beebe N, Josan PK, Wong RKW, Dunbar BJ, Selva D, Diaz-Artiles A. (2021). 3D Interactive Model of HERA to support ECLSS anomaly resolution using a Virtual Assistant. 2021 IEEE Aerospace Conference Proceedings.

After validation of the UE4 simulator, we designed and conducted a pilot study of the first “Baseline” prototype of the assistant. It included five male subjects within the age range of 22-40 years old, and with a Standard Deviation (SD) of ±7.2. Prior to the experimental sessions, subjects were provided a 2-hour virtual training session about all the experiment elements (Daphne-AT, ECLSS subsystems, 3D virtual HERA environment, and experimental protocols) and they were guided through an example anomaly scenario. After the training, subjects participated in two in-person experiment sessions: one with Daphne-AT and the other one without Daphne-AT (counterbalanced among subjects). These sessions were conducted in two consecutive days at approximately the same time of the day. In each one of the sessions, the subjects solved four anomalies. Once the experiment sessions were over, the subjects completed a set of surveys.

Dependent variables included performance, cognitive workload, situational awareness, and trust in the context of ECLSS anomaly treatment, with and without the use of a VA. Performance is described as an individual’s ability to successfully treat the anomaly. On average, subjects solved 3.2 ± 0.37 anomalies using Daphne-AT and 2.0 ± 0.without Daphne-AT. Based on these limited data, our preliminary results support the hypothesis that the use of Daphne-AT will increase performance in the context of ECLSS anomaly resolution operations.

The NASA Task Load Index (TLX) survey was used to measure the effect of using Daphne-AT on cognitive workload. NASA TLX is an evaluation technique developed by NASA to assess relative importance of six pre-determined factors in determining how much workload a subject experience during a particular task, in this case, the task of anomaly treatment. These six factors include mental demand, physical demand, temporal demand, perceived performance, perceived effort, and frustration level. On average, subjects indicated higher cognitive workload during the sessions without Daphne-AT (57.2 ± 5.78 with Daphne-AT and 66 ± 7.31 without Daphne). These preliminary results support the hypothesis that the use of Daphne-AT will decrease cognitive workload. More details about this pilot study can be found in:

Josan PK, Dutta P, Woodruff R, Beebe N, York K, Balcells-Quintana O, Kluis L, Viros-i-Martin A, Dunbar BJ, Wong RKW, Selva D, Diaz-Artiles A. (2021). Experimental Design & Pilot Testing for ECLSS Anomaly Resolution using Daphne-AT Virtual Assistant. 2021 IEEE Aerospace Conference Proceedings.

As a result of the pilot testing and HERA integration activities, we have substantially refined the VA interface and added functionality to the back end. Specifically: 1) We adapted the telemetry feed to the new version of the HERA Environmental Control and Life Support System (ECLSS) simulator (now called Habitat System Simulator or HSS). 2) We updated the anomaly database to include more information for each anomaly (e.g., a time signature). 3) We expanded the knowledge base with more information about ECLSS anomalies, their associated risks, and how to resolve them. 4) We expanded the anomaly detection functions beyond basic thresholding to be able to detect outliers in the telemetry feed. 5) We made slight changes to the knowledge graph and procedures to adapt to HERA C6 schedule constraints.

In terms of publications, we had two papers accepted at the 2021 IEEE Aerospace Conference which will be presented in March. We also gave a presentation at the 2020 American Institute of Aeronautics and Astronautics (AIAA) ASCEND conference (Ed. note 1/7/2021: see FY2020 Bibliography in Task Book report) and will have one talk presentation and one poster presentation at the 2021 Human Research Program workshop in February.

Bibliography Type: Description: (Last Updated: 03/17/2022) 

Show Cumulative Bibliography Listing
 
Papers from Meeting Proceedings Woodruff R, Beebe N, Josan PK, Wong RKW, Dunbar BJ, Selva D, Diaz-Artiles A. "A 3D Interactive Model of HERA to Support ECLSS Anomaly Resolution Using a Virtual Assistant." 2021 IEEE Aerospace Conference, Big Sky, MT, Virtual, March 6-13, 2021.

2021 IEEE Aerospace Conference, Big Sky, MT, Virtual, March 6-13, 2021. Meeting paper. https://doi.org/10.1109/AERO50100.2021.9438341 , Mar-2021

Papers from Meeting Proceedings Josan PK, Dutta P, Woodruff R, Beebe N, York K, Balcells-Quintana O, Kluis L, Viros-i-Martin A, Dunbar BJ, Wong RKW, Selva D, Diaz-Artiles A. "Experimental Design & Pilot Testing for ECLSS Anomaly Resolution Using Daphne-AT Virtual Assistant." 2021 IEEE Aerospace Conference, Big Sky, MT, Virtual, March 6-13, 2021.

2021 IEEE Aerospace Conference, Big Sky, MT, Virtual, March 6-13, 2021. Meeting paper. https://doi.org/10.1109/AERO50100.2021.9438497 , Mar-2021

Project Title:  HCAAM VNSCOR: Virtual Assistant for Spacecraft Anomaly Treatment During Long Duration Exploration Missions Reduce
Images: icon  Fiscal Year: FY 2020 
Division: Human Research 
Research Discipline/Element:
HRP HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Start Date: 03/06/2019  
End Date: 03/05/2023  
Task Last Updated: 01/06/2020 
Download report in PDF pdf
Principal Investigator/Affiliation:   Selva, Daniel  Ph.D. / Texas A&M University 
Address:  Aerospace Engineering Department 
701 Ross St 3141 TAMU 
College Station , TX 77843-0001 
Email: dselva@tamu.edu 
Phone: 607-255-6351  
Congressional District: 17 
Web:  
Organization Type: UNIVERSITY 
Organization Name: Texas A&M University 
Joint Agency:  
Comments:  
Co-Investigator(s)
Affiliation: 
Diaz Artiles, Ana  Ph.D. Texas A&M Engineering Experiment Station 
Dunbar, Bonnie  Ph.D. Texas A&M Engineering Experiment Station 
Wong, Raymond  Ka Wai Ph.D. Texas A & M, College Station 
Project Information: Grant/Contract No. 80NSSC19K0656 
Responsible Center: NASA JSC 
Grant Monitor: Williams, Thomas  
Center Contact: 281-483-8773 
thomas.j.will1@nasa.gov 
Solicitation / Funding Source: 2017-2018 HERO 80JSC017N0001-BPBA Topics in Biological, Physiological, and Behavioral Adaptations to Spaceflight. Appendix C 
Grant/Contract No.: 80NSSC19K0656 
Project Type: GROUND 
Flight Program:  
TechPort: Yes 
No. of Post Docs:
No. of PhD Candidates:
No. of Master's Candidates:
No. of Bachelor's Candidates:
No. of PhD Degrees:
No. of Master's Degrees:
No. of Bachelor's Degrees:
Human Research Program Elements: (1) HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Human Research Program Risks: (1) HSIA:Risk of Adverse Outcome Due to Inadequate Human Systems Integration Architecture (IRP Rev L)
Human Research Program Gaps: (1) HSIA-101:We need to identify the Human Systems Integration (HSI) – relevant crew health and performance outcomes, measures, and metrics, needed to characterize and mitigate risk, for future exploration missions (IRP Rev L)
(2) HSIA-201:We need to evaluate the demands of future exploration habitat/vehicle systems and mission scenarios (e.g. increased automation, multi-modal communication) on individuals and teams, and determine the risks these demands pose to crew health and performance (IRP Rev L)
(3) HSIA-401:We need to determine how HSI can be applied in the vehicle/habitat and computer interface Design Phase to mitigate potential decrements in operationally-relevant performance (e.g. problem-solving, execution procedures), during increasingly earth-independent, future exploration missions (including in-mission and at landing) (IRP Rev L)
(4) HSIA-701:We need to determine how human-automation-robotic systems can be optimized for effective enhancement and monitoring of crew capabilities, health, and performance, during increasingly earth-independent, future exploration missions (including in-mission and at landing) (IRP Rev L)
Task Description: This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

The research objective of this proposal is to investigate the impact of using Virtual Assistants (VA) to support crew members in the context of anomaly treatment during Long Duration Exploration Missions (LDEM), when ground support will be limited. A VA will be developed building upon the software architecture from existing VAs developed by the Principal Investigator (PI) for similar purposes. The VA will provide support for various aspects of anomaly treatment, including detecting and diagnosing the anomaly, as well as recommending a course of action. It will also have the ability to take initiative in the dialog with the user (mixed-initiative mode), and the ability to provide explanations for its actions. The impact of the VA on performance, cognitive workload, situational awareness, and trust, will be assessed through a set of three experiments with human subjects in a laboratory environment. The first experiment will establish the baseline impact (master-slave, no explanations), and subsequent experiments will study the effect of switching to the mixed-initiative mode and adding explanations. The system will also be deployed and tested in the Human Exploration Research Analog (HERA) analog environment.

Research Impact/Earth Benefits: This project will provide standards and guidelines that will help NASA design similar virtual assistants to support astronauts during future long duration exploration missions. Such standards and guidelines will concern both the functionality and the user interface of the virtual assistant.

Task Progress & Bibliography Information FY2020 
Task Progress: This project is one of seven projects in the Virtual NASA Specialized Center of Research (VNSCOR) for Human Capabilities Assessments for Autonomous Missions (HCAAM). The goal of our project is to develop a virtual assistant to help astronauts with anomaly resolution in long duration exploration missions.

In the first year of this project, we have completed a project definition phase in which we have more crisply defined the scientific objectives and the technical approach to design the virtual assistant. We started by selecting the Environmental Control and Life Support Subsystem (ECLSS) as the technical scope for the virtual assistant. We then proceeded to elicit high level requirements and use cases and perform the early design of the virtual assistant, which we are getting ready to test in our laboratory at Texas A&M University in the next month.

Specifically, in this first year, we have accomplished the following: 1) We have defined the high-level requirements and overall software architecture of the virtual assistant. 2) We have defined how the virtual assistant will interface with the Human Exploration Research Analog (HERA) infrastructure. 3) We have developed the first very basic prototype of the virtual assistant which includes: a) a simple anomaly database; b) a simple database containing knowledge about how the ECLSS system works in HERA and how to perform some maintenance and repair operations; c) an interface with the HERA simulated telemetry feed, which provides a data stream with all the measurements of the ECLSS subsystem; d) a web-based graphical user interface containing a main plot window displaying the telemetry feed and a chat box to chat with the virtual assistant; e) the natural language processing layer that the assistant uses to process the natural language requests; f) the machine learning layer that is used to understand the intent of the question and send it to the relevant part of the software, i.e., the one that knows how to answer it; f) the first version of the back end of the software, which is precisely the part of the software that performs the calculations needed to answer the questions and requests from the user. While this first prototype is not sophisticated in terms of the breadth and depth of questions it can answer, it still has most of the main functionality expected of the assistant. Specifically: 1) it warns the user when a measurement from the telemetry feed exceed some user-defined thresholds; 2) it also performs statistical analysis on the data stream and warns the user if the measurement is considered an outlier, even if it is within the user-defined thresholds; 3) once an anomaly is detected (e.g., increase in CO2 concentration), it provides the users with some possible root causes for it, in order of likelihood (e.g., a malfunctioning filter or a leak); 4) once the anomaly has been diagnosed (faulty filter), it provides the user with relevant procedures to reconfigure the system (how to change the CO2 filter).

In parallel of the software development, we have also started planning and designing the experiments. We are currently getting ready to start a pilot experiment at Texas A&M in order to refine both the software implementation and the experimental design. The results of this pilot experiment will help us get ready for the HERA campaign, which is currently scheduled for August 2020.

Since the beginning of the project, we have been working with the HERA team to prepare for the experiment. We obtained approval from the Institutional Review Boards of both NASA and Texas A&M. We provided our input to the experiment support specialist to create our science requirements document and are currently working on the investigator working group, getting ready for a software delivery to NASA by March.

We have also been in constant communication with the rest of the VNSCOR to ensure compatibility of our experiments and try to make the most of any potential synergies.

Finally, in terms of publications, we are about to present a peer-reviewed full paper at the 2020 American Institute of Aeronautics and Astronautics (AIAA) Scitech conference. We also have two poster presentations at the 2020 Human Research Program workshop later this month.

In the next few months, we will keep adding functionality to the assistant so that it can answer more questions better and provide more useful information to the user. We anticipate that some rework of the telemetry feed interface will be necessary to adapt to some changes on the HERA side. We will also refine the user interface with the results from the pilot. We are getting ready to conduct a preliminary design review in March before delivery to HERA. Between March and August 2020 we will work on integrating our software with HERA. We will conduct a Critical Design Review in June 2020 and an Operational Readiness Review in August 2020, right before ingress. Between August 2020 and September 2021 we will support the 4 missions of the C6 HERA campaign.

Bibliography Type: Description: (Last Updated: 03/17/2022) 

Show Cumulative Bibliography Listing
 
Papers from Meeting Proceedings Dutta P, Balcells Quintana O, Viros A, Whittle RS, Poonampreet KJ, Beebe N, Dunbar BJ, Wong RK, Diaz-Artiles A, Diaz-Artiles SD, Ana Selva D. "Virtual Assistant for Anomaly Treatment in Long Duration Exploration Missions." Presented at the 2020 AIAA SciTech Forum, Orlando, FL, January 6-10, 2020.

Paper AIAA 2020-2255. 2020 AIAA SciTech Forum, Orlando, FL, January 6-10, 2020. https://doi.org/10.2514/6.2020-2255 , Jan-2020

Project Title:  HCAAM VNSCOR: Virtual Assistant for Spacecraft Anomaly Treatment During Long Duration Exploration Missions Reduce
Images: icon  Fiscal Year: FY 2019 
Division: Human Research 
Research Discipline/Element:
HRP HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Start Date: 03/06/2019  
End Date: 03/05/2023  
Task Last Updated: 04/22/2019 
Download report in PDF pdf
Principal Investigator/Affiliation:   Selva, Daniel  Ph.D. / Texas A&M University 
Address:  Aerospace Engineering Department 
701 Ross St 3141 TAMU 
College Station , TX 77843-0001 
Email: dselva@tamu.edu 
Phone: 607-255-6351  
Congressional District: 17 
Web:  
Organization Type: UNIVERSITY 
Organization Name: Texas A&M University 
Joint Agency:  
Comments:  
Co-Investigator(s)
Affiliation: 
Diaz Artiles, Ana  Ph.D. Texas A&M Engineering Experiment Station 
Dunbar, Bonnie  Ph.D. Texas A&M Engineering Experiment Station 
Wong, Raymond  Ka Wai Ph.D. Texas A & M, College Station 
Project Information: Grant/Contract No. 80NSSC19K0656 
Responsible Center: NASA JSC 
Grant Monitor: Williams, Thomas  
Center Contact: 281-483-8773 
thomas.j.will1@nasa.gov 
Solicitation / Funding Source: 2017-2018 HERO 80JSC017N0001-BPBA Topics in Biological, Physiological, and Behavioral Adaptations to Spaceflight. Appendix C 
Grant/Contract No.: 80NSSC19K0656 
Project Type: GROUND 
Flight Program:  
TechPort: Yes 
No. of Post Docs:  
No. of PhD Candidates:  
No. of Master's Candidates:  
No. of Bachelor's Candidates:  
No. of PhD Degrees:  
No. of Master's Degrees:  
No. of Bachelor's Degrees:  
Human Research Program Elements: (1) HFBP:Human Factors & Behavioral Performance (IRP Rev H)
Human Research Program Risks: (1) HSIA:Risk of Adverse Outcome Due to Inadequate Human Systems Integration Architecture (IRP Rev L)
Human Research Program Gaps: (1) HSIA-101:We need to identify the Human Systems Integration (HSI) – relevant crew health and performance outcomes, measures, and metrics, needed to characterize and mitigate risk, for future exploration missions (IRP Rev L)
(2) HSIA-201:We need to evaluate the demands of future exploration habitat/vehicle systems and mission scenarios (e.g. increased automation, multi-modal communication) on individuals and teams, and determine the risks these demands pose to crew health and performance (IRP Rev L)
(3) HSIA-401:We need to determine how HSI can be applied in the vehicle/habitat and computer interface Design Phase to mitigate potential decrements in operationally-relevant performance (e.g. problem-solving, execution procedures), during increasingly earth-independent, future exploration missions (including in-mission and at landing) (IRP Rev L)
(4) HSIA-701:We need to determine how human-automation-robotic systems can be optimized for effective enhancement and monitoring of crew capabilities, health, and performance, during increasingly earth-independent, future exploration missions (including in-mission and at landing) (IRP Rev L)
Task Description: This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

The research objective of this proposal is to investigate the impact of using Virtual Assistants (VA) to support crew members in the context of anomaly treatment during Long Duration Exploration Missions (LDEM), when ground support will be limited. A VA will be developed building upon the software architecture from an existing VA developed by the Principal Investigator. The VA will provide support for various aspects of anomaly treatment, including detecting and diagnosing the anomaly, as well as recommending a course of action. It will also have the ability to take initiative in the dialog with the user (mixed-initiative mode), and the ability to provide explanations for its actions. The impact of the VA on performance, cognitive workload (CW), situational awareness (SA), and trust, will be assessed through a set of three experiments with human subjects in a laboratory environment. The first experiment will establish the baseline impact (master-slave, no explanations), and subsequent experiments will study the effect of switching to the mixed-initiative mode and adding explanations. Finally, the system will be deployed in an analog environment.

Research Impact/Earth Benefits:

Task Progress & Bibliography Information FY2019 
Task Progress: New project for FY2019.

Bibliography Type: Description: (Last Updated: 03/17/2022) 

Show Cumulative Bibliography Listing
 
 None in FY 2019