NOTE: End date changed to 02/28/2025 per A. Beitman/JSC (Ed., 1/25/23)

The research objective of this proposal is to investigate the impact of using Virtual Assistants (VA) to support crewmembers in the context of anomaly treatment during Long Duration Exploration Missions (LDEM), when ground support will be limited. A VA will be developed building upon the software architecture from existing VAs developed by the Principal Investigator (PI) for similar purposes. The VA will provide support for various aspects of anomaly treatment, including detecting and diagnosing the anomaly, as well as recommending a course of action. It will also have the ability to take initiative in the dialog with the user (mixed-initiative mode), and the ability to provide explanations for its actions. The impact of the VA on performance, cognitive workload, situational awareness, and trust, will be assessed through two experiments with human subjects in a laboratory environment. The first experiment will establish the baseline impact (master-slave, no explanations), and the second experiments will study the effect of adding explanations. The system will also be deployed and tested in the Human Exploration Research Analog (HERA) analog environment.

In addition, the project has developed recommendations for standards and guidelines for the development of such virtual assistants, which could be applicable as well for the use cases mentioned above.

The VA is continuously reading sensor data from a simulated “telemetry feed” produced by the Habitat System Simulator (HSS) software developed by NASA Johnson Space Center. Daphne-AT will provide a list of possible anomalies for the symptoms selected, sorted by a qualitative likelihood score (e.g., Very Likely, Somewhat Likely). To provide these recommendations, Daphne-AT has a knowledge graph containing information about all the habitat subsystems, anomalies that can happen to them, their symptoms, and the procedures for resolving them. The user can then select one of the anomalies identified by Daphne-AT as the most likely, and Daphne-AT will suggest a procedure to resolve it. The user can follow along the procedure, checking steps as they are completed. The steps of the procedures also contain hyperlinks to figures of the relevant components or subsystems.

At any point during the task, the user can chat with the agent via a natural language interface. The baseline version of the VA used a template-based restricted domain question answering system with language models developed in-house. Using what was a state-of-the-art approach before the arrival of large language models, the question answering system classified the user question into one of N known types of questions using a small neural network. Then, the agent extracted the parameters expected for that question type from the user question and produced a query to the corresponding back-end service (e.g., a query to the knowledge graph). Finally, it inserted the answer in a predefined answer template for that question type and fed that back to the user. This approach was extremely fast and consistent, and reliable for known types of questions, but it had two major drawbacks: it struggled with parameter extraction in the presence of typos or slightly different formulations, and it could only handle the known types of questions, so it was not very scalable.

An Enhanced version of the VA was developed and used for Lab Experiment 2 and HERA Campaign 7. The main change with respect to the Baseline version is the incorporation of additional information in the diagnosis, serving as explanations of the likelihood scores. In addition to the qualitative likelihood scores shown in the Baseline version, the Enhanced version shows a confidence level in the diagnosis. In addition to the changes in the diagnosis service, a significantly improved chat agent was implemented in the Enhanced VA that leverages GPT-4 and Retrieval Augmented Generation (RAG) techniques to provide much more flexible and scalable question answering.

The specific research aims of this project were as follows:

1. Specific Aim 1 – Baseline VA: To measure the impact of the baseline VA (with question answering, but without the ability to engage in dialogues or take initiative) and its various skills (anomaly detection, diagnosis, resolution) on performance, CW, SA, and trust in a laboratory environment. 2. Specific Aim 2 – Enhanced VA: To measure the impact of advanced capabilities of the VA in terms of providing explanations on performance, CW, SA, and trust in a laboratory environment. 3. Specific Aim 3 – Analog: To deploy and validate the system in an analog environment.

Two lab experiments were conducted to support specific aims 1 and 2 described above. In Experiment 1, we studied and characterized the use of the baseline VA (without the ability to provide advanced explanations) on human performance, CW, SA, and trust. We conducted a counterbalanced, within-subjects experiment where subjects were asked to resolve multiple anomalies that are representative of a flight scenario, with and without the help of our VA. Results showed a significant positive impact of the VA on human performance, workload, and situational awareness. The results were published in a paper in the Journal of Aerospace Information Systems.

In Experiment 2, we studied the impact of giving the VA the ability to provide explanations for its actions under various levels of scenario uncertainty and VA accuracy. Subjects were divided into three groups of different VA accuracy (high, medium, low). Each subject resolved two sets of 8 anomalies, one with explanations and one without. Similarly, half of the anomalies were of high uncertainty and the other half of low uncertainty. All independent variables were randomized and counterbalanced appropriately. We measured performance (# of anomalies correctly resolved, time to diagnosis), workload (TLX, Bedford), situational awareness (SART), and trust in automation. Results showed that the explanations had a positive significant effect on performance, situational awareness, and trust without affecting workload. The results were published in two papers in the Journal of Cognitive Engineering and Decision Making, one focusing on the effect of the explanations overall, and the other focusing on uncertainty.

A third lab experiment had been proposed in the original proposal as part of Specific Aim 2, but it was decided after seeing the results of the first experiments not to pursue it. This third experiment was intended to focus on the natural language interface –specifically on giving the ability to take initiative in the dialogue. However, we observed in both lab and analog experiments that the users did not use the question answering system at all to perform the tasks. Therefore, we decided to postpone this study to the future when we understand better what kinds of anomaly scenarios prompt the user to rely more on the question answering system.

In addition to the lab experiments, two analog experiments were performed as part of HERA Campaigns 6 and 7 to achieve Specific Aim 3. In both, a similar experimental design to that of Lab Experiment 1 was used, where the crewmembers solved some anomaly scenarios with and without the VA, and we compared the outcomes in terms of performance, SA, and CW.

The baseline VA was characterized in HERA Campaign 6. Unlike in Lab Experiment 1, we found no significant effects of the VA on any metrics, except that attentional demand was significantly higher with VA than without. Crewmembers mentioned they found the anomaly scenarios easy to solve with the emergency chart (the aid provided in the no VA scenarios), so the VA was not really needed. Overall, we found lower workload, higher performance, and higher SA in C6 compared to Lab Experiment 1.

In HERA Campaign 7, the enhanced VA was used. Again, no significant effects of the VA on any metrics were found. Comparing the results with those of HERA Campaign 6, we observed Higher Understanding (a SART subscale) in C7 than in C6, but also higher attentional demand (another SART subscale) and mental demand (a TLX subscale). These results are partially consistent with those of Lab Experiment 2, since they suggest an improvement in SA due to the explanations. However, in the analog case, it comes at the cost of an increase in mental demand, which was not observed in the lab.

Overall, these four studies show potential for the use of VAs to support astronauts with anomaly resolution in LDEM but also show that the gains are strongly dependent on context, particularly for diagnosing anomalies. When it comes to diagnosis, for simpler known anomalies, a simple emergency chart like the ones used today is a sufficient aid. The VA is likely to be most useful for diagnosing more complex, unknown anomalies.

NOTE: End date changed to 02/28/2025 per A. Beitman/JSC (Ed., 1/25/23)

The research objective of this proposal is to investigate the impact of using Virtual Assistants (VA) to support crewmembers in the context of anomaly treatment during Long Duration Exploration Missions (LDEM), when ground support will be limited. A VA will be developed building upon the software architecture from existing VAs developed by the Principal Investigator (PI) for similar purposes. The VA will provide support for various aspects of anomaly treatment, including detecting and diagnosing the anomaly, as well as recommending a course of action. It will also have the ability to take initiative in the dialog with the user (mixed-initiative mode), and the ability to provide explanations for its actions. The impact of the VA on performance, cognitive workload, situational awareness, and trust, will be assessed through a set of three experiments with human subjects in a laboratory environment. The first experiment will establish the baseline impact (master-slave, no explanations), and subsequent experiments will study the effect of switching to the mixed-initiative mode and adding explanations. The system will also be deployed and tested in the Human Exploration Research Analog (HERA) analog environment.

Specific Aim 3: HERA C7 campaign Concerning Specific Aim 3, we prepared for and supported the HERA C7 campaign, which was completed in December 2024. Data analysis is in progress as of January 10, 2025. In preparation for the campaign, we conducted two major tasks:

We added several new “complex” anomaly scenarios, consisting of two and in some cases 3 anomalies. These anomalies were cascading in most cases, with the second one being phased in a few seconds or minutes after the first one. In a few cases, the individual anomalies were simultaneous. These scenarios were meant to simulate both independent and common-cause failures, as well as cases where the first failure causes the second one. The new scenarios were designed by HERA personnel with input from us and were largely combinations of existing (C6) scenarios. The design of these anomaly scenarios was subject to operational constraints on duration, number of operators, etc. Developing these scenarios necessitated incorporating them into Daphne’s knowledge graph and making changes to the Daphne diagnosis skill to allow the calculation of likelihood scores for these complex scenarios.

We also used the “enhanced” version of Daphne in C7. Compared to the “baseline” version used in C6, the enhanced version had the following added functionalities: 1) provide explanations for the likelihood scores of each anomaly; 2) a more advanced question-answering system leveraging large language models (GPT4).

With data from the first 3 missions (N=12), the major findings from the HERA C7 campaign are relatively similar to those of the C6 campaign:

–No significant differences in any of the main measures –Time to diagnosis (min): 9.29±1.48 (without VA) vs 8.51±1.73 (with VA) –Situational Awareness Rating Technique (SART) 25.33±2.10 (without VA) vs 24.69±1.65 (with VA) –NASA Task Load Index (TLX) 30.13±3.21 (without VA) vs 29.02±2.99 (with VA)

Insights from the exit interviews are largely consistent with those of C6. By and large, subjects thought the scenarios became easy and repetitive towards the end of the mission, due to a lack of variety. Indeed, many scenarios designed by HERA were different combinations of the same few C6 scenarios (CDRA failure, cabin fan failure, MOXIE failure).

All subjects established trust in Daphne very quickly and cited the tool’s accuracy as the primary factor for this. Some subjects also identified Daphne’s ability to provide explanations for its diagnoses as a positive factor that contributed to the establishment of trust.

Almost none of them found the question-answering capabilities essential because the task was clear, they were trying to finish the task as quickly as possible, and they didn’t feel like they needed to ask any questions. Some subjects did mention that question-answering was more useful in cases where Daphne recommended more than one diagnosis with the same confidence level, and thought it would be even more useful if the anomaly was completely new and unknown.

Overall, crewmembers felt that in more complex scenarios, a more iterative approach to diagnosis with Daphne asking questions to the crewmembers to try to gather more evidence and refine its diagnosis would be useful.

Note that detailed data regarding the interactions between the crewmembers and Daphne has not been analyzed yet.

Guidelines We have developed a list of 20+ potential guidelines. Each guideline consists of a short title, a recommendation, and some justification/evidence. An example guideline is provided below.

Short title: On-demand adaptive explanations Recommendation: AI agents should give succinct or on-demand explanations for users who already know and trust the tool and longer explanations for more novice users who are expected to not trust the tool yet.

Justification: More explanations for novices can accelerate training and develop trust, but long explanations can be frustrating and lead to a higher workload for advanced users (Bansal et al., 2021). In C7 exit interviews, several crewmembers mentioned that they don’t need a symptom comparison table because they already trust the model and they want to go as fast as possible.

The complete list of current guidelines is provided below. This list is still in progress and the guidelines are being more fully developed.

–Ranked anomaly scenarios: Provide a list of plausible anomaly causes ranked by likelihood –Symptom comparison table: Show a table comparing the signature of current vs potential anomalies –Diagnosis confidence: Agent should convey the degree of uncertainty in its diagnosis –Anomaly signature changes: Alert the user when the signature of the anomaly has changed –Automatic TM display: Automatically pull up time series for relevant parameters when an anomaly is detected –Potential misdiagnosis: Alert the user when asking to diagnose symptoms that were likely created by user actions –Agent information space: AI agent should have access to 1) the telemetry feed; 2) a database of the most recent N days of telemetry; 3) a knowledge base of subsystems, anomalies, procedures; 4) a historical anomaly database; 5) information about its own internal workings –Voice interface: Agent should have a voice interface to help the users, particularly with following procedures –Sufficient training: For proper agent trust, users should receive both hands-on training and theoretical training on 1) the tool and how it works internally; 2) the habitat systems –Latency of the QA system: The QA system shall be fast enough to provide immediate answers –Flexibility of the QA system: Use an LLM to increase flexibility and scalability. However, it should be a local LLM that can be downloaded and fine-tuned on a relevant dataset –Top-down Workflow: Users find a top-down workflow (detection, diagnosis, and resolution) intuitive, with the chatbot on the right –Evidence from other agents: Agent should incorporate additional evidence provided by the user and provide an updated diagnosis –Bayesian diagnosis: Agent should be able to update initial beliefs based on all evidence provided –Propose actions to reduce diagnostic uncertainty: Agent should recommend actions to gather additional evidence to refine the diagnosis in case of uncertainty –Agent learning: Agent should be able to learn from its own experience –Explanation types: The agent should provide explanations based on both expert knowledge and historical anomaly occurrence –Outlier detection: The agent should flag outlier signatures even if no thresholds are exceeded –Multiple anomalies: The agent should be able to deal with multiple simultaneous anomalies –Anomaly priority: In case of multiple anomalies, the agent should help the user prioritize the most important ones to attend to first –Full-wave diagnosis: The agent should diagnose anomalies based on the current and past values of the parameters in the telemetry feed (as opposed to the thresholds exceeded)

NOTE: End date changed to 02/28/2025 per A. Beitman/JSC (Ed., 1/25/23)

The research objective of this proposal is to investigate the impact of using Virtual Assistants (VA) to support crewmembers in the context of anomaly treatment during Long Duration Exploration Missions (LDEM), when ground support will be limited. A VA will be developed building upon the software architecture from existing VAs developed by the Principal Investigator (PI) for similar purposes. The VA will provide support for various aspects of anomaly treatment, including detecting and diagnosing the anomaly, as well as recommending a course of action. It will also have the ability to take initiative in the dialog with the user (mixed-initiative mode), and the ability to provide explanations for its actions. The impact of the VA on performance, cognitive workload, situational awareness, and trust, will be assessed through a set of three experiments with human subjects in a laboratory environment. The first experiment will establish the baseline impact (master-slave, no explanations), and subsequent experiments will study the effect of switching to the mixed-initiative mode and adding explanations. The system will also be deployed and tested in the Human Exploration Research Analog (HERA) analog environment.

Concerning Specific Aim 2, we concluded Lab Experiment 2 on the effect of explanations and have submitted the corresponding two journal papers. This experiment addresses the following research question: How do explanations affect human performance, trust, cognitive workload (CW), situational awareness (SA), user satisfaction, and self-confidence for different levels of agent accuracy and uncertainty in human-AI collaborative anomaly diagnosis? The first paper focuses on the effects of explanations for different levels of agent accuracy, while the second paper focuses on the effects of uncertainty.

The protocol for this experiment was as follows. Subjects start with a background survey and test and then they proceed to do two sessions, one in each explanation condition (order counterbalanced). In each session they work on 8 anomaly scenarios, 4 with high uncertainty and 4 with low uncertainty, in random order. After each anomaly, subjects fill out a confidence survey, a Jian survey, and a satisfaction survey. After each session, they also fill out a NASA Task Load Index (TLX), a Situational Awareness Rating Technique (SART) questionnaire, another Jian survey, and another satisfaction survey.

30 subjects were recruited and performed the experiment. The major findings from the experiment are as follows:

Effect of explanations (within-subjects comparison using surveys done after each condition) 1. Explanations improve trust (p<0.001) 2. Explanations improve #anomalies correctly diagnosed (p=0.0039), but do not significantly change the time to diagnosis (p=0.37) 3. Explanations improve SA (p=0.009) 4. Explanations slightly increase CW, but effect is not significant (p=0.108) 5. Explanations improve user satisfaction (p<0.001) 6. Explanations improve user confidence (p<0.001)

Effect of uncertainty (within-subjects comparison using surveys done after each anomaly) 1. (High) Uncertainty decreases trust (p<0.001) 2. (High) Uncertainty decreases #anomalies correctly diagnosed (p<0.001) and time to diagnosis (p<0.001) 3. (High) Uncertainty decreases user satisfaction (p<0.001) 4. (High) Uncertainty decreases user confidence (p<0.001)

The effect of uncertainty on CW and SA could not be measured due to the experimental design – they were only measured once with explanations and once without, averaging over the effects of uncertainty in both cases.

The effects of accuracy were not significant, presumably due to individual differences leading to large variability between subjects. The only exception is that accuracy significantly improved the number of anomalies correctly diagnosed (p<0.001). The interaction effect between explanations and accuracy was also not significant.

Concerning Specific Aim 3, we completed the C6 campaign and prepared for the C7 campaign.

Results from the HERA C6 campaign: The major findings from the HERA C6 campaign (N=16) are as follows: • All subjects correctly resolved all scenarios. • Primary results did now show significant effects of the VA on any metrics. • The only exception is that attentional demand (a component of SA) was significantly higher with VA than without (n=16, V=21, p=0.03). • For group scenarios only, CW was higher with Daphne (t (15) = -6.0207, p <0.001).

Some of the insights we got from the exit interviews are as follows. The crew generally exhibited strong interest in using VAs for anomaly resolution, and enjoyed using Daphne. Many subjects showed an interest in the social aspects of VA and mentioned that they “attributed a personality to Daphne” and “talked about her as if she were another crewmember”. They all mentioned establishing trust very quickly once and for all thanks to Daphne “getting it right the first 3 times or so”. Almost none of them found the question answering capabilities essential because they were “going for speed” and didn’t feel like they needed to ask any questions. However, many subjects mentioned that question answering would be very useful in cases where Daphne recommended more than one diagnosis with the same confidence level. Moreover, crewmembers expressed an interest in the more interactive diagnosis and advanced explanations capabilities we are currently developing as something that would “significantly increase the usefulness” of the tool. Finally, they all confirmed that the scenarios generally felt very easy to diagnose and adding some more complexity would make it more interesting and fun. Note that detailed data regarding the interactions between the crewmembers and Daphne has not been analyzed yet.

Preparation of HERA C7: In C7, we will test a more advanced version of the Daphne agent with improved self-explaining abilities while also testing more complex anomaly scenarios in which there are cascading and simultaneous failures. The timeline for the campaign is shown below. • Phase 1 – Group sessions to study the effect of time delay (crew allowed to talk to MCC) – MD1-4: Group study with and without VA at 0s time delay – 2x1h – MD5-10: Group study with and without VA at 5s time delay – 2x1h – MD11-12: Group study with and without VA at 1min time delay – 2x1h • Phase 2 – Individual sessions for main study (not allowed to talk to MCC) – Week 3: Individual training, 1h per crewmember, 2 simple scenarios, 1 with VA and 1 without – Weeks 4-6: 2 sessions per crewmember per week, 1 with VA and 1 without, 3x4x2x1h=24h

As shown above, the group scenarios from C6 have been replaced with a shorter study on the effect of cislunar time delays (5sec) compared to no delays or longer delays (1 min). For these scenarios, the crew will also be allowed to communicate with MCC, to gain insight into the roles a VA can play for cislunar operators given that MCC is available albeit with a non-negligible time delay of a few seconds.

The second phase is the main phase of individual scenarios and follows the same design as in C6, with the caveat that a (re)-training session is done for each crewmember in week 3 before they start their individual tests.

Finally, in the next year, we also plan on finalizing the 3 publications under review (which should finalize work related to Specific Aims 1 and 2) and submitting a fourth one on the results of C6.

The research objective of this proposal is to investigate the impact of using Virtual Assistants (VA) to support crewmembers in the context of anomaly treatment during Long Duration Exploration Missions (LDEM), when ground support will be limited. A VA will be developed building upon the software architecture from existing VAs developed by the Principal Investigator (PI) for similar purposes. The VA will provide support for various aspects of anomaly treatment, including detecting and diagnosing the anomaly, as well as recommending a course of action. It will also have the ability to take initiative in the dialog with the user (mixed-initiative mode), and the ability to provide explanations for its actions. The impact of the VA on performance, cognitive workload, situational awareness, and trust, will be assessed through a set of three experiments with human subjects in a laboratory environment. The first experiment will establish the baseline impact (master-slave, no explanations), and subsequent experiments will study the effect of switching to the mixed-initiative mode and adding explanations. The system will also be deployed and tested in the Human Exploration Research Analog (HERA) analog environment.

Concerning Specific Aim 2, we added the explanation capabilities to the VA and started data collection and analysis for Lab Experiment 2. This experiment addresses the following research question: How do explanations affect human performance, trust, cognitive workload (CW), situational awareness (SA), user satisfaction, and self-confidence for different levels of accuracy and uncertainty in human-AI collaborative anomaly diagnosis?

The protocol is as follows. Subjects start with a background survey and test, and then they proceed to do two sessions, one in each explanation condition (order counterbalanced). In each session, they work on 8 anomaly scenarios, 4 with high uncertainty and 4 with low uncertainty, in random order. After each anomaly, subjects fill out a confidence survey, a Jian survey, and a satisfaction survey. After each session, they also fill out a NASA Task Load Index (TLX), a Situational Awareness Rating Technique (SART) questionnaire, another Jian survey, and another satisfaction survey. A Situation Awareness Global Assessment Technique (SAGAT) test is done twice per session (once per uncertainty level).

So far, 18 subjects have been recruited and performed the experiment, but data for 3 subjects had to be deleted because of a procedural mistake during those sessions. Currently, data collection is halted while we wait for approval of an IRB modification regarding the details of this experiment’s protocol and compensation. The major findings from the preliminary results (N=15) are as follows: 1. Explanations improve trust (p=0.01) 2. Explanations do not significantly improve performance; #anomalies correctly diagnosed increases (p=0.14), but time to diagnosis also increases (p=0.30) 3. Explanations improve SA (p=0.018), primarily by improving understanding (P<0.01) 4. Explanations increase CW but effect is not significant (p=0.14) 5. Explanations improve user satisfaction (p<0.01) 6. Explanations improve user confidence (p<0.01) 7. Performance is higher for low uncertainty (p<0.01) 8. Trust is higher for low uncertainty (p<0.01)

A more detailed analysis, including the accuracy levels, will be done with the full dataset (N=36).

Concerning Specific Aim 3, we completed the C6M3 mission and will complete the last mission by the end of this year’s period of performance, which should get us close to completing the nominal scope of Specific Aim 3, pending publication of the results. The major findings from the HERA C6M1-M3 missions (N=12) are as follows: 1. All subjects correctly resolved all scenarios. 2. Time to resolution is lower without Daphne but not significant (p=0.2). We attribute this to some logon issues we encountered, especially in mission 1. Our perception is that the diagnosis was very fast, both with and without Daphne, suggesting that the anomaly scenarios were “too easy.” The subjects did spend considerable time conducting the procedures to resolve the anomalies, but that was not the focus of our study. 3. CW is higher with Daphne, but this difference was not significant (p=0.3). 4. SA is lower with Daphne, but this difference was not significant (p=0.19).

Some of the insights we got from the exit interviews are as follows: The crew generally exhibited a strong interest in using VAs for anomaly resolution and enjoyed using Daphne. Most of them showed an interest in the social aspects of VA and mentioned that they “attributed a personality to Daphne” and “talked about her as if she were another crewmember.” They all mentioned establishing trust very quickly once and for all thanks to Daphne “getting it right the first 3 times or so.” Almost none of them found the question-answering capabilities essential because they were “going for speed” and didn’t feel like they needed to ask any questions. However, many subjects mentioned that question-answering would be very useful in cases where Daphne recommended more than one diagnosis with the same confidence level. Moreover, crewmembers expressed an interest in the more interactive diagnosis and advanced explanations capabilities we are currently developing as something that would “significantly increase the usefulness” of the tool. Finally, they all confirmed that the scenarios generally felt very easy to diagnose and adding some more complexity would make it more interesting and fun. Note that detailed data regarding the interactions between the crewmembers and Daphne has not been analyzed yet.

The final data analysis will be done once the C6M4 mission is complete. Of note, we have extended the scope of specific Aim 3 through an extension of the grant that will allow us to participate in the HERA C7 campaign, which we have started to prepare. In C7, we will be able to test a more advanced version of the Daphne agent while also testing more complex anomaly scenarios.

In terms of publications, we submitted a journal paper on Specific Aim 1 to the Human Factors journal, which is currently under a second round of review and presented a paper on Specific Aim 2 at the 2022 Applied Human Factors and Ergonomics (AHFE) conference [15]. We will also have one talk and one poster presentation at the 2023 Human Research Project Investigator's Workshop (HRP IWS) in February.

As for the next steps, in Year 4, we will finalize the C6 campaign and start C7 (Specific Aim 3). We plan on publishing a journal paper with the results of C6. Otherwise, most of the activity will focus on finalizing lab experiments 2 and 3. We will work on additional journal papers on lab experiment, lab experiment 3, and the C7 campaign.

The research objective of this proposal is to investigate the impact of using Virtual Assistants (VA) to support crewmembers in the context of anomaly treatment during Long Duration Exploration Missions (LDEM), when ground support will be limited. A VA will be developed building upon the software architecture from existing VAs developed by the Principal Investigator (PI) for similar purposes. The VA will provide support for various aspects of anomaly treatment, including detecting and diagnosing the anomaly, as well as recommending a course of action. It will also have the ability to take initiative in the dialog with the user (mixed-initiative mode), and the ability to provide explanations for its actions. The impact of the VA on performance, cognitive workload, situational awareness, and trust, will be assessed through a set of three experiments with human subjects in a laboratory environment. The first experiment will establish the baseline impact (master-slave, no explanations), and subsequent experiments will study the effect of switching to the mixed-initiative mode and adding explanations. The system will also be deployed and tested in the Human Exploration Research Analog (HERA) analog environment.

We measured performance metrics by evaluating the number of anomalies that participants could successfully resolve with and without the use of Daphne-AT. Additional performance metrics included the time it took for each subject to solve each anomaly, whether or not the subject detected the anomaly, diagnosed the root cause, and selected the correct anomaly resolution procedure. We also recorded the number of attempts for the latter three metrics. In addition, we measured cognitive workload by employing the NASA Task Load Index (TLX) survey, and situational awareness using the Situational Awareness Rating Technique (SART). Finally, the participants’ trust in autonomous systems was measured using Jian's trust scale.

The summary of the results is as follows. For performance, a significant difference (p=0.002) was found in the number of anomalies correctly resolved with Daphne (4.1±0.3) vs. without Daphne (2.7±0.4). This supports our hypothesis that a VA can improve performance in anomaly resolution. For cognitive workload, a significant difference (p=0.002) was found in the TLX score with Daphne (45.53±6.67) vs. without Daphne (65.97±4.81). This supports our hypothesis that a VA can reduce cognitive workload in anomaly resolution. For situational awareness, a marginally significant difference (p=0.052) was found in the SART score with Daphne (5.19±0.56) vs. without Daphne (4.41±0.71). This does not fully support our hypothesis that a VA can improve situational awareness in anomaly resolution. Finally, the responses to the Jian scale items showed generally high trust in automation. A much more detailed description of the experiment and the results is in the paper about to be submitted to the Human Factors and Ergonomics Society (HFES) journal. The draft paper will be provided to the NASA Human Research Program (HRP) as soon as it is submitted.

This year we also completed the first HERA mission Campaign 6, Mission XXII (C6M1). Campaign 6 / C6M2 will also be complete by the time the period of performance of this report is exhausted, which gets us about halfway in meeting Specific Aim 3. Our HERA study was divided into two phases. In Phase 1, each crewmember performed two experiment sessions per week. In each session, the crewmember worked on a single anomaly scenario, either with Daphne or without. Thus, at the end of Phase 1, each subject had worked individually on 3 anomalies with Daphne and 3 without. Because of a technical issue with Daphne, one of the crewmembers ended up with only 2 anomalies with Daphne and 4 without Daphne. In Phase 2, crew members worked together as a team. They conducted two sessions per week, and worked on one anomaly scenario per session. This resulted in a total of 3 anomalies with Daphne and 3 without. Preliminary results are as follows.

Concerning performance, all scenarios were successfully resolved and therefore this was not an interesting metric. We are currently looking at the time to diagnosis of the anomaly to see if there are any interesting trends there. Trends appear to indicate that in some cases the crew took more time to solve scenarios with Daphne than without, especially in Phase 2, but we attribute this to some logon issues we encountered towards the end of the mission. Our perception is that the diagnosis was very fast both with and without Daphne, suggesting that the anomaly scenarios were “too easy”. The subjects did spend considerable time conducting the procedures to resolve the anomalies, but that was not the focus of our study.

Concerning cognitive workload, we observed a small difference between the TLX scores with Daphne (23.42±5.17) and without Daphne (25.20±3.54) but this difference was not significant with only 4 subjects (p=0.42). It remains to be seen if the effect becomes significant with the full 16 subjects. It is also interesting to note that these scores are very low compared to the scores obtained in the laboratory experiments. This could be due to the shorter training and less experience of the participants of the lab experiments compared to the HERA crew. This is also consistent with the explanation that the scenarios were too “easy” for the HERA crew.

Concerning situational awareness, we observed the opposite trend compared to the lab experiments, although in both cases the effects are not significant. In this case, SART scores were slightly lower with Daphne (20.46±3.35) than without Daphne (21.21±3.50), for a p-value of 0.41. We note that the effects were not consistent among crewmembers, and the overall result is driven by one of the 4 crewmembers, who reported a significantly higher level of situational awareness without Daphne vs. with Daphne. We are looking in more detail into the components of the SART score (understanding, demand, supply) because the trends appear to be conflicting there too. While the literature has reported cases where using some kind of assistance results in a decrease of situational awareness (e.g., autopilots in commercial aircraft), this may be due to other factors (e.g., minor technical issues with the VA) as well. More research is needed to understand this – we hope that missions 2,3,4 will help us elucidate this issue. We also note that crewmembers mentioned in the exit debrief interviews that adding explanations would likely significantly increase situational awareness with Daphne, and this is something that we plan on testing in our lab experiments 2 and 3.

Concerning trust, the results were similar to the ones from the lab experiments, showing high trust in the VA across the board. In the exit interviews, crewmembers indicated that they trusted Daphne “right away”, after the first few of her recommendations were proven to be correct.

Some of the insights we got from the exit interviews are as follows. The crew generally exhibited strong interest in using VAs for anomaly resolution, and enjoyed using Daphne. Most of them showed an interest in the social aspects of VA and mentioned that they “attributed a personality to Daphne” and “talked about her as if she were another crewmember”. They all mentioned establishing trust very quickly, once and for all, thanks to Daphne “getting it right the first 3 times or so”. Almost none of them found the question answering capabilities essential because they were “going for speed” and didn’t feel like they needed to ask any questions. However, all of them mentioned that question answering would be very useful in cases where Daphne recommended more than one diagnosis with the same confidence level. Moreover, crewmembers expressed an interest in the more interactive diagnosis and advanced explanations capabilities we are currently developing as something that would “significantly increase the usefulness” of the tool. Finally, they all confirmed that the scenarios generally felt very easy to diagnose and adding some more complexity would make it more interesting and fun.

As for next steps, in Year 4 we will support the remaining missions of C6, thus finalizing Specific Aim 3. We plan on publishing a journal paper with the results. Otherwise, most of the activity will focus on achieving Specific Aim 2: we will finalize the development of the enhanced VA capabilities (explanations and mixed initiative) and conduct lab experiments 2 and 3.

The research objective of this proposal is to investigate the impact of using Virtual Assistants (VA) to support crew members in the context of anomaly treatment during Long Duration Exploration Missions (LDEM), when ground support will be limited. A VA will be developed building upon the software architecture from existing VAs developed by the Principal Investigator (PI) for similar purposes. The VA will provide support for various aspects of anomaly treatment, including detecting and diagnosing the anomaly, as well as recommending a course of action. It will also have the ability to take initiative in the dialog with the user (mixed-initiative mode), and the ability to provide explanations for its actions. The impact of the VA on performance, cognitive workload, situational awareness, and trust, will be assessed through a set of three experiments with human subjects in a laboratory environment. The first experiment will establish the baseline impact (master-slave, no explanations), and subsequent experiments will study the effect of switching to the mixed-initiative mode and adding explanations. The system will also be deployed and tested in the Human Exploration Research Analog (HERA) analog environment.

To be able to do experiments at Texas A&M, we developed an immersive 3d user interface to simulate the Human Exploration Research Analog (HERA) habitat during our experiments at TAMU. The goal was for the simulator to have reasonable fidelity in terms of the general dimensions and shape as well as the main components present in the habitat related to the ECLSS system. The simulator is basically a first-person game developed in Unreal Engine 4 (UE4). All of the major components of the HERA Environmental Control and Life Support Subsystem (ECLSS) system are present and the user can interact with them (e.g., press buttons, open/close panels) as required by the anomaly resolution procedures. More details about this simulator are provided in

Woodruff R, Beebe N, Josan PK, Wong RKW, Dunbar BJ, Selva D, Diaz-Artiles A. (2021). 3D Interactive Model of HERA to support ECLSS anomaly resolution using a Virtual Assistant. 2021 IEEE Aerospace Conference Proceedings.

After validation of the UE4 simulator, we designed and conducted a pilot study of the first “Baseline” prototype of the assistant. It included five male subjects within the age range of 22-40 years old, and with a Standard Deviation (SD) of ±7.2. Prior to the experimental sessions, subjects were provided a 2-hour virtual training session about all the experiment elements (Daphne-AT, ECLSS subsystems, 3D virtual HERA environment, and experimental protocols) and they were guided through an example anomaly scenario. After the training, subjects participated in two in-person experiment sessions: one with Daphne-AT and the other one without Daphne-AT (counterbalanced among subjects). These sessions were conducted in two consecutive days at approximately the same time of the day. In each one of the sessions, the subjects solved four anomalies. Once the experiment sessions were over, the subjects completed a set of surveys.

Dependent variables included performance, cognitive workload, situational awareness, and trust in the context of ECLSS anomaly treatment, with and without the use of a VA. Performance is described as an individual’s ability to successfully treat the anomaly. On average, subjects solved 3.2 ± 0.37 anomalies using Daphne-AT and 2.0 ± 0.without Daphne-AT. Based on these limited data, our preliminary results support the hypothesis that the use of Daphne-AT will increase performance in the context of ECLSS anomaly resolution operations.

The NASA Task Load Index (TLX) survey was used to measure the effect of using Daphne-AT on cognitive workload. NASA TLX is an evaluation technique developed by NASA to assess relative importance of six pre-determined factors in determining how much workload a subject experience during a particular task, in this case, the task of anomaly treatment. These six factors include mental demand, physical demand, temporal demand, perceived performance, perceived effort, and frustration level. On average, subjects indicated higher cognitive workload during the sessions without Daphne-AT (57.2 ± 5.78 with Daphne-AT and 66 ± 7.31 without Daphne). These preliminary results support the hypothesis that the use of Daphne-AT will decrease cognitive workload. More details about this pilot study can be found in:

Josan PK, Dutta P, Woodruff R, Beebe N, York K, Balcells-Quintana O, Kluis L, Viros-i-Martin A, Dunbar BJ, Wong RKW, Selva D, Diaz-Artiles A. (2021). Experimental Design & Pilot Testing for ECLSS Anomaly Resolution using Daphne-AT Virtual Assistant. 2021 IEEE Aerospace Conference Proceedings.

As a result of the pilot testing and HERA integration activities, we have substantially refined the VA interface and added functionality to the back end. Specifically: 1) We adapted the telemetry feed to the new version of the HERA Environmental Control and Life Support System (ECLSS) simulator (now called Habitat System Simulator or HSS). 2) We updated the anomaly database to include more information for each anomaly (e.g., a time signature). 3) We expanded the knowledge base with more information about ECLSS anomalies, their associated risks, and how to resolve them. 4) We expanded the anomaly detection functions beyond basic thresholding to be able to detect outliers in the telemetry feed. 5) We made slight changes to the knowledge graph and procedures to adapt to HERA C6 schedule constraints.

In terms of publications, we had two papers accepted at the 2021 IEEE Aerospace Conference which will be presented in March. We also gave a presentation at the 2020 American Institute of Aeronautics and Astronautics (AIAA) ASCEND conference (Ed. note 1/7/2021: see FY2020 Bibliography in Task Book report) and will have one talk presentation and one poster presentation at the 2021 Human Research Program workshop in February.

The research objective of this proposal is to investigate the impact of using Virtual Assistants (VA) to support crew members in the context of anomaly treatment during Long Duration Exploration Missions (LDEM), when ground support will be limited. A VA will be developed building upon the software architecture from existing VAs developed by the Principal Investigator (PI) for similar purposes. The VA will provide support for various aspects of anomaly treatment, including detecting and diagnosing the anomaly, as well as recommending a course of action. It will also have the ability to take initiative in the dialog with the user (mixed-initiative mode), and the ability to provide explanations for its actions. The impact of the VA on performance, cognitive workload, situational awareness, and trust, will be assessed through a set of three experiments with human subjects in a laboratory environment. The first experiment will establish the baseline impact (master-slave, no explanations), and subsequent experiments will study the effect of switching to the mixed-initiative mode and adding explanations. The system will also be deployed and tested in the Human Exploration Research Analog (HERA) analog environment.

In the first year of this project, we have completed a project definition phase in which we have more crisply defined the scientific objectives and the technical approach to design the virtual assistant. We started by selecting the Environmental Control and Life Support Subsystem (ECLSS) as the technical scope for the virtual assistant. We then proceeded to elicit high level requirements and use cases and perform the early design of the virtual assistant, which we are getting ready to test in our laboratory at Texas A&M University in the next month.

Specifically, in this first year, we have accomplished the following: 1) We have defined the high-level requirements and overall software architecture of the virtual assistant. 2) We have defined how the virtual assistant will interface with the Human Exploration Research Analog (HERA) infrastructure. 3) We have developed the first very basic prototype of the virtual assistant which includes: a) a simple anomaly database; b) a simple database containing knowledge about how the ECLSS system works in HERA and how to perform some maintenance and repair operations; c) an interface with the HERA simulated telemetry feed, which provides a data stream with all the measurements of the ECLSS subsystem; d) a web-based graphical user interface containing a main plot window displaying the telemetry feed and a chat box to chat with the virtual assistant; e) the natural language processing layer that the assistant uses to process the natural language requests; f) the machine learning layer that is used to understand the intent of the question and send it to the relevant part of the software, i.e., the one that knows how to answer it; f) the first version of the back end of the software, which is precisely the part of the software that performs the calculations needed to answer the questions and requests from the user. While this first prototype is not sophisticated in terms of the breadth and depth of questions it can answer, it still has most of the main functionality expected of the assistant. Specifically: 1) it warns the user when a measurement from the telemetry feed exceed some user-defined thresholds; 2) it also performs statistical analysis on the data stream and warns the user if the measurement is considered an outlier, even if it is within the user-defined thresholds; 3) once an anomaly is detected (e.g., increase in CO2 concentration), it provides the users with some possible root causes for it, in order of likelihood (e.g., a malfunctioning filter or a leak); 4) once the anomaly has been diagnosed (faulty filter), it provides the user with relevant procedures to reconfigure the system (how to change the CO2 filter).

In parallel of the software development, we have also started planning and designing the experiments. We are currently getting ready to start a pilot experiment at Texas A&M in order to refine both the software implementation and the experimental design. The results of this pilot experiment will help us get ready for the HERA campaign, which is currently scheduled for August 2020.

Since the beginning of the project, we have been working with the HERA team to prepare for the experiment. We obtained approval from the Institutional Review Boards of both NASA and Texas A&M. We provided our input to the experiment support specialist to create our science requirements document and are currently working on the investigator working group, getting ready for a software delivery to NASA by March.

We have also been in constant communication with the rest of the VNSCOR to ensure compatibility of our experiments and try to make the most of any potential synergies.

Finally, in terms of publications, we are about to present a peer-reviewed full paper at the 2020 American Institute of Aeronautics and Astronautics (AIAA) Scitech conference. We also have two poster presentations at the 2020 Human Research Program workshop later this month.

In the next few months, we will keep adding functionality to the assistant so that it can answer more questions better and provide more useful information to the user. We anticipate that some rework of the telemetry feed interface will be necessary to adapt to some changes on the HERA side. We will also refine the user interface with the results from the pilot. We are getting ready to conduct a preliminary design review in March before delivery to HERA. Between March and August 2020 we will work on integrating our software with HERA. We will conduct a Critical Design Review in June 2020 and an Operational Readiness Review in August 2020, right before ingress. Between August 2020 and September 2021 we will support the 4 missions of the C6 HERA campaign.

The research objective of this proposal is to investigate the impact of using Virtual Assistants (VA) to support crew members in the context of anomaly treatment during Long Duration Exploration Missions (LDEM), when ground support will be limited. A VA will be developed building upon the software architecture from an existing VA developed by the Principal Investigator. The VA will provide support for various aspects of anomaly treatment, including detecting and diagnosing the anomaly, as well as recommending a course of action. It will also have the ability to take initiative in the dialog with the user (mixed-initiative mode), and the ability to provide explanations for its actions. The impact of the VA on performance, cognitive workload (CW), situational awareness (SA), and trust, will be assessed through a set of three experiments with human subjects in a laboratory environment. The first experiment will establish the baseline impact (master-slave, no explanations), and subsequent experiments will study the effect of switching to the mixed-initiative mode and adding explanations. Finally, the system will be deployed in an analog environment.