NOTE: Element change to Human Factors & Behavioral Performance; previously Space Human Factors & Habitability (Ed., 1/19/17)

This project addressed the Risk of Inadequate Critical Task Design of the Human Research Program (HRP), specifically the SHFE-TASK-02 gap: What model-based HF (human factors) tools can assist with the design and evaluation of spacecraft systems and task procedures. A toolkit is needed to support dynamic task design, particularly for design by non-programmers who would be making the procedures. There exists no taxonomy of cognitive aid design and task type nor standards for interactive cognitive aids to drive development. A cognitive aid should provide guidance to support efficiency and success while minimizing cognitive workload, but it was not yet clear what attributes, such as adaptability, the aid should contain.

The project also addressed the HRP’s Risk of Performance Errors Due to Training Deficiencies, specifically the TRAIN-03 gap: We need to develop guidelines for effective onboard training systems that provide training traditionally assumed for pre-flight. (Previously: SHFE-TRAIN-03 - How can onboard training systems be designed to address Just in Time (JIT) and recurrent training needs for nominal and off nominal scenarios?). To contribute to the closure of this gap, we investigated cognitive aids for non-expert operators.

Our goals in this project were two-fold: The first goal was to investigate a new form of cognitive aid that incorporates the beneficial attributes of static, adaptable, adaptive, and dynamic aids into a hybrid aid. We see the benefit of this hybrid aid to be:

1. a technology able to be immediately deployed (unlike augmented reality or other heavily technology-dependent advances),

2. a technology able to be used by non-programmers as they design aids for crewmembers (the audiences would be procedure designers and scientists creating procedures for in-flight experiments),

3. a form of aid that offers the beneficial attributes of a dynamic aid that responds to each step in a procedure with the security of a static or adaptable aid that does not require functioning sensors for performance.

We designed a prototype of a hybrid aid for medical equipment maintenance tasks, while at the same time building a library of tools to create similar aids for other procedures. Second, developed a taxonomy of cognitive aid design that considers the most important variables affecting performance with aids: user knowledge and experience, time pressure on performance, task criticality, and the number of operators expected to interact with the aid.

Our Hybrid Aid toolkit is a prototype of an innovative cognitive aid technology that is designed to support non-expert operators in complex tasks, while also offering tools to the procedure designers and the operators themselves as they perform the task.

To contribute to the closure of these gaps, the research focused on investigating cognitive aids for non-expert operators. Our goals were two-fold:

The first goal was to investigate a new form of cognitive aid that incorporates the beneficial attributes of static, adaptable, adaptive, and dynamic aids into a hybrid aid. We see the benefit of this hybrid aid to be: - a technology able to be immediately deployed (unlike augmented reality or other heavily technology-dependent advances), - a technology able to be used by non-programmers as they design aids for crewmembers (the audiences would be procedure designers and scientists creating procedures for in-flight experiments), - a form of aid that offers the beneficial attributes of a dynamic aid that responds to each step in a procedure with the security of a static or adaptable aid that does not require functioning sensors for performance. The second goal was to develop a taxonomy of cognitive aid design that considers the most important variables affecting performance with aids: user knowledge and experience, resources demanded by the task and sub-tasks, time pressure, and the number of operators expected to interact with the aid. Hypotheses were that a hybrid aid would have the following benefits:

- a technology able to be immediately deployed (unlike augmented reality or other heavily technology-dependent advances),

- a technology able to be used by non-programmers as they design aids for crewmembers (the audiences would be procedure designers and scientists creating procedures for in-flight experiments),

- a form of aid that offers the beneficial attributes of a dynamic aid that responds to each step in a procedure with the security of a static or adaptable aid that does not require functioning sensors for performance.

Specific Questions Addressed

The following specific research questions were addressed across four experiments, a validation study, and an integrative literature review. All experiments and studies focused on capable but non-expert operators using a cognitive aid.

First, how does the sensory modality of the aid interact with the demands of the task being aided? Is it better to avoid cognitive resource conflict or to consider the task and aid as an integrated system that might benefit from matching the resources needed for each? Second, how does the method of interaction (manual, voice) affect use of the aid in various sensory modalities? Third, how do teams of two operators differ from single operators in how they utilize the cognitive aid and perform the task? Does this interaction change when under time pressure? Fourth, how do individual differences in ability needed for the task (e.g., spatial ability) affect use of a cognitive aid? Is there a benefit to providing aid from perspectives that do not require mental rotation? Fifth, using the information gained from the literature review and experiments, can we create an aid that supports significantly better performance from space-experienced operators at Johnson Space Center compared to a traditional procedure?

Connection to the NASA Human Research Roadmap : The overall project addresses Risk of Inadequate Critical Task Design of the Human Research Program (HRP), specifically the SHFE- TASK-02 gap: What model-based HR tools can assist with the design and evaluation of spacecraft systems and task procedures? A toolkit is needed to support dynamic task design, particularly for design by non-programmers who would be making the procedures. There exists no taxonomy of cognitive aid design and task type nor standards for interactive cognitive aids to drive development. A cognitive aid should provide guidance to support efficiency and success while minimizing cognitive workload, but it is not yet clear what attributes the aid should contain. The experiments included in this project are examining the variables of aid modality, number of team members, and support for off-nominal procedures. The first experiment, reported here, will draw conclusions about the importance of reducing sensory modality conflict between the cognitive aid and the task.

The project also addresses the HRP’s Risk of Performance Errors Due to Training Deficiencies, specifically the TRAIN-03 gap: We need to develop guidelines for effective onboard training systems that provide training traditionally assumed for pre-flight.

Last, Risks not included in the original proposal (but discovered to be applicable to the current project ) were: - Risk of Adverse Health Outcomes & Decrements in Performance due to Inflight Medical Conditions: Med-05: We do not know how to train crew for medical decision making and medical skills to enable extended mission or autonomous operations. Cognitive aids can provide just-in-time training and job support; - Risk of Inadequate Human-Computer Interaction and HCI-06: We need guidelines to ensure operators receive all of the information required to accomplish necessary tasks in a timely fashion, even when operating autonomously. (Previous title: SHFE-HCI-06). Discovering the best way to design cognitive aids for previously unstudied scenarios (e.g., non-expert operators, numbers of operators) results in guidelines for the cognitive aids of the future.

As projected, the overall project addressed the Risk of Inadequate Critical Task Design of the Human Research Program (HRP), specifically the SHFE- TASK-02 gap: What model-based HR tools can assist with the design and evaluation of spacecraft systems and task procedures? A toolkit was created to support dynamic procedure design. Further, the literature review and taxonomy provides a structure and boundary conditions for the development of new cognitive aids, even those using unexplored technologies (e.g., Augmented or Virtual reality). One of the main take home messages from the literature review was the lack of research on supporting non-expert operators and a lack of focus on broad generalized knowledge of cognition that can be applied to all cognitive aids.

The project also addressed the HRP’s Risk of Performance Errors Due to Training Deficiencies, specifically the TRAIN-03 gap: We need to develop guidelines for effective onboard training systems that provide training traditionally assumed for pre-flight. All of the studies performed used non-expert operators and provided information on how to create aids that assist with just-in-time training. Another risk addressed was the Risk of Adverse Health Outcomes & Decrements in Performance due to Inflight Medical Conditions: Med-05: We do not know how to train crew for medical decision making and medical skills to enable extended mission or autonomous operations. Again, our focus on supporting non-expert operators applies to this risk, under the assumption that crewmembers may be called upon to perform medical tasks for which they are not fully trained.

Last, we addressed the Risk of Inadequate Human-Computer Interaction and HCI-06: We need guidelines to ensure operators receive all of the information required to accomplish necessary tasks in a timely fashion, even when operating autonomously. Cognitive aids, as was shown in the literature review and taxonomy, require study both of how the aid supports cognition in the task and how cognition must be supported in using the aid itself. Discovering the best way to design cognitive aids for previously unstudied scenarios (e.g., non-expert operators, numbers of operators) resulted in guidelines for the cognitive aids of the future.

[Ed. note October 2020--HRP gaps have been changed since this report in May 2019]

NOTE: Element change to Human Factors & Behavioral Performance; previously Space Human Factors & Habitability (Ed., 1/19/17)

This proposal addresses the Risk of Inadequate Critical Task Design of the Human Research Program (HRP), specifically the SHFE-TASK-02 gap: What model-based HF (human factors) Tools can assist with the design and evaluation of spacecraft systems and task procedures. A toolkit is needed to support dynamic task design, particularly for design by non-programmers who would be making the procedures. There exists no taxonomy of cognitive aid design and task type nor standards for interactive cognitive aids to drive development. A cognitive aid should provide guidance to support efficiency and success while minimizing cognitive workload, but it is not yet clear what attributes, such as adaptability, the aid should contain.

The proposal also addresses the HRP’s Risk of Performance Errors Due to Training Deficiencies, specifically the TRAIN-03 gap: We need to develop guidelines for effective onboard training systems that provide training traditionally assumed for pre-flight. (Previously: SHFE-TRAIN-03 - How can onboard training systems be designed to address Just in Time (JIT) and recurrent training needs for nominal and off nominal scenarios?). To contribute to the closure of this gap, the research proposed will investigate cognitive aids for non-expert operators.

Our goals in this proposal are two-fold: The first goal is to investigate a new form of cognitive aid that incorporates the beneficial attributes of static, adaptable, adaptive, and dynamic aids into a hybrid aid. We see the benefit of this hybrid aid to be:

1. a technology able to be immediately deployed (unlike augmented reality or other heavily technology-dependent advances),

2. a technology able to be used by non-programmers as they design aids for crewmembers (the audiences would be procedure designers and scientists creating procedures for in-flight experiments),

3. a form of aid that offers the beneficial attributes of a dynamic aid that responds to each step in a procedure with the security of a static or adaptable aid that does not require functioning sensors for performance.

We will iteratively design a prototype of a hybrid aid for medical equipment maintenance tasks, while at the same time building a library of tools to create similar aids for other procedures. Second, we will develop a taxonomy of cognitive aid design that considers the most important variables affecting performance with aids: user knowledge and experience, resources demanded by the task and sub-tasks, time pressure on performance, and the number of operators expected to interact with the aid.

The project received a no-cost extension until February, 2019, to complete software iterations on the cognitive aid, dissemination of the experiment data, and completion of the taxonomy of variables involved in cognitive aid design.

The two Risks and Gaps addressed by this research were:

Risk of Inadequate Critical Task Design of the Human Research Program (HRP), specifically the SHFE-TASK-02 gap: What model-based HF Tools can assist with the design and evaluation of spacecraft systems and task procedures?

Risk of Performance Errors Due to Training Deficiencies, specifically the TRAIN-03 gap: We need to develop guidelines for effective onboard training systems that provide training traditionally assumed for pre-flight.

The results inform the Risk of Inadequate Critical Task Design by: 1) discovering the helpful tools and forms of flexibility that aid in procedure use, in our case via a hybrid cognitive aid, 2) informing on the effects of aid-modality as it conflicts or does not conflict with the modality of the task described by the procedure, 3) the functioning of cognitive aids when operators are under time pressure, and 4) the real-world use of cognitive aids when operators work in teams. The findings also gauged the impact of interruptions during procedures and recovery from errors. Regarding the Risk of Performance Errors due to Training Deficiencies, the findings demonstrated how novices perform a new task with just-in-time training offered by the procedure and how this training might be improved by the added experiences of other operators who used the procedure in the same environment.

In the last year we fully completed, analyzed, and wrote up Experiments 1 and 2, which served as the dissertation of Dr. John (Jojo) Sprufera. Research questions in these experiments included task-aid modality conflict and the tools provided in a hybrid cognitive aid. We collected pilot data and full data for Experiment 3 concerning team interactions with the cognitive aid and the effects of time pressure on aid use, with and without a second operator. We collected data for Experiment 4, regarding the best perspective (first person vs. third person) for media shown in the cognitive aid. Last, we developed a working aid prototype with the help of a research assistant in computer science based on our findings from these experiments. This working prototype was tested in a validation study at Johnson Space Center where 28 experienced participants interacted with the aid compared to a procedure not augmented with the cognition aids.

Summary of progress on Experiment 3: Participants and Materials and methods were similar to Experiment 1. Changes included: the ventilator setup task was divided into sections that were either performed under time pressure (using video and audio to mimic an emergency) and participants were randomly assigned to perform the setup task alone or in a team of two. Twenty participants operated alone and and forty in teams of two.

In general, the setup task was performed faster and more accurately by teams and faster (but less accurately) when under time pressure. Use of the cognitive aid was observed to be distributed among the team members, where one tended to take ownership of reading the aid while the other followed instruction. Negotiation of understanding of each step occurred frequently between team members.This finding points to the importance of an operator (either human or potentially part of the aid) providing directive instruction while the task is being performed by another. This keeps the operator engaged in the task.

Summary of progress on Experiment 4: When participants used the aid in Experiments 1, 2, and 3, we observed the benefits and difficulties our media augmentations provided. This prompted a new question that needed to be answered in design of the aid: from which perspective should video and picture augmentations be provided? A media visual taken from the first person perspective requires no mental rotation when the task is performed, while a third-person perspective requires mental rotation. We hypothesized that, under ideal conditions with highly able operators, the perspective of the media would have little effect on performance. However, when the task was highly spatial, difficult, and or complex, we expected perspective to have an effect, such that the addition of mental rotation would decrement performance. Similarly, when the operators were under high cognitive demand, as can occur with multi-tasking or interruptions, we expected perspective to have an effect.

To test these hypotheses, we again used the setup of a medical ventilator as our task and provided video instruction via a cognitive aid. This instruction was manipulated to show the first or third person perspective. Our participants were younger adults (18-28) with relatively high fluid abilities (as measured via tests of spatial ability) and older adults (65-75) with relatively lower fluid abilities. Such a sample allowed us to test the general hypothesis that the importance of perspective was linked to available mental resources. Participants were randomly assigned to a perspective condition. The full design contained two age groups and two perspective conditions. Errors (omission and commission) and completion times were collected.

Data were gathered in the spring and summer of 2018. Analyses are ongoing.

Software Products: The hybrid aid prototype was iteratively designed and tested during the validation study at Johnson Space Center.

The designs are the result of several iterations of usability testing of the hybrid-aid concept via the validation study. They are currently being implemented in software.

NOTE: Element change to Human Factors & Behavioral Performance; previously Space Human Factors & Habitability (Ed., 1/19/17)

This proposal addresses the Risk of Inadequate Critical Task Design of the Human Research Program (HRP), specifically the SHFE-TASK-02 gap: What model-based HF (human factors) Tools can assist with the design and evaluation of spacecraft systems and task procedures. A toolkit is needed to support dynamic task design, particularly for design by non-programmers who would be making the procedures. There exists no taxonomy of cognitive aid design and task type nor standards for interactive cognitive aids to drive development. A cognitive aid should provide guidance to support efficiency and success while minimizing cognitive workload, but it is not yet clear what attributes, such as adaptability, the aid should contain.

The proposal also addresses the HRP’s Risk of Performance Errors Due to Training Deficiencies, specifically the TRAIN-03 gap: We need to develop guidelines for effective onboard training systems that provide training traditionally assumed for pre-flight. (Previously: SHFE-TRAIN-03 - How can onboard training systems be designed to address Just in Time (JIT) and recurrent training needs for nominal and off nominal scenarios?). To contribute to the closure of this gap, the research proposed will investigate cognitive aids for non-expert operators.

Our goals in this proposal are two-fold: The first goal is to investigate a new form of cognitive aid that incorporates the beneficial attributes of static, adaptable, adaptive, and dynamic aids into a hybrid aid. We see the benefit of this hybrid aid to be:

1. a technology able to be immediately deployed (unlike augmented reality or other heavily technology-dependent advances),

2. a technology able to be used by non-programmers as they design aids for crewmembers (the audiences would be procedure designers and scientists creating procedures for in-flight experiments),

3. a form of aid that offers the beneficial attributes of a dynamic aid that responds to each step in a procedure with the security of a static or adaptable aid that does not require functioning sensors for performance.

We will iteratively design a prototype of a hybrid aid for medical equipment maintenance tasks, while at the same time building a library of tools to create similar aids for other procedures. Second, we will develop a taxonomy of cognitive aid design that considers the most important variables affecting performance with aids: user knowledge and experience, resources demanded by the task and sub-tasks, time pressure on performance, and the number of operators expected to interact with the aid.

In the last year, we have assembled a team at NC State and worked with our colleague at Johnson Space Center to stay on track for the proposed timeline. The team at NC State included one PhD student supported by the project as a research assistant, another PhD student supported by a Provost fellowship but dedicated to the project for research experience, and several undergraduate research assistants working in the lab for experience and course credit. After a thorough task analysis of the medical ventilator and interviews with subject matter experts (SMEs) at the College of Veterinary Medicine, we created a SME-validated simulator that collects data. We have also developed a prototype of the cognitive aid that allows operation through various sensory modalities as prescribed by the experimenter. Further, we have identified a small follow-up to the first experiment that we plan to run as well, with higher-order Wizard-of-Oz audio (voice inputs) interfacing with the cognitive aid than was possible in Experiment 1 (where conditions needed to be kept identical between the auditory and visual aid conditions). The undergraduate and graduate research assistants on this project presented the project development at the North Carolina Cognition Conference (NCCC) and the Southeastern Human Factors Applied Research Conference (SHARC).

We have recruited a sample of veterinary students from the NC State College of Veterinary medicine and several psychology PhD students aged 22-35 into the LACElab at NC State University. This group was sampled due to their similarities to crewmember non-experts: highly educated, highly motivated to perform procedures accurately, often expected to perform in emergencies, yet not experienced in the procedure. Our power analysis indicated a need for 65 participants: we have currently run 58 (10 pilot participants).

Challenges we have faced include turnover of our Co-Investigators (Co-Is) at Johnson Space Center (KBRWyle). Dr. Aniko Sandor was the original co-I who helped to write the proposal, but left her position at KBRWyle in late 2016. Since then we have had great relations with substitute co-Is and currently are working with Vicky Byrne, a specialist in medical human factors, but this has been a change. We are planning an in-person team meeting at NC State for August/September 2107. Another challenge has been hiring a graduate student in computer science to program the Hybrid-Aid Toolkit (HAT). These students generally start their research assistantships in fall, but our funding was not received in time to advertise a position in 2016. We are currently recruiting for a computer scientist to join our team in summer 2017 and continue through summer 2018.

This proposal addresses the Risk of Inadequate Critical Task Design of the Human Research Program (HRP), specifically the SHFE-TASK-02 gap: What model-based HF Tools can assist with the design and evaluation of spacecraft systems and task procedures. A toolkit is needed to support dynamic task design, particularly for design by non-programmers who would be making the procedures. There exists no taxonomy of cognitive aid design and task type nor standards for interactive cognitive aids to drive development. A cognitive aid should provide guidance to support efficiency and success while minimizing cognitive workload, but it is not yet clear what attributes, such as adaptability, the aid should contain.

The proposal also addresses the HRP’s Risk of Performance Errors Due to Training Deficiencies, specifically the TRAIN-03 gap: We need to develop guidelines for effective onboard training systems that provide training traditionally assumed for pre-flight. (Previously: SHFE-TRAIN-03 - How can onboard training systems be designed to address Just in Time (JIT) and recurrent training needs for nominal and off nominal scenarios?). To contribute to the closure of this gap, the research proposed will investigate cognitive aids for non-expert operators.

Our goals in this proposal are two-fold: The first goal is to investigate a new form of cognitive aid that incorporates the beneficial attributes of static, adaptable, adaptive, and dynamic aids into a hybrid aid. We see the benefit of this hybrid aid to be:

1. a technology able to be immediately deployed (unlike augmented reality or other heavily technology-dependent advances),

2. a technology able to be used by non-programmers as they design aids for crewmembers (the audiences would be procedure designers and scientists creating procedures for in-flight experiments),

3. a form of aid that offers the beneficial attributes of a dynamic aid that responds to each step in a procedure with the security of a static or adaptable aid that does not require functioning sensors for performance.

We will iteratively design a prototype of a hybrid aid for medical equipment maintenance tasks, while at the same time building a library of tools to create similar aids for other procedures. Second, we will develop a taxonomy of cognitive aid design that considers the most important variables affecting performance with aids: user knowledge and experience, resources demanded by the task and sub-tasks, time pressure on performance, and the number of operators expected to interact with the aid.