NOTE: Element change to Human Factors & Behavioral Performance; previously Space Human Factors & Habitability (Ed., 1/19/17)

We will conduct literature research followed by piloting and experimentation to a) define appropriate measures and manipulations and b) assess the extent that integrated S&K generalizes well and is well-retained relative to S&K covering related content but without the integrating relationships. Experimentation will aim to produce integrated knowledge though a variety of learning activities that a) build links from procedural skills to abstract principles through processes such as self-explanation, thus facilitating their decomposition and reuse and b) build links from abstract principles to specific procedures through processes such as instantiation, thus aiding their application. Evidence shows that integrated knowledge is better retained, as the many links of integrated knowledge provide multiple, alternative retrieval cues accessible in many contexts. We will assess whether manipulations that produce integrated knowledge also produce better generalization (as well as measuring retention), relative to knowledge that spans related content but is not integrated, and whether such effects can be attributed to the mediating role of integrated knowledge.

We will use the work domain of operating and trouble-shooting complex technology, specifically spacecraft life-support systems as the test case for the proposed research. We will investigate “performing system-related tasks in highly autonomous environments” (Human Exploration Research Opportunities (HERO) Appendix A-25), such as operation, maintenance, and troubleshooting components within habitat life-support systems. Three factors motivate this choice.1) Such work is highly relevant to long-distance, crewed missions. 2) Prior research has studied operation of devices and thus provides findings on representations supporting generalization in similar contexts. 3) We will use an existing software suite including simulation of life-support devices on the International Space Station (ISS) and procedures for their operation as our test bed. This “micro-world” enables experimental control yet is a close analog of real mission work. In this domain the critical skills and knowledge are primarily cognitive (e.g., deciding what procedure to use), rather than sensory-motor (e.g., how hard a wrench must be turned). We include “meta-knowledge” such as identifying what information is missing when resources such as just-in-time-training are relevant.

We will assess whether interventions related to integration produce forms of S&K that generalize well and are well retained, a prediction motivated by prior findings but not investigated directly, nor in NASA-relevant context. If integration is the basis for the predicted outcomes, it will provide a powerful principle for identifying and creating forms of S&K that are both generalizable and retainable. If not, we have discovered how learning is affected by important training interventions in a domain highly relevant to NASA future crewed exploration missions. This research will narrow the gap in understanding the factors that make skills and knowledge for NASA-critical tasks more generalizable and more retainable.

We conducted an opportunistic study on retention of skills learned in the first two studies done under this grant. We contacted and arranged the return of previous participants--5 returning from Experiment 1 after a bit less than two years and 7 from experiment 2 returning after a bit less than one year. We recruited 14 new matched participants. We provided all participants with very basic (re) training on how to use the software, but minimal explanation or practice. Individual variation was large and overall performance was broadly similar. One of the key transfer tasks showed a significantly higher completion rate by returning users (6 of 12 vs 0 of 14) and the number of action-fails triggered by users was lower for returning than new users.

The fourth study under this research grant compared a condition with integrated training of model and procedural knowledge to a condition only training procedural knowledge. We found that after training participants in the integrated model condition produced better diagrammatic and text explanations of how the Carbon Dioxide Removal Systems (CDRS) work than it did participants in the no-model condition. Thus, we did succeed in teaching the participants at least model information (whether or not integrated with procedural skill). A Monitoring Task asked participants to task discriminate and explain whether a statically presented situation is normal or problematic; participants in the Integrating Model Condition provided better explanations than did those in the No Model Condition. However, participants were similarly likely to produce appropriate, successful performance on procedure execution tasks that required recognition of a problem or potential problem.

Across these studies we have also informally observed variety in behavior that seems to be influenced by a variety of other factors, both in training and in the transfer tasks. Identification and organization of these factors may be helpful in identifying factors limiting performance in a specific technical situation. We have investigated some of these factors but further understanding is needed.

NOTE: Element change to Human Factors & Behavioral Performance; previously Space Human Factors & Habitability (Ed., 1/19/17)

We will conduct literature research followed by piloting and experimentation to a) define appropriate measures and manipulations and b) assess the extent that integrated S&K generalizes well and is well-retained relative to S&K covering related content but without the integrating relationships. Experimentation will aim to produce integrated knowledge though a variety of learning activities that a) build links from procedural skills to abstract principles through processes such as self-explanation, thus facilitating their decomposition and reuse and b) build links from abstract principles to specific procedures through processes such as instantiation, thus aiding their application. Evidence shows that integrated knowledge is better retained, as the many links of integrated knowledge provide multiple, alternative retrieval cues accessible in many contexts. We will assess whether manipulations that produce integrated knowledge also produce better generalization (as well as measuring retention), relative to knowledge that spans related content but is not integrated, and whether such effects can be attributed to the mediating role of integrated knowledge.

We will use the work domain of operating and trouble-shooting complex technology, specifically spacecraft life-support systems as the test case for the proposed research. We will investigate “performing system-related tasks in highly autonomous environments” (Human Exploration Research Opportunities (HERO) Appendix A-25), such as operation, maintenance, and troubleshooting components within habitat life-support systems. Three factors motivate this choice.1) Such work is highly relevant to long-distance, crewed missions. 2) Prior research has studied operation of devices and thus provides findings on representations supporting generalization in similar contexts. 3) We will use an existing software suite including simulation of life-support devices on the International Space Station (ISS) and procedures for their operation as our test bed. This “micro-world” enables experimental control yet is a close analog of real mission work. In this domain the critical skills and knowledge are primarily cognitive (e.g., deciding what procedure to use), rather than sensory-motor (e.g., how hard a wrench must be turned). We include “meta-knowledge” such as identifying what information is missing when resources such as just-in-time-training are relevant.

We will assess whether interventions related to integration produce forms of S&K that generalize well and are well retained, a prediction motivated by prior findings but not investigated directly, nor in NASA-relevant context. If integration is the basis for the predicted outcomes, it will provide a powerful principle for identifying and creating forms of S&K that are both generalizable and retainable. If not, we have discovered how learning is affected by important training interventions in a domain highly relevant to NASA future crewed exploration missions. This research will narrow the gap in understanding the factors that make skills and knowledge for NASA-critical tasks more generalizable and more retainable.

Technical work across science, technology, engineering, and mathematics (STEM) disciplines, on Earth as in space, requires mastery of complex suites of knowledge and skills. Whether presented as education or as job-specific training, these skills and knowledge must be learned. Much of this work has an open, changing character such that even if time were available, it is not possible to anticipate and teach all the components that will be needed in the work. Thus, training that allows a person effectively to transfer skills and knowledge to situations and tasks that were not trained is extremely valuable.

Our work investigates how to produce and to measure skills and knowledge that support significant generalization. Generalizing beyond the information and activities presented during training to novel goals, conditions, and resources is frequently a key training goal. However, the mental representations that best support this and the learning experiences most likely to produce these representations are not well understood.

Our work studies how to train participants to enable them to generalize to novel situations and activities in technical work domains, specifically understanding and using equipment. In technical work domains of this sort generalization to new tasks not used in training is improved if the training provides experience both with the principles or model explaining how the technology works and with procedures for accomplishing domain goals with the technology. In working with equipment, it is valuable to learn both about how a device works and about the procedures or methods for using the device.

Further, we claim that learning will be improved by training in which learners actively build relationships between a) the device model with its underlying principles and b) the procedures for operating the device. In particular, we propose that linking an understanding of the device with procedural skills for its operation will improve learners’ ability to generalize. Learners with integrated skills and knowledge of this sort will be better able to generalize their knowledge to new situations. They will be better able to accomplish work with goals, constraints, or resources different from than those encountered in training. For example, given training that only covered bringing up and powering down equipment, trainees with integrated training might be better able to generalize to a maintenance goal of testing a component. They might also be better able to generalize to unexpected conditions in which equipment was initially configured differently than had been encountered in training. They might also be better able to predict or explain what will happening in a situation or how equipment works. From a methodological point of view, both design of training methods and of transfer measures are challenging. Many factors impact training and still more the application of what has been learned to the generalization tasks, including individual differences.

This year we continued research on training for generalization using the domain of habitat equipment on the International Space Station (ISS). We aimed to further develop methods and measures, in a conceptual replication and extension of Experiment 1, reported last year. Both Experiments 1 & 2 compared a training condition emphasizing integration (Integrating Condition) with a training condition that taught the same device model components and the same procedural skill components but did not emphasize linking the two types. Last year, we informally observed individual differences in attitudes about how the task should be approached. For example, some participants seemed more to expect that they would be directed quite specifically in what to do and that good performance would be close compliance, while others seemed more to expect they might need to be proactive, solve problems, or “think on their feet” and that good performance would include flexibility. We thought that the more active, flexible learning stance would facilitate linking of information and thinking about how this could be applied in unexpected situations. Therefore, we added an additional difference between conditions.

In the Experiment 2 Integrating Condition, we added training designed to instill into participants a take-responsibility, problem-solving attitude. Training sections were added that stressed the participant’s responsibility for safety of the International Space Station; the need to learn how systems, procedures, and NASA guidance are related; and the need to generalize knowledge to many situations. They were told that the best operators are active, alert, and integrate what they know. They were taught that NASA priorities place safety first, mission objectives next, then efficiency, and then consistency; and that the most desirable thing was to prevent problems, next to recover from problems, and then to report problems to Mission Control. As well as directly instructing users about the importance of these goals, users were confronted with problems they needed to address. These problems, however, were not in any of the ISS systems used in the training and transfer tasks. Instead, an unrelated Rover camera device and procedure was introduced and this completely different device was used to illustrate several types of example problems (e.g., the camera memory was full). Thus we aimed to train participants in the integrating condition to expect and to solve problems, without giving any exposure to problems they would encounter in the training domain.

For the comparison condition in Experiment 2, as in the Integrating Condition, participants were told the NASA priorities of safety, mission goal, efficiency, and consistency; and the problem management strategies of prevent, recover, and report. However this information was not repeated and integrated with other material and participants did not apply the problem management strategies. Thus, as with information about the device model and procedures, the same facts were presented, in the comparison as well as integrating condition, but they were not linked to practice nor to other information. However, rather than simply removing any higher level motivation or goal, we thought a more interesting comparison would be to provide this for the control condition as well, should we be able to identify and appropriate “neutral” goal or motivation. The best alternative motivation we identified was efficiency. Thus, the control condition emphasized the importance of fluent execution and getting as much done as possible given limited time; further, participants here had additional practice with strategies for efficiency, including use of the automaton software. This condition is labeled the Fluency Condition.

We predicted that participants in the Integrating Condition would perform better on generalization tasks better than participants in the Fluency Condition; that this advantage would particularly be found for tasks less similar to those encountered in training; and that this would be found both for conceptual tasks and for operational tasks, particularly where operational tasks needed to draw on an understanding of how the device works.

Method. Our method is quite similar to that of Experiment 1, for which more detail is provided in the Task Book report for 2017. Concerning participants, we continued to use engineering students, primarily from astro-aero departments as our participants. Fifteen students participated in each condition.

Concerning domain, we continued to use the micro-world with simulated habitat equipment, including the Carbon Dioxide Removal System (CDRS), and the accompanying procedure execution software.

Concerning training, most of the materials and activity were quite similar to Experiment 1. We did expand ways that the Integrating and Fluency differed in integrating or separating procedural “how” information and model-based “why” information. Specifically, the Integrating Condition participants saw an animated functional model of the device, integrating how things worked with the device model, while the Fluency Condition participants saw an animated structural model of the device showing how components were made up of parts, to reduce the linkages to procedures suggested by the model. In addition, Experiment 2 provided different motivational emphasis, learning for flexibility and problem solving in the Integrating Condition versus learning for fluency.

Concerning measuring transfer, we modified several of our conceptual tasks to add more guidance on what we were looking for. We aimed to aid scoring and ensure participants were providing information on the topic intended. We added two operational tasks emphasizing fluency, to see if speed executing familiar tasks was all that was required and if participants in the Fluency Condition would perform better than in the Integrating Condition. Our coding was extended and revised on various tasks.

Results. Fortunately, most of our tasks are designed so that correctness and strategy are the primary measures rather than completion times. Due to a software problem, the time of computer actions, such as automatic execution of a procedure, increased very substantially over time, with the last third of users seeming to have considerably slowed performance. Therefore we have not prioritized tasks or measures dependent on software speed; we have not scored the two tasks designed to assess efficiency nor have we looked at completion times on any task. In addition, we have been investigating performance of the first cohort of 20 users, for whom the software operated closer to the expected speeds.

Scoring and analysis are still in progress, and findings are presented descriptively. Measures typically have large within-condition variability; condition differences are small on many measures; however, none of the measures scored to date suggest better performance in the Fluency Condition.

Several of the conceptual and operational tasks show patterns favoring the Integrating Condition. The quality of the narrative and diagram produced for the Conceptual Explanation task has been scored for the first 20 users. These Integrated Condition participants, mean of 26.6 9 (stdev = 3.4) points, scored notable higher than the Fluency participants , mean of 18.6 (stdev=6.8), particularly on the narrative component. The Required Order task asked participants to list ordered action pairs from the CDRS Activation Procedure, where the earlier action must come before the later action, because of the way the CDRS works. Four pairs of “boxes” were provided for the participant to write in responses. Conditions did not differ in the number or correctness of pairs. They did, however, differ in diversity. Four pairs giving information about valves, fuses, water pump, and blower provides broader coverage than would a four pairs each stating a fuse must be closed before Valve 1...4 is opened. The Integrating Condition had a summed diversity score of 6.9 (stdev= .79) types of components and actions compared to the Fluency Condition score of 5.4 (stdev=1.6). This pattern is of note as the three diversity variables correlate significantly with condition, suggesting a real difference. Two other conceptual tasks showed no evidence of condition differences. One, the procedure writing task, had shown a performance advantage in Experiment 1; Integrating Condition participants had a median of 14 on a 16-point scale, compared to a Fluency Condition median of 5.5. In Experiment 2, both conditions had a median of 14, apparently near ceiling.

On the procedural tasks, three were designed to require reasoning about how the device works to solve correctly, Tasks D, E, and H. On these three the conditions required by the procedure are not met and the participant must reason about what other (unnamed) procedures must be run first to allow successful execution of the procedure that accomplishes the goal. If the participant reasons or understands the situation initially, the task can be accomplished without producing action failures, what we termed “preventive solutions.” These are designed in increasing difficulty, where more indirect relations must be reasoned about. Conditions did not differ on the first of these problems, with performance pool in both (14% and 13% in Integrating and Fluency Conditions). Performance improved substantially on later problems (Task E & H) in the Integrating Condition, much more than in the Fluency Conditions (Task E: 46% vs. 13% and Task H: 43% vs. 21%). These differences are larger in the first cohort. A different type of operational task asked participants to open and close one fuse. This can be done by executing a small part of a startup and shutdown procedure. We were interested in whether participants formed the more precise method of executing only the relevant part of procedures or simply ran the whole procedure intact. In the Integrating Condition, 79% of users generated a selective strategy, versus 47% in the Fluency Condition.

Experiment 3.

We are conducting a retention study, in which participants in Experiments 1 or 2 return. We will assess their ability to carry out some of the same transfer tasks they did previously as well as some quite different tasks, about a Sabatier reactor. This will provide a window into retention of this material for durations of months to over a year.

In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 62). SAGE Publications. In press as of May 2018. , May-2018

We will conduct literature research followed by piloting and experimentation to a) define appropriate measures and manipulations and b) assess the extent that integrated S&K generalizes well and is well-retained relative to S&K covering related content but without the integrating relationships. Experimentation will aim to produce integrated knowledge though a variety of learning activities that a) build links from procedural skills to abstract principles through processes such as self-explanation, thus facilitating their decomposition and reuse and b) build links from abstract principles to specific procedures through processes such as instantiation, thus aiding their application. Evidence shows that integrated knowledge is better retained, as the many links of integrated knowledge provide multiple, alternative retrieval cues accessible in many contexts. We will assess whether manipulations that produce integrated knowledge also produce better generalization (as well as measuring retention), relative to knowledge that spans related content but is not integrated, and whether such effects can be attributed to the mediating role of integrated knowledge.

We will use the work domain of operating and trouble-shooting complex technology, specifically spacecraft life-support systems as the test case for the proposed research. We will investigate “performing system-related tasks in highly autonomous environments” (Human Exploration Research Opportunities (HERO) Appendix A-25), such as operation, maintenance, and troubleshooting components within habitat life-support systems. Three factors motivate this choice.1) Such work is highly relevant to long-distance, crewed missions. 2) Prior research has studied operation of devices and thus provides findings on representations supporting generalization in similar contexts. 3) We will use an existing software suite including simulation of life-support devices on the International Space Station (ISS) and procedures for their operation as our test bed. This “micro-world” enables experimental control yet is a close analog of real mission work. In this domain the critical skills and knowledge are primarily cognitive (e.g., deciding what procedure to use), rather than sensory-motor (e.g., how hard a wrench must be turned). We include “meta-knowledge” such as identifying what information is missing when resources such as just-in-time-training are relevant.

We will assess whether interventions related to integration produce forms of S&K that generalize well and are well retained, a prediction motivated by prior findings but not investigated directly, nor in NASA-relevant context. If integration is the basis for the predicted outcomes, it will provide a powerful principle for identifying and creating forms of S&K that are both generalizable and retainable. If not, we have discovered how learning is affected by important training interventions in a domain highly relevant to NASA future crewed exploration missions. This research will narrow the gap in understanding the factors that make skills and knowledge for NASA-critical tasks more generalizable and more retainable.

In addition, we claim that learning will be improved by training in which learners actively build relationships between a) the device model with its underlying principles and b) the procedures for operating the device. In particular, we propose that linking an understanding of the device with procedural skills for its operation will improve learners’ ability to generalize. Learners with integrated skills and knowledge of this sort will be better able to generalize their knowledge to new situations. They will be better able to accomplish work with goals, constraints, or resources different from those encountered in training. For example, given training that only covered bringing up and powering down equipment, trainees with integrated training might be better able to generalize to a maintenance goal of testing a component. They might also be better able to generalize to unexpected conditions in which equipment was initially configured differently than had been encountered in training. Because technical work is often complex and unpredictable, it can be impossible or impractical to train all tasks or situations. Thus generalizing to untrained cases is often very valuable.

Experiment Prediction.

Our experiment compared a training condition emphasizing integration (Integrated Condition) with a training condition that taught the same device model components and the same procedural skill components but did not emphasize linking the two (Component-wise). We predicted enhanced generalization in the Integrated versus Component-wise Condition, particularly as the novel tasks became increasingly different from those in training. An alternative possibility is that separating, rather than integrating, training of these two aspects produces better learning because it places less burden on working memory, and reduces cognitive load to appropriate levels.

Method.

Our experiment trained users how to operate simulated habitat equipment on the International Space Station, using procedure automation software to carry out procedures. It compared Integrated versus Component-wise Conditions, a between-subjects factor. Conditions provided different training and performance on novel, generalization tasks during the transfer phase was assessed.

Participants. We recruited students majoring in aero-astronautics engineering. We believed this population of participants would likely be familiar with basic engineering concepts and skills, such as familiarity with schematic representation of devices and with learning complex software. Students were recruited from two universities and ranged from undergraduate to Ph.D. students. The Integrated Condition had 13 participants and the Component-wise had 14.

Materials. The experiment used PRIDE procedure software and associated simulation developed by TRAClabs.

Both conditions had three main components of the training materials: expository slides, slides presenting questions for the participant to answer orally for the experimenter, and directed hands-on use of the software suite. These materials were closely matched for the two Conditions, except for differences carefully introduced to aid integration. Two key differences were organization of material and the nature of “paper and pencil” study exercises. The Integrated Condition interleaved instruction about the systems to be controlled with instruction about the procedures and procedure software used to operate the systems; it provided exercises that required users to map back and forth between device schematics and operational procedures and others that required reasoning about how device principles constrained the order of procedure steps. The Component-wise Condition blocked instruction by topic and provided study exercises based on identifying elements either in schematics or in procedures and recall rather than inference.

Both conditions had the same transfer tasks. The four tasks presented immediately after training all were tasks executing procedures to operate equipment. Task 1 was very similar to activities in training, requiring little generalization. Task 2 differed from training activities in requiring participants to identify and execute parts of a procedure. Tasks 3 and 4 differed in that the conditions required for successful execution were not met, and participants had to reason about what procedures would change these conditions so that the task goals could be achieved. Task 5 differed in the nature of the work: participants were asked to write a procedure for a novel goal of testing a valve. After drafting their procedure they were asked to now execute the valve test. Additional transfer tasks asked users to reason about system states and learn about a new device.

Procedure. The experiment was conducted at NASA Ames, was scheduled for 6.5 hours (though sometimes completed in less time) including breaks. Participants worked through the training slides and exercises, self-paced but with experimenter guidance and feedback. The goal was to ensure each participant learned the material well enough to answer the questions posed in the slides and do the hands-on exercises successfully. Following training, participants in both conditions did the identical transfer tasks.

Coding. For each execution task we built a coding scheme identifying key behaviors and allowing categorization of accomplishing or failing to accomplish the task goals. We then coded the activity stream from logfiles, to determine task success; for successful trials we compared execution times.

Results

Broadly, tasks were of appropriate difficulty and participants were engaged in the experiment. Individual difference were large, but patterns of performance on the transfer tasks suggested better generalization in the Integrated than Component-wise Condition. As expected, performance on both success rate and time was similar in both conditions for Task 1.

Two tasks required new manipulation of software, either using components of a procedure in Task 2 or designing a new procedure from components in Task 5. Success rate on Task 2 was 92% in the Integrated versus 50% percent in the Component-wise, a reduction in failure to accomplish task goals of 42%. Success rate on Task 5 was 55% in the Integrated versus 22 % percent in the Component-wise, a reduction in failure to accomplish task goals of 33%. Execution times were similar between conditions.

Tasks 3&4 required reasoning about how to identify what procedures should be run in order to prevent procedure failures and correct the conditions needed for procedures named in the task. Execution times on successful trials were 7m50sec and 7m07sec in the Integrated Condition but 12m02sec and 9m40sec in the Component-wise Condition.

Discussion

This study found suggestive evidence of better generalization following training that emphasized integration. The bulk of training was the same in both condition; the same information and activities were provided about the systems to be controlled and about the software and procedures for controlling them. The differences introduced to foster integration were relatively modest. Nevertheless, performance patterns showed in increased success rate and decreased times for the Integrated Condition.

Future work is needed to replicate and extend these findings, with larger numbers of participants and perhaps participants with more homogeneous initial knowledge as indexed by current education level. We also plan to use an initial, engineering knowledge assessment. In addition, we will assess retention over periods of weeks of these skills and knowledge including the ability to flexibly apply them to novel situations.

In: Proceedings of the 38th Annual Conference of the Cognitive Science Society (Papafragou, A., Grodner, D., Mirman, D., & Trueswell, J.C. (Eds.). Austin, TX: Cognitive Science Society, 2016. , Aug-2016

We will conduct literature research followed by piloting and experimentation to a) define appropriate measures and manipulations and b) assess the extent that integrated S&K generalizes well and is well-retained relative to S&K covering related content but without the integrating relationships. Experimentation will aim to produce integrated knowledge though a variety of learning activities that a) build links from procedural skills to abstract principles through processes such as self-explanation, thus facilitating their decomposition and reuse and b) build links from abstract principles to specific procedures through processes such as instantiation, thus aiding their application. Evidence shows that integrated knowledge is better retained, as the many links of integrated knowledge provide multiple, alternative retrieval cues accessible in many contexts. We will assess whether manipulations that produce integrated knowledge also produce better generalization (as well as measuring retention), relative to knowledge that spans related content but is not integrated; and whether such effects can be attributed to the mediating role of integrated knowledge.

We will use the work domain of operating and trouble-shooting complex technology, specifically spacecraft life-support systems as the test case for the proposed research. We will investigate “performing system-related tasks in highly autonomous environments” (Human Exploration Research Opportunities (HERO) Appendix A-25), such as operation, maintenance, and troubleshooting components within habitat life-support systems. Three factors motivate this choice.1) Such work is highly relevant to long-distance, crewed missions. 2) Prior research has studied operation of devices and thus provides findings on representations supporting generalization in similar contexts. 3) We will use an existing software suite including simulation of life-support devices on the International Space Station (ISS) and procedures for their operation as our test bed. This “micro-world” enables experimental control yet is a close analog of real mission work. In this domain the critical skills and knowledge are primarily cognitive (e.g., deciding what procedure to use), rather than sensory-motor (e.g., how hard a wrench must be turned). We include “meta-knowledge” such as identifying what information is missing when resources such as just-in-time-training are relevant.

We will assess whether interventions related to integration produce forms of S&K that generalize well and are well retained, a prediction motivated by prior findings but not investigated directly, nor in NASA-relevant context. If integration is the basis for the predicted outcomes, it will provide a powerful principle for identifying and creating forms of S&K that are both generalizable and retainable. If not, we have discovered how learning is affected by important training interventions in a domain highly relevant to NASA future crewed exploration missions. This research will narrow the gap in understanding the factors that make skills and knowledge for NASA-critical tasks more generalizable and more retainable.

To support the required increase in self-reliance, crew must be trained for altered operations. They must be able to generalize and adapt old skills and knowledge to transfer to new situations. While important for all human spaceflight, the skills and knowledge to support such transfer, as well as to retain the specifics of what one has been taught, are particularly critical for operations far from earth. Currently, NASA recognizes that errors from training deficiencies threaten the success of planetary missions, described in the SHFE “Risk of Performance Errors Due to Training Deficiencies”:

Given that training content, timing, intervals, and delivery methods must support crew task performance, and given that training paradigms will be different for long-duration missions with increased crew autonomy, there is a risk that operators will lack the skills or knowledge necessary to complete critical tasks, resulting in flight and ground crew errors and inefficiencies, failed mission and program objectives, and an increase in crew injuries. http://humanresearchroadmap.nasa.gov/risks/?i=166

Of particular concern, NASA has traditionally emphasized training for specific skills and has limited experience training for generalizable skills and knowledge while also supporting retention. Indeed, the science of learning and training provides only incomplete understanding of the types of knowledge and skills that are needed, as recognized in the SHFE-TRAIN-04 gap:

SHFE-TRAIN-04: We do not know the types of skills and knowledge that can be retained and generalized across tasks for a given mission to maximize crew performance. http://humanresearchroadmap.nasa.gov/Gaps/?i=605

Our approach emphasized the value of building rich relations among components of skills and knowledge, to build an integrated representation. We suggest that a learner equipped with integrated representations of procedural and conceptual knowledge will be particularly able to transfer or generalize, that is, to perform well on novel tasks and situations. We investigate these claims in the domain of operating complex equipment.

We hypothesize that when procedural knowledge from operating a device by executing procedures is integrated with conceptual knowledge about how the device works, the learner will be better able to transfer to new tasks and new devices. More specifically, training which emphasizes integration will result in better transfer.

In the first year of the grant many of our key results are methodological. We have developed contrasting training methods to generate integrated versus component-wise representation. We will have modified an existing, simulated task-environment to increase its fidelity as needed for our use. We have conducted a task analysis of the work, identifying the component actions, information, and sub-goals. This analysis allows us to compare the components included in alternative training methods and to identify the components needed for transfer tasks that are or are not taught in the training phase. We will have extended our current set of transfer tasks and environments to include materials that convey a device model (such as schematics) and device procedures for a novel device different from that used in training. In addition we will have piloted to adjust difficulty, and gathered an initial data set.

We will conduct literature research followed by piloting and experimentation to a) define appropriate measures and manipulations and b) assess the extent that integrated S&K generalizes well and is well-retained relative to S&K covering related content but without the integrating relationships. Experimentation will aim to produce integrated knowledge though a variety of learning activities that a) build links from procedural skills to abstract principles through processes such as self-explanation, thus facilitating their decomposition and reuse and b) build links from abstract principles to specific procedures through processes such as instantiation, thus aiding their application. Evidence shows that integrated knowledge is better retained, as the many links of integrated knowledge provide multiple, alternative retrieval cues accessible in many contexts. We will assess whether manipulations that produce integrated knowledge also produce better generalization (as well as measuring retention), relative to knowledge that spans related content but is not integrated; and whether such effects can be attributed to the mediating role of integrated knowledge.

We will use the work domain of operating and trouble-shooting complex technology, specifically spacecraft life-support systems as the test case for the proposed research. We will investigate “performing system-related tasks in highly autonomous environments” (HERO Appendix A-25), such as operation, maintenance, and troubleshooting components within habitat life-support systems. Three factors motivate this choice.1) Such work is highly relevant to long-distance, crewed missions. 2) Prior research has studied operation of devices and thus provides findings on representations supporting generalization in similar contexts. 3) We will use an existing software suite including simulation of life-support devices on the International Space Station (ISS) and procedures for their operation as our test bed. This “micro-world” enables experimental control yet is a close analog of real mission work. In this domain the critical skills and knowledge are primarily cognitive (e.g., deciding what procedure to use), rather than sensory-motor (e.g., how hard a wrench must be turned). We include “meta-knowledge” such as identifying what information is missing when resources such as just-in-time-training are relevant.

We will assess whether interventions related to integration produce forms of S&K that generalize well and are well retained, a prediction motivated by prior findings but not investigated directly, nor in NASA-relevant context. If integration is the basis for the predicted outcomes, it will provide a powerful principle for identifying and creating forms of S&K that are both generalizable and retainable. If not, we have discovered how learning is affected by important training interventions in a domain highly relevant to NASA future crewed exploration missions. This research will narrow the gap in understanding the factors that make skills and knowledge for NASA-critical tasks more generalizable and more retainable.