Task Progress:
|
Year 3 Annual Report 2018- Training for Generalizable Skills & Knowledge: Integrating Principles and Procedures (#NNX15AP26G)
Technical work across science, technology, engineering, and mathematics (STEM) disciplines, on Earth as in space, requires mastery of complex suites of knowledge and skills. Whether presented as education or as job-specific training, these skills and knowledge must be learned. Much of this work has an open, changing character such that even if time were available, it is not possible to anticipate and teach all the components that will be needed in the work. Thus, training that allows a person effectively to transfer skills and knowledge to situations and tasks that were not trained is extremely valuable.
Our work investigates how to produce and to measure skills and knowledge that support significant generalization. Generalizing beyond the information and activities presented during training to novel goals, conditions, and resources is frequently a key training goal. However, the mental representations that best support this and the learning experiences most likely to produce these representations are not well understood.
Our work studies how to train participants to enable them to generalize to novel situations and activities in technical work domains, specifically understanding and using equipment. In technical work domains of this sort generalization to new tasks not used in training is improved if the training provides experience both with the principles or model explaining how the technology works and with procedures for accomplishing domain goals with the technology. In working with equipment, it is valuable to learn both about how a device works and about the procedures or methods for using the device.
Further, we claim that learning will be improved by training in which learners actively build relationships between a) the device model with its underlying principles and b) the procedures for operating the device. In particular, we propose that linking an understanding of the device with procedural skills for its operation will improve learners’ ability to generalize. Learners with integrated skills and knowledge of this sort will be better able to generalize their knowledge to new situations. They will be better able to accomplish work with goals, constraints, or resources different from than those encountered in training. For example, given training that only covered bringing up and powering down equipment, trainees with integrated training might be better able to generalize to a maintenance goal of testing a component. They might also be better able to generalize to unexpected conditions in which equipment was initially configured differently than had been encountered in training. They might also be better able to predict or explain what will happening in a situation or how equipment works. From a methodological point of view, both design of training methods and of transfer measures are challenging. Many factors impact training and still more the application of what has been learned to the generalization tasks, including individual differences.
This year we continued research on training for generalization using the domain of habitat equipment on the International Space Station (ISS). We aimed to further develop methods and measures, in a conceptual replication and extension of Experiment 1, reported last year. Both Experiments 1 & 2 compared a training condition emphasizing integration (Integrating Condition) with a training condition that taught the same device model components and the same procedural skill components but did not emphasize linking the two types. Last year, we informally observed individual differences in attitudes about how the task should be approached. For example, some participants seemed more to expect that they would be directed quite specifically in what to do and that good performance would be close compliance, while others seemed more to expect they might need to be proactive, solve problems, or “think on their feet” and that good performance would include flexibility. We thought that the more active, flexible learning stance would facilitate linking of information and thinking about how this could be applied in unexpected situations. Therefore, we added an additional difference between conditions.
In the Experiment 2 Integrating Condition, we added training designed to instill into participants a take-responsibility, problem-solving attitude. Training sections were added that stressed the participant’s responsibility for safety of the International Space Station; the need to learn how systems, procedures, and NASA guidance are related; and the need to generalize knowledge to many situations. They were told that the best operators are active, alert, and integrate what they know. They were taught that NASA priorities place safety first, mission objectives next, then efficiency, and then consistency; and that the most desirable thing was to prevent problems, next to recover from problems, and then to report problems to Mission Control. As well as directly instructing users about the importance of these goals, users were confronted with problems they needed to address. These problems, however, were not in any of the ISS systems used in the training and transfer tasks. Instead, an unrelated Rover camera device and procedure was introduced and this completely different device was used to illustrate several types of example problems (e.g., the camera memory was full). Thus we aimed to train participants in the integrating condition to expect and to solve problems, without giving any exposure to problems they would encounter in the training domain.
For the comparison condition in Experiment 2, as in the Integrating Condition, participants were told the NASA priorities of safety, mission goal, efficiency, and consistency; and the problem management strategies of prevent, recover, and report. However this information was not repeated and integrated with other material and participants did not apply the problem management strategies. Thus, as with information about the device model and procedures, the same facts were presented, in the comparison as well as integrating condition, but they were not linked to practice nor to other information. However, rather than simply removing any higher level motivation or goal, we thought a more interesting comparison would be to provide this for the control condition as well, should we be able to identify and appropriate “neutral” goal or motivation. The best alternative motivation we identified was efficiency. Thus, the control condition emphasized the importance of fluent execution and getting as much done as possible given limited time; further, participants here had additional practice with strategies for efficiency, including use of the automaton software. This condition is labeled the Fluency Condition.
We predicted that participants in the Integrating Condition would perform better on generalization tasks better than participants in the Fluency Condition; that this advantage would particularly be found for tasks less similar to those encountered in training; and that this would be found both for conceptual tasks and for operational tasks, particularly where operational tasks needed to draw on an understanding of how the device works.
Method. Our method is quite similar to that of Experiment 1, for which more detail is provided in the Task Book report for 2017. Concerning participants, we continued to use engineering students, primarily from astro-aero departments as our participants. Fifteen students participated in each condition.
Concerning domain, we continued to use the micro-world with simulated habitat equipment, including the Carbon Dioxide Removal System (CDRS), and the accompanying procedure execution software.
Concerning training, most of the materials and activity were quite similar to Experiment 1. We did expand ways that the Integrating and Fluency differed in integrating or separating procedural “how” information and model-based “why” information. Specifically, the Integrating Condition participants saw an animated functional model of the device, integrating how things worked with the device model, while the Fluency Condition participants saw an animated structural model of the device showing how components were made up of parts, to reduce the linkages to procedures suggested by the model. In addition, Experiment 2 provided different motivational emphasis, learning for flexibility and problem solving in the Integrating Condition versus learning for fluency.
Concerning measuring transfer, we modified several of our conceptual tasks to add more guidance on what we were looking for. We aimed to aid scoring and ensure participants were providing information on the topic intended. We added two operational tasks emphasizing fluency, to see if speed executing familiar tasks was all that was required and if participants in the Fluency Condition would perform better than in the Integrating Condition. Our coding was extended and revised on various tasks.
Results. Fortunately, most of our tasks are designed so that correctness and strategy are the primary measures rather than completion times. Due to a software problem, the time of computer actions, such as automatic execution of a procedure, increased very substantially over time, with the last third of users seeming to have considerably slowed performance. Therefore we have not prioritized tasks or measures dependent on software speed; we have not scored the two tasks designed to assess efficiency nor have we looked at completion times on any task. In addition, we have been investigating performance of the first cohort of 20 users, for whom the software operated closer to the expected speeds.
Scoring and analysis are still in progress, and findings are presented descriptively. Measures typically have large within-condition variability; condition differences are small on many measures; however, none of the measures scored to date suggest better performance in the Fluency Condition.
Several of the conceptual and operational tasks show patterns favoring the Integrating Condition. The quality of the narrative and diagram produced for the Conceptual Explanation task has been scored for the first 20 users. These Integrated Condition participants, mean of 26.6 9 (stdev = 3.4) points, scored notable higher than the Fluency participants , mean of 18.6 (stdev=6.8), particularly on the narrative component. The Required Order task asked participants to list ordered action pairs from the CDRS Activation Procedure, where the earlier action must come before the later action, because of the way the CDRS works. Four pairs of “boxes” were provided for the participant to write in responses. Conditions did not differ in the number or correctness of pairs. They did, however, differ in diversity. Four pairs giving information about valves, fuses, water pump, and blower provides broader coverage than would a four pairs each stating a fuse must be closed before Valve 1...4 is opened. The Integrating Condition had a summed diversity score of 6.9 (stdev= .79) types of components and actions compared to the Fluency Condition score of 5.4 (stdev=1.6). This pattern is of note as the three diversity variables correlate significantly with condition, suggesting a real difference. Two other conceptual tasks showed no evidence of condition differences. One, the procedure writing task, had shown a performance advantage in Experiment 1; Integrating Condition participants had a median of 14 on a 16-point scale, compared to a Fluency Condition median of 5.5. In Experiment 2, both conditions had a median of 14, apparently near ceiling.
On the procedural tasks, three were designed to require reasoning about how the device works to solve correctly, Tasks D, E, and H. On these three the conditions required by the procedure are not met and the participant must reason about what other (unnamed) procedures must be run first to allow successful execution of the procedure that accomplishes the goal. If the participant reasons or understands the situation initially, the task can be accomplished without producing action failures, what we termed “preventive solutions.” These are designed in increasing difficulty, where more indirect relations must be reasoned about. Conditions did not differ on the first of these problems, with performance pool in both (14% and 13% in Integrating and Fluency Conditions). Performance improved substantially on later problems (Task E & H) in the Integrating Condition, much more than in the Fluency Conditions (Task E: 46% vs. 13% and Task H: 43% vs. 21%). These differences are larger in the first cohort. A different type of operational task asked participants to open and close one fuse. This can be done by executing a small part of a startup and shutdown procedure. We were interested in whether participants formed the more precise method of executing only the relevant part of procedures or simply ran the whole procedure intact. In the Integrating Condition, 79% of users generated a selective strategy, versus 47% in the Fluency Condition.
Experiment 3.
We are conducting a retention study, in which participants in Experiments 1 or 2 return. We will assess their ability to carry out some of the same transfer tasks they did previously as well as some quite different tasks, about a Sabatier reactor. This will provide a window into retention of this material for durations of months to over a year.
|