This project relies on strong teaming, and is organized with system repair studies taking place at University of California (UC) Davis and robotics studies at Massachusetts Institute of Technology (MIT). At UC Davis, we have completed the Refresher Training (Part A) segment of the proposal, including subject selection, aptitude screening, initial training, and performance evaluation on the complex system repair task, then re-evaluation after a six-month period, refresher training (half the subject group developed their own refresher training), and performance evaluation with a total of 16 subjects. We have also developed and benchmarked quantitative methods for subject performance evaluation. These include assessing subtask timing as subjects work through the repair task, procedure flow, and a detailed taxonomy of error types. Major effort has been made to make these evaluations objective, so that different researchers can analyze the same subject video and produce virtually the same results.

We have also instrumented subjects' hands with three-axis accelerometers to gather hand-motion data as they repair the surrogate system. Since we have defined task performance in terms of elapsed time for each sub-task, and also in terms of deviations from procedures (errors), we developed a new and detailed taxonomy of procedural deviations, which allowed us to quantify the type and sequence of errors made by the subjects. Results of the refresher experiment show that use of self-made refresher videos after 6 months since initial training resulted in only a slight improvement in overall performance for subjects in the treatment group, but that the types of errors made are significantly different between the control and treatment groups.

At MIT, the manually-commanded robotics version of the proposed refresher training has also been completed. At the end of initial robotics skills and task training, subjects in the experimental group created a short reminder video to be used as a review when they came back in six months. Subjects then completed a hands-on evaluation that incorporated all the skills and theories taught in training. The evaluation performance was assessed with a qualitative rubric based on NASA Johnson Space Center (JSC) robotics training scoring strategies, and with quantitative measurements based on simulator outputs. At the six month evaluation, control subjects only had access to the procedures and a PowerPoint based written skill review, while the experimental group had access to their personalized review video and the procedures. Results showed that metrics of a subject's spatial abilities predicted performance and retention in procedurally complex tasks. Spatial ability had more effect on the control group's retention than those who received customized retraining.

Part B of the project was to develop and assess a method of Just-In-Time Training (JITT), such as will be required for long-duration astronauts challenged with critical tasks for which they have not been trained. In this case, ground-learned skills must be integrated into task-capable expertise through JITT. After considering a variety of methods, a self-customization approach was settled upon, in which subjects individually adjust their procedures by selecting from three levels of detail for each sub-task. After selection screening, subjects underwent skills training, and were then presented with a new and complex task to perform with the use of procedures. The control group could not alter their procedures while the treatment group could customize them.

UC Davis Part B experiments with complex-system repair tasks are ongoing, while at MIT, early robotics results suggest that provision of customized JITT enables subjects to perform the critical task more quickly than their (non-customized JITT) counterparts. Further, the control group has a markedly higher rate of task errors, as well as a higher response time to the side-task. It is speculated that the customization of JITT reduces subjects' workload and error rate, as they will not spend extraneous attentional resources perusing detailed procedural lists for sub-tasks in which they already demonstrate excellence.

- Completed analysis and internal reporting for both complex-system repair at UC Davis and manual-operated robotics at MIT

- Developed and benchmarked quantitative methods for subject performance evaluation. These include assessing subtask timing as subjects work through the repair task, procedure flow, and a detailed taxonomy of error types. Major effort has been made to make these evaluations objective, so that different researchers can analyze the same subject video and produce virtually the same results.

- Continued development of an instrumentation system for subjects' hands, consisting of three-axis accelerometers to gather hand-motion data as they repair the surrogate system.

- Developed a new and detailed taxonomy of procedural deviations, which allowed us to quantify the type and sequence of errors made by the subjects.

Part B: Just-In-Time Training

- Review and consideration of candidate techniques for Just-In-Time Training resulted in novel concept of self-customization of procedures by subjects faced with an unfamiliar, complex task.

- Software developed to expand/contract procedures on command, by sub-task, to three levels of increasing detail and multimedia support.

- Completed experiment design for both system repair and robotics studies.

- Recruited subjects and completed training and critical-task evaluation for both control and treatment groups using the Space Station Remote Manipulator System (SSRMS) simulator in the Man Vehicle Lab at MIT.

- Through analysis of surrogate critical repair task, extracted core skills required for pre-training.

- Designed and fabricated a part-task training device for skills-training in complex-systems repair in the Human/Robotics/Vehicle Integration and Performance Lab at UC Davis.

The overall objectives of our research are addressed in two parts: Part A: Self-Produced Refresher training for re-acquisition of expert performance. Part B: Customized Just-in-Time training for tasks that have not been specifically trained previously, but require the integration of existing astronaut skills.

The second (current) year (2015/16) of the grant has addressed Part A studies only. Part B will be the focus of Year 3. Part A is therefore reported upon herein. Our research considers two spaceflight-appropriate tasks, one requiring the repair of a complex electro-mechanical system representative of those found aboard spacecraft, and the other requiring manual control of a simulated International Space Station (ISS) robotic arm. This project was launched in the Fall of 2014, and is organized with systems repair studies at University of California (UC) Davis and robotics studies at Massachusetts Institute of Technology (MIT), with the two teams working concurrently and in close contact.

During Part A, for both tasks, we developed training materials which follow the NASA style of briefings, procedures, demonstration, and hands-on practice. 18 subjects each at UC Davis and at MIT received initial training with these materials, and their baseline performance was evaluated. Per our experiment design, after training and final evaluation (at peak of expertise) half of each group was asked to make a short summary video of themselves describing their training. After a period of about 6 months, subjects will return to the lab, review the training materials (with half of each group viewing the refresher video they made six months previously), and be re-evaluated on how well they have retained or re-acquired their task skills.

Key Findings for Part A (Refresher Training): For the system-repair task at UC Davis, we spent significant effort developing lab logistics, selecting the spacecraft-system surrogate, writing and testing procedures, and developing training materials. 18 subjects completed the first phase of testing in November 2015, and are about to return (after 6 months) for the refresher phase, beginning May 9, 2016. During the six months between subject testing phases, the UC Davis team has been developing techniques to objectively quantify subject performance on the basis of time, accuracy, and hand-movements. A formal hierarchical task analysis was undertaken, resulting in the segmentation of each subject's performance into a collection of timed subtasks with associated hand-acceleration statistics binned by subtask. In-process errors are also recorded by type, influence, and duration. The development of these performance quantification techniques have given us an objective (independent of the evaluator) toolset for collecting performance statistics that may then be compared over a variety of experimental independent variables, such as training method, subject background, or training environment. For the robotics task at MIT, an experimental apparatus (robot-arm simulator) was pre-existing, so the Part A experiment design was executed and 18 subjects underwent initial training, a 6-month delay, and re-evaluation as described above. Statistical analysis of subject performance results is now underway, and will soon be complete as the graduate student (Lynne Geiger) completes her thesis. Preliminary results suggest that the self-made refresher video improved performance for at least one robotic sub-task, but with mixed results for other sub-tasks.

Impact of key findings on hypotheses, technology requirements, objectives, and specific aims of the original proposal: During the course of Year 2, preliminary findings have raised the following experiment design issues:

- How best to instruct a subject in self-producing a refresher video (specific instructions, time limit, editing limits, standardization across subjects). Based on the MIT experiences, which came first, the entire team developed strict guidelines for the above.

- How to organize and sequence re-training after the 6-month break – the result was a specific strategy to minimize uncontrolled variables and document subjects' use of training materials as they refresh.

- For the systems repair team at UC Davis, we added (at Lynne Geiger's suggestion) instrumentation to characterize subjects' hand movement via glove-mounted accelerometers. Although not included in the original proposal, literature showed that surgeons' hand-movements were correlated with training state, and so the additional measurement class was added here.

Proposed research plan for the coming year: Complete Part A, Phase 2 (refresher training) subject testing for complex system repair (UC Davis, 18 subjects) ; Complete performance analysis and hypothesis evaluation for Part A, both tasks ; Based upon Part A results, modify and execute Part B (Just-in-Time Training) subject testing at UC Davis (system repair) and at MIT (robotics).

Publications: planned journal submittals in 2016: Human performance quantitative evaluation for spaceflight (combined UC Davis and MIT) ; Part A results (UC Davis) ; Part A results (MIT).

The project will begin with the development of two tasks that represent typical complex and critical activities that are carried out by astronauts in space relatively infrequently. We will develop a set of training materials that follow the NASA style of briefings, procedures and hands-on practice. Once these tasks are developed, we will conduct an experiment with human subjects to determine if customized, self-made video training materials prove to be better refresher training materials than the generic materials that would typically be used by crewmembers. Subjects will receive initial training and their baseline performance evaluated. After a period of about 6 months, subjects will return to the lab, review the refresher training materials, and be re-evaluated on how well they have retained or re-acquired their task skills.

A second experiment will be conducted using the same two representative tasks, but will examine if the customized training materials developed in the first experiment could be used as just-in-time training materials for a new group of subjects with basic training, but no specific training in the given task. In both experiments, we also examine the correlation of subject learning styles with the content of the training materials, to determine whether learning style is a useful characterization for developing customized content. The results of this research project will provide operational guidelines and pedagogy for developing customized video training for astronauts on long-duration missions beyond Earth orbit. Since subject testing has just started, key findings are not yet available. For the coming year, the team will complete the experiment design, procedure development, subject testing, and preliminary analysis of results for Part A of the project (refresher training utilizing subject-produced video training synopses).

In the first year, the UC Davis section of the research team hired two undergraduate research assistants and two graduate research assistants to help develop the test facilities and tool station, select the mechanical system to be used for the repair task, develop procedures, write subject protocols and instructions, apply for UC Davis Institutional Review Board (IRB) approval, and identify video equipment and tools to purchase. In selecting the repair task, one of the drivers was that the equipment could be put into operation for testing after subject repair. After considering a variety of candidates (air compressor, water pump, cooling system pump, four-barrel carburetor, aircraft magneto, etc), we settled on a gas-powered electrical generator with its integration of mechanical and electrical systems, plus modern design and repair manuals to work from. Also during this first year, the MIT side of the team hired a graduate research assistant, and a freshman undergraduate student. Lynn Geiger (the GRA) traveled to NASA Johnson Space Center (JSC) in August 2014 to participate in a 1 week Generic Robotics Training (GRT) course set up by the Robotics Training Group. We developed an experimental protocol for pilot testing of our customized refresher training for a simulated set of International Space Station (ISS) robotic operations, including the robotic task simulations, training materials including videos and evaluation rubrics. Subjects will be trained to perform two specific robotics tasks and their performance measured to establish a baseline. Training consists of 4 3-hour sessions over three days. To date, 2 pilot subjects were tested at MIT in December and reevaluated 2 months later to develop a preliminary understanding of skill deterioration.. Based on the feedback and responses of these pilot subjects, the protocol and scenarios were modified to increase the overall practice time and increased strategy training to help solidify the subject's understanding. Two more pilot subjects have completed the training and baseline evaluation with the new protocol and will be retested in 6 months. We plan to test 18 additional subjects before the end of the May 2015.

Our overall objective is to develop novel, context-sensitive, and customized onboard training techniques that can be adapted to different tasks and crewmembers, with ability to address both refresher training (for re-acquisition of expert performance) and just-in-time training (for tasks that have not been specifically trained previously, but require the integration of existing astronaut skills). To achieve this objective, we propose to test the hypothesis that multimedia training which is customized for the crewmember can be more efficient than traditional, generic format training for the same measured effectiveness.

The project will begin with the development of two tasks that represent typical complex and critical activities that are carried out by astronauts in space relatively infrequently. We will develop a set of training materials that follow the NASA style of briefings, procedures and hands-on practice. Once these tasks are developed, we will conduct an experiment with human subjects to determine if customized, self-made video training materials prove to be better refresher training materials than the generic materials that would typically be used by crewmembers. Subjects will receive initial training and their baseline performance evaluated. After a period of about 6 months, subjects will return to the lab, review the refresher training materials, and be re-evaluated on how well they have retained or re-acquired their task skills. A second experiment will be conducted using the same two representative tasks, but will examine if the customized training materials developed in the first experiment could be used as “just-in-time” training materials for a new group of subjects with basic training, but no specific training in the given task. In both experiments, we also examine the correlation of subject learning styles with the content of the training materials, to determine whether learning style is a useful characterization for developing customized content. The results of this research project will provide operational guidelines and pedagogy for developing customized video training for astronauts on long-duration mission beyond Earth orbit.