NOTE: End date changed to 12/31/2012 per NSSC information (Ed., 6/1/2012)

The research proceeded along five partially parallel tracks: (1) developing the function allocation tradeoff model, (2) carrying out empirical human in the loop (HITL) research, (3) developing and (4) validating a computational model of the robotics operator, and (5) implementing the model in the context of the MIDAS-FAST tool. These five major components will be described separately:

1. Function allocation model. A key aspect of function allocation between human and automation is the degree of automation: that is, the relative amount of perceptual, cognitive, and motor “work” carried out by the automation versus human in their collaborative effort in completing task goals. A taxonomy of stages and levels of automation developed by Parasuraman, Sheridan, and Wickens (2000, 2008) describes this degree of automation. One of the important components of the degree of automation is the stage of task information processing at which automation operates to support or replace human activity. Earlier stages involve information acquisition and integration to support situation assessment. Later stages involve action selection and implementation to support task completion. The function allocation tradeoff model underlying FAST proposes that later stages of automation better support routine human-system performance and lower human workload. However these later stages become more problematic if automation fails to perform its functions appropriately, a failure caused in part by the loss of situation awareness. Our review of the literature on human-automation interaction, incorporated into a meta-analysis, supported these tradeoff relationships with statistically significant trends; and the guidance from this FAST tradeoff model have been incorporated into the MIDAS FAST tool (see Items 4 and 5 below).

Using this model, we identified several different types of automation to include in the robotic simulation. These required modifications to the existing BORIS software. Trajectory control automation was implemented in one of three degrees: manual, guided, and automated. To help ensure consistency in experimental participant behavior, we developed 3-segment trajectories that crossed a table (an obstruction) in the operating environment. The first and third segments required movement in 1 axis only; the second segment required movement along 2 axes. Manual trajectories were performed without guidance being given to participants. They were informed of the trajectory to follow, but they were required to determine how to implement it, and to move the arm using the hand controllers. In the automation guided condition, participants were shown a trajectory (or “flight path”) to follow. In the autocontrol condition, the trajectory was shown, and the automation executed the trajectory. Hazard alerting and hazard avoidance automation were identified and included. Hazard alerting included color coding to indicate to participants when they had encountered a no-fly zone; hazard avoidance included the color coding as well as stopping the arm to prevent a collision. Camera recommendation logic was also developed. Manual camera control required the operator to make decisions about camera selection, whereas the camera recommendation automation provided a visual alert to suggest a camera switch when needed, and recommended which camera to use. These different types of automation allowed us to research different stages and levels of automation as identified by our framework.

2. Three empirical studies were performed to investigate human performance with different types of robotic arm system automation. The first experiment examined different interface designs, including enhanced (over the current BORIS simulation) graphics for presenting hazards, integrated graphical hazard alerting, and tactile alerting. The second experiment - used for model parameterization and validation - matched the modeling conditions, and examined human performance in conditions with different degrees of automation and with unreliable automation. The third experiment (also used for model validation efforts) investigated adaptive versus adaptable versus fixed automation.

3. The team developed human performance models of scenarios of interest, based on robotic arm task analyses performed in cooperation with subject matter experts (SMEs). The team verified the task analyses by talk-through sessions with SMEs. Human performance model and human-automation interaction predictions were validated in empirical, Human in the loop (HITL) studies identified in Item 2. Results of the validations were used to refine the models. The models included sub models (also referred to as modules) to predict operator visual scanning, operator performance decrements due to poor camera views, and operator decision making. The scanning model is based on SEEV (Salience, Expectancy, Effort and Value) and the performance impacts of camera view quality were predicted using FORT (Frame of Reference Transformation). SEEV and FORT are relatively mature models, having been developed, refined, and validated under previous NASA research efforts. The decision model was developed specifically for tasks associated with the robotic arm, based on the Generic Robotics Training.

4. A primary goal of this research was to verify and validate our model of the robotic arm operator, to be employed in the function allocation tool, and to collect data that would further validate the Function Allocation Support Tool tradeoff model. To accomplish these purposes, data from the Human in the loop (HITL) Experiments 2 and 3 were analyzed, and both models developed and refined.

5. One particular focus of the project was on developing the MIDAS-FAST tool, a prototype model- and simulation-based product that is both usable and useful for researchers, allowing them to easily modify robotic arm scenarios and evaluate different potential automation conditions. This tool offers data entry screens that guide the user through the process of building a scenario. It allows the researchers to specify numerous relevant factors, e.g., operators, tasks, environmental conditions, and function allocation strategy. It offers a visualization capability that provides an animation of the scenario, showing operators interacting with the simulation. The output of the model run includes, in addition to the animation, data files with parameters of interest such as predicted operator situation awareness, workload, visual scanning, camera selection, and trajectory control.

In summary, the MIDAS-FAST project provided a validated model- and simulation based tool for predicting operator performance when working with a robotic arm in different function allocation situations. The function allocation model developed, and the empirical research conducted in this effort were used to identify conditions and provide data development of the tool.

MIDAS-FAST allows analysts (e.g., researchers, system developers, and concept developers) to enter data regarding the proposed robotic system, allocation of tasks, and the type of automation that is included. The tool uses a variety of sub-models, called modules, to evaluate particular aspects of operator performance (e.g., focus of visual attention, situation awareness, disorientation, and performance decrements due to control-response incompatibilities). The tool provides feedback on predicted operator performance (e.g., time to complete task; trajectory deviations), workload, situation awareness, visual scanning, and camera selection. This will help analysts evaluate and compare potential robotic systems in terms of their predicted effects on operator performance. Model predictions were evaluated and refined with data collected during two human in the loop studies.

Three human in the loop experimental studies, and one meta-analytic literature review conducted during this effort provide empirical data to extend the scientific research in human-automation interaction. All four studies have been submitted for publication in either the Human Factors Journal or presentation at the Human Factors and Ergonomics annual conference.

A primary goal of year 3 was to verify and validate our model of the robotic arm operator, to be employed in the function allocation tool, and to collect data that would further validate the Function Allocation Support Tool tradeoff model. To accomplish these purposes, data from the Human in the loop (HITL) Experiments 2 and 3 were analyzed, and both models developed and refined. The nature of Experiment 2, which was the most critical in both validations, will first be described in some detail.

HITL simulation: In Experiment 2, 36 participants were first trained extensively to manually operate a simulated version of the robotic arm manipulation task, in which a payload was first raised, then moved across a horizontal “table top”, and then lowered to a target destination. While maneuvering, participants needed to avoid proximity with hazards (wall and the table top) and joint states that would freeze the arm motion. In carrying out the task, they were assisted by three forms of automation. (1) All participants were alerted when proximity to a hazard was violated. (2) All participants were advised, on half the trials, as to the appropriate selection of two of the 4 possible camera views that would provide them with optimal viewing perspective. Finally, (3) participants were divided into three groups receiving different levels of trajectory control automation: none, presence of a 3D guidance path for the correct trajectory (autoguidance), and full autopilot control (autocontrol). In terms of the FAST tradeoff model, the two automation conditions varied in the stage of automation supported (early versus late).

Automation of all types functioned correctly through most of the trials. However on the final trial (last few minutes of the 6 hour experiment), both automation systems “failed.” Either the line directed the arm too close to a hazard (autoguidance), or the autopilot actually moved the arm into this close proximity with the hazard (autocontrol). In both instances, the collision warning system also failed.

Consistent with our FAST tradeoff model, increasing degrees of automation (from none, to autoguidance, to autocontrol) produced progressively better routine performance and lower workload. However, also consistent with the tradeoff model, late stage automation produced significantly worse performance in automation failure management on this final trial, and was associated with a significantly different visual scan path. The camera advisory automation was not failed. The advice of this automation was complied with, and did improve participant’s view of the workspace.

Validation of robotic arm operator model. We developed a computational model of the robotic arm operator on MIDAS. Because it was MIDAS controlling Boris, we called the model MORIS. MORIS consisted of three primary sub models: (1) A decision model, based on utility theory, chose the best camera views and decided which 4D trajectories to take (XYZ and speed) to reach the ultimate goal and avoid hazards. (2) A spatial cognition model called FORT (Frame of reference transformation) continuously calculated the challenges to spatial cognition caused by different levels of motion ambiguity (portrayed by camera views), control incompatibility (created by misalignment between control motion and perceived display motion), and by visibility challenges within the workplace. (3) a visual scanning model, across the workplace, known as SEEV, which controls simulated eye movement particularly on the basis of the bandwidth of information source changes, their value to the task, and their location in the workplace. MORIS then generated outputs of performance time, trajectory error, workload, camera selection, and scan-based situation awareness. Predictions were different across the three degrees of trajectory automation (none, autoguidance, autocontrol). MORIS model parameters were adjusted so that close fits were obtained between MORIS predictions and the empirical data from the HITL.

While the proceeding was essentially “parameterization” of the model, two efforts were made to make true validation: that is, predictions of the model in which the model parameters were not adjusted to maximize the fit with what was predicted. First, we predicted reasonably well, performance of participants in the one condition of experiment 3 that corresponded to one of our conditions in experiment 2. Second, and more significantly, we used MORIS to predict performance of participants in all three conditions in responding to the unexpected automation failures (see above). We did this by modeling, with MORIS, a reduction in the scanning of critical displays, thereby using SEEV to produce an automation complacency prediction. It was this prediction that was validated, with a high correlation between predicted and obtained failure management performance, with the actual performance of participants in the three conditions. In fact, as predicted by the FAST (Function Allocation Simulation Tool) tradeoff model, our complacency measure precisely predicted the poorer performance in the auto-control, compared to the auto-guidance automation.

The third experimental study evaluated context-sensitive function allocation. Specifically, four conditions were used to compare operator performance: (1) adaptable automation, where changes in the allocation of functions are initiated by the user, (2) adaptive automation, where the automation triggers changes based on operator or system performance, (3) a hybrid approach, where the system and operator collaborate on selecting and activating automation levels, and (4) a fixed automation approach, in which moderate degrees of automation were consistently applied.

Twelve participants (university students with 6-9 hours of previous robotics experience) worked with the BORIS simulation. They were instructed to perform their tasks (executing 3-segment trajectories) in a safe, accurate, and efficient manner. All subjects performed tasks in all 4 automation conditions (within-subjects design). Performance measures included trajectory completion time, trajectory deviation, subjective workload, and subjective preferences regarding automation type. Results indicate that trajectory deviations were significantly smaller in the adaptive, adaptable, and hybrid automation conditions compared with the fixed automation condition. Further, deviations in the hybrid automation condition were smaller than deviations in the adaptive and adaptable conditions. While the difference in time to complete the trajectory was not significant, the trend showed the similar results (slowest for fixed automation, fastest for hybrid). Workload was highest in the fixed condition, and lowest in both the adaptable and hybrid conditions. Finally, participants preferred the adaptable and hybrid automation types for the control it allowed them and – in the hybrid condition – the knowledge that the system was monitoring their performance.

In summary, Experiment 3 identified empirical support for the adaptable and hybrid automation schemes over fixed and adaptive schemes. The experiment also provided data (from the fixed and hybrid conditions) for validation of model predictions, described previously.

Based on the parameterization and validation efforts, the robotics operator model was updated. Further refinements in the model and simulation integration were implemented. New capabilities were added to the MIDAS-FAST tool, to allow researchers to examine a variety of potential conditions. Further, a user manual was developed and provided as part of the project deliverables.

During the third year of research, long-term hospitalization of two key personnel (different issues) resulted in a no-cost extension of the contract until April 30, 2013.

On June 5, 2013, the MIDAS-FAST tool was delivered to NASA Johnson Space Center (JSC) and a final briefing was delivered. This briefing included a summary of the project (purpose, goals, approach) with a focus on the experimental results and model parameterization and validation efforts (methods, results). The final report and user manual were also delivered to NASA.

NOTE: End date changed to 12/31/2012 per NSSC information (Ed., 6/1/2012)

In this project, the team will develop a research-based module of human-automation interaction. The team will develop human performance models of scenarios of interest. These models will be based on task analyses performed in cooperation with subject matter experts (SMEs). Various validation studies will be performed throughout this project. The team will validate the task analyses by talk-through sessions with SMEs. Human performance model and human-automation interaction module predictions will be validated in empirical, human-in-the-loop studies. Results of the validations will be used to refine the models.

One particular focus of the project is on developing a prototype tool that is both usable and useful for researchers, allowing them to easily modify scenarios and evaluate different potential automation conditions. This tool will provide for data entry screens that guide the user through the process of building a scenario. It will allow the researchers to specify numerous relevant factors, e.g., operators, tasks, environmental conditions, and function allocation strategy. It will offer a visualization capability that provides a virtual video (or animation) of the scenario, showing operators interacting with equipment. The output of the model run will include, in addition to the video file, parameters of interest such as situation awareness, workload, time to initiate tasks, time to complete tasks, and task accuracy.

MIDAS-FAST will allow analysts (e.g., researchers, system developers, and concept developers) to enter data regarding the proposed robotic system, allocation of tasks, and the potential for automation failures. The tool will use a variety of sub-models, called modules, to evaluate particular aspects of operator performance (e.g., focus of visual attention, situation awareness, disorientation and performance decrements due to control-response incompatibilities). The tool will then provide feedback on predicted operator performance (e.g., time to complete task; error such as reversals, collisions, rule violations), workload, situation awareness. This will help analysts evaluate and compare potential robotic systems in terms of their predicted effects on operator performance. Model predictions will be evaluated and refined with data collected during two human in the loop studies.

In Year 3 of the MIDAS-FAST effort, the operator models have been refined to address more detailed aspects of performance, tool development has progressed, product demonstrations and meetings were held with the project stakeholders at NASA Johnson Space Center (JSC) and Ames Research Center (ARC), the first empirical study was conducted, and validation plans have been refined.

The operator performance model includes aspects such as visual scanning, building and maintaining awareness of key task parameters, and executing robotic tasks in the Basic Operational Robotics Instructional System (BORIS) simulation. In Year 3 the models have been refined to include operator errors and corrections during trajectory control operations. In addition, the visual scanning model has been modified to address the scanning patterns typical of robotics missions. Robotics operators devote more attention to camera and window views of the robotic arm, rather than to other displays that provide additional information. Experienced operators employ a pattern of frequent, short scans to the supplementary information, where novices are more likely to fixate on camera and window view displays. In Year 3, the detailed models for operator situation awareness (SA) were also developed. Key parameters of importance for robotics missions were identified (e.g., hazard awareness, trajectory awareness, rate awareness, camera awareness), and the specific visual displays that provide data to support those types of SA were identified. These have now been incorporated into the model of operator performance.

The MIDAS-FAST tool has been further developed, and it now provides an animation capability to show predicted operator performance during a model run. Because BORIS provides limited automation capabilities, the team developed and implemented a guided trajectory condition (where the automation shows a recommended "flight path") and an automated condition (different from the BORIS automated condition, that flies an arc trajectory, the MIDAS-FAST modification flies a 3-segment trajectory). Further, hazard alerting and hazard avoidance automation was included, to help prevent collisions and singularities. Finally, the team developed and implemented logic to provide camera control recommendations. The MIDAS-FAST interface has been modified to address the latest automation capabilities.

The MIDAS-FAST team members met with NASA project stakeholders from NASA JSC and ARC. During these meetings we provided video demonstrations of the MIDAS-FAST tool, and gathered feedback on the usefulness of the tool. In one meeting, we attempted to conduct a "live" demo, using a version of BORIS installed at NASA JSC. Due to slight differences between that version of BORIS and the one used by the project team, we were unable to run the demo. We have since (with NASA contracts approval) purchased a dedicated laptop, and installed BORIS on it. This will be used for future demos, and provided to NASA at the end of the project.

The first empirical study has been conducted at the University of Michigan. In this experiment, 36 participants performed 6 or 7 trajectory flight tasks with the BORIS simulation. Participants worked in either a manual, guided, or fully-automated condition, and they experienced either hazard alerting or hazard avoidance automation. In addition, some scenarios included camera recommendations, whereas others provided no camera recommendation. The final two scenarios included automation failures. Extensive data were collected to characterize operator performance in the different conditions. These included time to complete trajectory segments, errors made (collision alerts, singularity alerts), actual versus recommended trajectory, participant eye movements and fixations, camera selections, self-reported workload, as well as subjective preferences and comments regarding the automation. The data are currently being analyzed. A second experimental study is currently being planned.

Finally, the team has updated the model validation plan to address the latest model improvements. This was used to help plan the experimental data collection. Using the validation plan, data gathered during model runs will be compared with empirical data to identify if the model accurately predicts operator performance. Discrepancies will be identified, investigated, and improved. These improvements will then be re-evaluated in the upcoming second experiment.

A no-cost-extension has been requested and approved. The project is now scheduled to complete on December 31, 2012.

In this project, the team will develop a research-based module of human-automation interaction. The team will develop human performance models of scenarios of interest. These models will be based on task analyses performed in cooperation with subject matter experts (SMEs). Various validation studies will be performed throughout this project. The team will validate the task analyses by talk-through sessions with SMEs. Human performance model and human-automation interaction module predictions will be validated in empirical, human-in-the-loop studies. Results of the validations will be used to refine the models.

One particular focus of the project is on developing a prototype tool that is both usable and useful for researchers, allowing them to easily modify scenarios and evaluate different potential automation conditions. This tool will provide for data entry screens that guide the user through the process of building a scenario. It will allow the researchers to specify numerous relevant factors, e.g., operators, tasks, environmental conditions, and function allocation strategy. It will offer a visualization capability that provides a virtual video of the scenario, showing operators interacting with equipment and each other. The output of the model run will include, in addition to the video file, parameters of interest such as situation awareness, workload, time to initiate tasks, time to complete tasks, and task accuracy.

MIDAS-FAST will allow analysts (e.g., researchers, system developers, and concept developers) to enter data regarding the proposed robotic system, allocation of tasks, and the potential for automation failures. The tool will use a variety of sub-models, called modules, to evaluate particular aspects of operator performance (e.g., focus of visual attention, situation awareness, disorientation and performance decrements due to control-response incompatibilities). The tool will then provide feedback on predicted operator performance (e.g., time to complete task; error such as reversals, collisions, rule violations), workload, situation awareness. This will help analysts evaluate potential robotic systems in terms of their predicted effects on operator performance. Model predictions will be evaluated and refined with data collected during two human in the loop studies.

The main overall objective of the proposed research is to develop tools and empirically-based guidelines that support designers in developing new technologies. Specifically, the products from this research will help designers and mission planners (a) anticipate and avoid potential problems in function allocation strategies in system design before new systems are introduced, and therefore (b) assure that these systems and their function allocation strategies can be implemented seamlessly and in a way to minimize transient or longer-term impacts on performance in space exploration missions. The proposed work contributes to the Program Requirements Document (PRD) Risk Associated with Poor Task Design (20.0 – D X I), and specifically addresses Integrated Research Plan (IRP) Gap Space Human Factors Engineering SHFE4: Guidelines are needed for appropriate task automation as well as for effective allocations of tasks between humans and automation to increase performance, efficiency, and safety.

To help NASA achieve these objectives, Alion Science and Technology, together with Dr. Christopher Wickens, the University of Michigan, the San Jose State University Research Foundation, and Dr. Thomas Jones, proposed to develop and empirically validate the MIDAS-FAST simulation tool. MIDAS-FAST is based on human-performance models, together with a robotic simulation environment, to allow system designers and concept developers evaluate the effects of function allocation strategies, varying types of automation (e.g., fixed, adaptive, and adaptable automation), and automation reliability on operator and system performance. In developing this tool, our plan of work consists of six key tasks, identified below. The following paragraphs identify the task, describe the progress to date, and briefly outline the plan for further research.

1.1 Identify an appropriate domain and simulation environment

Progress: This task is complete. The team, with input from NASA personnel, has identified the space robotics domain. We have obtained from NASA the necessary set of software tools to simulate robotics tasks, in particular, the Basic Operational Robotics Instructional System (BORIS). We have installed these tools at Alion and University of Michigan. In the summer of 2010, five team members attended the NASA General Robotics Training (GRT) at NASA Johnson Space Center (JSC). We are currently working with the BORIS tools. The BORIS simulation at Alion has been integrated with the MIDAS human performance modeling environment and other performance modules, as described in other tasks below. The simulation at the University of Michigan is being customized to provide the task environment for the human in the loop simulation studies.

1.2 Conduct a literature review of human automation interaction

Progress: This task is complete. An extensive literature review has been conducted: 1) to assess the effects of automation at varying stages (i.e., information acquisition, information analysis and integration, choosing and deciding, and executing an action) and levels (i.e., high, moderate and low degrees of automation) on operator performance, 2) to identify the effects of unreliable automation on performance, and 3) to evaluate context sensitive (adaptive and adaptable) automation. The “stages and levels” and “reliability” aspects of the literature review have been further refined to identify and extract the data that are relevant for module development. Further, the effects of context-sensitive automation have been briefly summarized.

1.3 Develop and validate modules of human performance

Progress: This task is currently ongoing. We have identified a number of relevant modules to include in MIDAS-FAST. Modules are reusable computational models that predict specific aspects of human performance. These interact with the MIDAS human performance model to predict operator behavior and outcomes for the user-specified conditions. We leverage existing models developed under other NASA efforts. These include the Salience, Expectancy, Effort, and Value (SEEV) model of visual attention, and the Frame of Reference Transformation (FORT) model of human error due to control action – robotic arm movement incompatibility in robotic tasks. These are being customized for use in MIDAS-FAST. Other modules under development include operator performance in different stages and levels of automation, in varying conditions of automation reliability, and in different conditions of context-sensitive automation.

1.4 Build and verify human performance models

Progress: This task is currently ongoing. The team is evaluating the robotic task domain, and implementing modules to simulate operator performance. MIDAS modeling efforts will focus on progressively more detailed issues as the team identifies relevant scenarios and automation effects. Models address operator performance in a variety of conditions. Specifically they focus on decision making, trajectory control, camera selection, rate of robotic arm movement, and operator visual attention.

1.5 Plan, conduct, and evaluate empirical studies

Progress: This task is currently ongoing. During the past year, the UM team developed and set up the apparatus and procedures (including hardware and software) that will be used for training participants and for conducting the planned experiments. This includes the BORIS simulation with associated data collection routines and equipment, eye-tracking data analysis software, the two hand controllers, training tools (e.g., a model of the robotic arm, coordinate frames), and, importantly, the design and implementation of automation functions (including modifications of the original BORIS GUI). The team developed a training plan and tutorials. Also, the protocol for a pilot study and the first experiment has been submitted to, and approved by, the UM Institutional Review Board (IRB). The pilot study was recently completed. Its purpose was to compare the effectiveness of different notifications and alerts in case of undesirable arm configurations. Its findings also helped refine the experiment setup and training plan, and they informed the design of automation schemes and scenarios for the two subsequent experiments. Experiment 1 which focuses on a comparison of performance effects of fixed automation schemes will be conducted during Summer/Fall 2011. Subsequently, and based in part on findings from experiment 1, experiment 2 will examine possible benefits and limitations of employing context-sensitive (adaptable and adaptive) automation schemes.

1.6 Integrate the software tools to develop MIDAS-FAST

Progress: This task is currently ongoing. During this year the team has effectively integrated the BORIS simulation environment and MIDAS modeling tool, where the MIDAS (or MORRIS) simulated operator control actions drive the robotic arm, and the feedback from the robotic environment affects simulated operator attention and decision making. Further work will include identifying the simulation data to collect and developing the capabilities to store this data and present it in a readable, usable format.

One challenge this year has been identifying and developing workaround solutions to limitations in the BORIS environment. BORIS provides an excellent robotic simulation, but it offers limited automation conditions. Because our modeling and research efforts focus specifically on the effects of varying automation strategies and failures on operator and system performance, we need to be able to simulate different automation situations. This year we have identified and developed implementation strategies for trajectory control automation, hazard alerting and avoidance, camera control recommendations, and rate control.

Results of the validations will be used to refine the models. One particular focus of the project is on developing a prototype tool that is both usable and useful for researchers, allowing them to easily modify scenarios and evaluate different potential automation conditions. This tool will provide for data entry screens that guide the user through the process of building a scenario. It will allow the researchers to specify numerous relevant factors, e.g., operators, tasks, environmental conditions, and function allocation strategy. It will offer a visualization capability that provides a virtual video of the scenario, showing operators interacting with equipment and each other. The output of the model run will include, in addition to the video file, parameters of interest such as situation awareness, workload, time to initiate tasks, time to complete tasks, and task accuracy.

MIDAS-FAST will allow analysts (e.g., researchers, system developers, and concept developers) to enter data regarding the proposed robotic system, allocation of tasks, and the potential for automation failures. The tool will use a variety of sub-models, called modules, to evaluate particular aspects of operator performance (e.g., focus of visual attention, situation awareness, disorientation and performance decrements due to control-response incompatibilities). The tool will then provide feedback on predicted operator performance (e.g., time to complete task; error such as reversals, collisions, rule violations), workload, situation awareness. This will help analysts evaluate potential robotic systems in terms of their predicted effects on operator performance. Model predictions will be evaluated and refined with data collected during two human in the loop studies.

To help NASA achieve these objectives, Alion Science and Technology, together with Dr. Christopher Wickens, the University of Michigan, the San Jose State University Research Foundation, and Dr. Thomas Jones, proposed to develop and empirically validate the MIDAS-FAST simulation tool. MIDAS-FAST is based on human-performance models, together with a simulation environment, to allow system designers and concept developers evaluate the effects of function allocation strategies, varying types of automation (e.g., fixed, adaptive, and adaptable automation), and automation reliability on operator and system performance. In developing this tool, our plan of work consists of six key tasks, identified below. The following paragraphs identify the task, describe the progress to date, and (where appropriate) briefly outline the plan for further research.

1.1 Identify an appropriate domain and simulation environment

Progress: This task is complete. The team, with input from NASA personnel, has identified the space robotics domain. We have obtained from NASA the necessary set of software tools to simulate robotics tasks, in particular, the Basic Operational Robotics Instructional System (BORIS). We have installed these tools at Alion and University of Michigan. We are currently familiarizing ourselves with their capabilities by working with the tools and attending the NASA General Robotics Training (GRT). The BORIS simulation at Alion will be integrated with the MIDAS human performance model and other performance modules, as described in other tasks below. The simulation at the University of Michigan will provide the task environment for the human in the loop simulation studies.

1.2 Conduct a literature review of human automation interaction Progress: This task is currently ongoing. An extensive literature review has been conducted: 1) to assess the effects of automation at varying stages (i.e., information acquisition, information analysis and integration, choosing and deciding, and executing an action) and levels (i.e., high, moderate and low degrees of automation) on operator performance, 2) to identify the effects of unreliable automation on performance, and 3) to evaluate context sensitive (adaptive and adaptable) automation. The “stages and levels” and “reliability” aspects of the literature review have been further refined to identify and extract the data that are relevant for module development. For next year, the effects of context-sensitive automation will be evaluated for a possible meta analysis, and the literature review will be summarized as a report. Relevant guidelines will be extracted and provided as a document.

1.3 Develop and validate modules of human performance

Progress: This task is currently ongoing. We have identified a number of relevant modules to include in MIDAS-FAST. Modules are reusable computational models that predict specific aspects of human performance. These will interact with the MIDAS human performance model to predict operator behavior and outcomes for the user-specified conditions. We will, when possible, leverage existing models developed under other NASA efforts. These include the Salience, Expectancy, Effort, and Value (SEEV) model of visual attention, and the Frame of Reference Transformation (FORT) model of human error due to control action – robotic arm movement incompatibility in robotic tasks. These are being evaluated to customize them for MIDAS-FAST. Other modules under development include operator performance in different stages and levels of automation, in varying conditions of automation reliability, and in different conditions of context-sensitive automation.

1.4 Build and verify human performance models

Progress: This task is currently ongoing. The team is evaluating the robotic task domain, and identifying how this differs from aviation. MIDAS has been developed for aviation, so it will need to be updated to accommodate the differences between domains that affect operator cognition and performance. For example, aviation MIDAS models pilots scanning discrete displays, and performing aviation, navigation, communication, and systems monitoring tasks. In robotics applications, MIDAS will model operators planning tasks, configuring displays, initiating movements, monitoring progress, and scanning displays. MIDAS modeling efforts will focus on progressively more detailed issues as the team identifies relevant scenarios and automation effects.

1.5 Plan, conduct, and evaluate empirical studies Progress: This task is currently ongoing. Both Alion and the University of Michigan (UM) team members are attending the NASA GRT course this summer to become knowledgeable about the domain and proficient in the domain tasks. Once the UM team members have attended this training (late July), conducted additional interviews with instructors and astronauts, and completed cognitive task analyses of robotic missions, we will begin development of scenarios for our two planned simulation studies and model evaluations. In preparation for those studies, we have also been familiarizing ourselves with the capabilities of the BORIS simulation tool. We have developed a stages and levels table to identify relevant automation conditions in the robotics domain, and we have identified a number of potential automation situations and possible failures.

Based on observations made by Alion team members during their recent training at NASA, we have reconsidered the requirements for participants for both simulation studies. The robotics tasks are not simple and can not adequately be taught in a simple 2-4 hour session. Therefore, we will need to identify and ensure access to appropriate personnel for even the initial experiment. Students in fields such as robotics and aeronautics will likely be asked to participate. For the final experiment, we are currently investigating the possibilities (advantages, disadvantages, availability) of having astronauts, general robotic training instructors, or other skilled personnel serve as participants. One continual challenge with this project is keeping the focus on issues that are relevant in actual robotic applications, and not simply BORIS-specific concerns.

1.6 Integrate the software tools to develop MIDAS-FAST Progress: This task is currently ongoing. As part of setting up BORIS and the related software (TRICK, the simulation environment, and EDGE, the graphic visualization), we have been familiarizing ourselves with the structure of these tools. We have identified how to collect data regarding operator rate selection, joystick movements, and the grapple trigger. We have also verified that it is possible to use a Microsoft Windows C# program to implement control actions in BORIS (running on Linux). This demonstration is the first step in connecting the MIDAS model (written in C#) to BORIS. Further work will include identifying the simulation data to collect, developing the capabilities to store this data, and integrating MIDAS with BORIS.

Results of the validations will be used to refine the models. One particular focus of the project is on developing a prototype tool that is both usable and useful for researchers, allowing them to easily modify scenarios and evaluate different potential automation conditions. This tool will provide for data entry screens that guide the user through the process of building a scenario. It will allow the researchers to specify numerous relevant factors, e.g., operators, tasks, environmental conditions, and function allocation strategy. It will offer a visualization capability that provides a virtual video of the scenario, showing operators interacting with equipment and each other. The output of the model run will include, in addition to the video file, parameters of interest such as situation awareness, workload, time to initiate tasks, time to complete tasks, and task accuracy.