NOTE: End date change to 4/30/2018 per NSSC information (Ed., 8/9/17)

NOTE: Element change to Human Factors & Behavioral Performance; previously Space Human Factors & Habitability (Ed., 1/19/17)

Specific aims of this proposal are threefold: (1) develop a framework for human-systems integration requirements, (2) identify and develop a metrics criteria in which safety and efficiency can be characterized in human-automation teams, and (3) design, develop, and validate a theoretically-driven, empirically-based metrics toolkit that characterizes the safety and efficiency of human automation interactions. This proposal meets NASA goals and objectives by mitigating the risk of inadequate design of human and automation/robotic integration through the development of safety and efficiency metrics for human-automation systems. The proposal is divided into three primary phases. Phase 1 will consist of synthesizing and translating findings from the extant literature relevant to human automation/robotic integration. The result of this effort will be the development of objective metrics generalizable to individual and team levels that characterize the safety and efficiency of a human automation interaction. The final outcome of Phase 1 will be the development of a human automation interaction metrics (HAIM) toolkit. Phase 2 will involve in-depth preparation for scientifically sound experiments. Phase 2 ensures adequate time and methodology for meaningful outcomes for Phase 3. The central outcome of Phase 2 will be the final development of the experimental testbed and experimental protocol. Phase 3 will involve preparation for, and execution of, experiments. This will include the design and execution of a set of multi-level empirical studies aimed at validating the metrics toolkit. The validation studies will focus on testing different aspects of human automation interaction (e.g., levels of automation, task complexity, and the number and configuration of system operators). The outcome of the proposed effort will provide NASA a set of evidence-based, empirically-validated guidelines and a measurement toolkit for mitigating the risk of inadequate design of human and automation/robotic integration as it pertains to the development of safety and efficiency metrics for human automation systems.

Phase 2: The original goal for this second phase was to complete in-depth preparation for scientifically sound experiments. This detailed preparation time in phase 2 was meant to ensure adequate time and methodology for meaningful outcomes for Phase 3. The intended outcome of Phase 2 was be the final development of the experimental testbed and experimental protocol. However, as more was learned throughout phase 1 about Human-Automation measurement and as the toolkit development was progressing, it was determined that Phase 2 should be adjusted to better meet the goals of the overall project. Therefore, Phase 2 was adjusted to be preparation for Subject Matter Expert (SME) interviews and iterative usability evaluations of the HAIM Toolkit instead of validation experiments to test the theoretical framework. This proved to be a very fruitful decision as the SME interviews provided valuable feedback and data that was able to be incorporated into the final design of the toolkit.

Phase 3: Preparation for and execution of, experiments. This final phase of the project was originally planned to include the design and execution of a set of multi-level empirical studies aimed at validating the theoretical framework and metrics toolkit. The focus of the validation studies was to be on testing different aspects of human automation interaction (e.g., levels of automation, task complexity, and the number and configuration of system operators). The intended outcome of the proposed effort was to provide NASA a set of evidence-based, empirically-validated guidelines and a measurement toolkit for mitigating the risk of inadequate design of human and automation/robotic integration as it pertains to the development of safety and efficiency metrics for human automation systems. Phase 3 was adjusted to focus on validation of the automated HAIM Toolkit through execution of usability evaluations by potential end users as this was deemed more valuable than the originally planned validation studies. The usability studies that were conducted were designed to evaluate both usability of the automated HAIM Toolkit as well as the value of the information that the toolkit provided. Multiple rounds of testing took place along with subsequent periods or toolkit redesign to incorporate feedback gleaned during the usability testing. The final outcome of this effort along with the research conducted in Phase 1 is a set of principles, guidelines, and tips for measurement that have been submitted to NASA in white papers and are in the process of being transformed into multiple manuscripts currently under preparation. Additionally, the final outcome of this phase is the development of the HAIM Toolkit, which is complete and will be delivered to NASA with the final report.

Manuscripts in various stages:

Hughes, A.M., Marlow, S.L., Hancock, G.M., Oglesby, J.M., Stowers, K. & Salas, E. (Under Revision). Physiological assessment of workload: A meta-analysis. Human Factors

Stowers, K., Iwig, C., Salas, E. (In preparation). Measurement in the Second Machine Age: Considerations for Future Human-Machine Systems. To be submitted to Human Factors.

Stowers, K., Sonesh, S., Iwig, C., Salas, E. (In preparation). Technology and Teams: The Impact of Technological Evolution on Team Performance. To be submitted to Group and Organization Management.

Stowers, K., Sonesh, S.C., Iwig, C., Salas, E. (In preparation.) Capturing Human Machine System Safety in Spaceflight. To be submitted to Aerospace Medicine and Human Performance.

NOTE: End date change to 4/30/2018 per NSSC information (Ed., 8/9/17)

NOTE: Element change to Human Factors & Behavioral Performance; previously Space Human Factors & Habitability (Ed., 1/19/17)

Specific aims of this proposal are threefold: (1) develop a framework for human-systems integration requirements, (2) identify and develop a metrics criteria in which safety and efficiency can be characterized in human-automation teams, and (3) design, develop, and validate a theoretically-driven, empirically-based metrics toolkit that characterizes the safety and efficiency of human automation interactions. This proposal meets NASA goals and objectives by mitigating the risk of inadequate design of human and automation/robotic integration through the development of safety and efficiency metrics for human-automation systems. The proposal is divided into three primary phases. Phase 1 will consist of synthesizing and translating findings from the extant literature relevant to human automation/robotic integration. The result of this effort will be the development of objective metrics generalizable to individual and team levels that characterize the safety and efficiency of a human automation interaction. The final outcome of Phase 1 will be the development of a human automation interaction metrics (HAIM) toolkit. Phase 2 will involve in-depth preparation for scientifically sound experiments. Phase 2 ensures adequate time and methodology for meaningful outcomes for Phase 3. The central outcome of Phase 2 will be the final development of the experimental testbed and experimental protocol. Phase 3 will involve preparation for, and execution of, experiments. This will include the design and execution of a set of multi-level empirical studies aimed at validating the metrics toolkit. The validation studies will focus on testing different aspects of human automation interaction (e.g., levels of automation, task complexity, and the number and configuration of system operators). The outcome of the proposed effort will provide NASA a set of evidence-based, empirically-validated guidelines and a measurement toolkit for mitigating the risk of inadequate design of human and automation/robotic integration as it pertains to the development of safety and efficiency metrics for human automation systems.

Development of HASMAT has followed an iterative design and development process including multiple phases of prototype design, testing, and revision. Major accomplishments during this year have included (1) transitioning the toolkit decision tree architecture, which was developed in the previous year, into the design of the first HASMAT prototype, (2) testing the prototype using validated usability evaluation techniques, (3) developing a second HASMAT prototype based on results from initial evaluations, (4) testing of the second prototype in a structured laboratory based usability study utilizing NASA SMEs (subject matter experts), (5) development of a vast database of measurement information, guidelines, and tips to incorporate into the toolkit (in-progress), and (6) development of the final, fully automated, HASMAT prototype (in-progress).

Research efforts this year began with the refinement of the toolkit decision tree architecture, which was developed during the previous year. Once refinement of this architecture was complete, it was used as a basis of the first toolkit prototype. This initial prototype was created using Axure prototyping software. The main goal for this initial prototype was to incorporate all decision tree questions, determine the most appropriate visual delivery method for the questions, and develop and ideal organization and flow of the questions within the decision tree. Once this initial prototype was complete, initial evaluation techniques were conducted by the Rice research team to evaluate all aspects of the initial prototype's design, flow, and support features. The results of these evaluations informed revisions to the toolkit.

The second HASMAT prototype incorporated all recommendations for improvement that resulted from evaluations of the initial prototype. The purpose of this prototype was to provide a semi-functional prototype that could be tested with potential end-users. This prototype was still lacking automated measure selection features, but the decision tree architecture had been refined in this prototype and other features were added. Therefore, usability testing of this prototype with NASA SMEs focused on obtaining input as to how the decision tree architecture could be improved as well as the content that had already been incorporated into this version of HASMAT.

Results from the structured usability studies aimed at evaluating the second HASMAT prototype were very enlightening and led to an extensive overhaul of the entire toolkit design. Additionally, this extensive update of HASMAT, which is currently in progress, is also aimed at adding in full functionality including automation of the dynamic measure selection feature that allows the toolkit to provide customized measurement recommendations based on system and contextual information provided by the toolkit’s users during their completion of the decision tree questions. Once complete, this final HASMAT prototype will be tested in a final round of usability testing using NASA SMEs, followed by completion of necessary revisions, and resulting in delivery of the final HASMAT prototype to NASA. Overall, major progress on the development of the final HASMAT prototype has been made in year 4 and will continue to be made through the end of July.

Specific aims of this proposal are threefold: (1) develop a framework for human-systems integration requirements, (2) identify and develop a metrics criteria in which safety and efficiency can be characterized in human-automation teams, and (3) design, develop, and validate a theoretically-driven, empirically-based metrics toolkit that characterizes the safety and efficiency of human automation interactions. This proposal meets NASA goals and objectives by mitigating the risk of inadequate design of human and automation/robotic integration through the development of safety and efficiency metrics for human-automation systems. The proposal is divided into three primary phases. Phase 1 will consist of synthesizing and translating findings from the extant literature relevant to human automation/robotic integration. The result of this effort will be the development of objective metrics generalizable to individual and team levels that characterize the safety and efficiency of a human automation interaction. The final outcome of Phase 1 will be the development of a human automation interaction metrics (HAIM) toolkit. Phase 2 will involve in-depth preparation for scientifically sound experiments. Phase 2 ensures adequate time and methodology for meaningful outcomes for Phase 3. The central outcome of Phase 2 will be the final development of the experimental testbed and experimental protocol. Phase 3 will involve preparation for, and execution of, experiments. This will include the design and execution of a set of multi-level empirical studies aimed at validating the metrics toolkit. The validation studies will focus on testing different aspects of human automation interaction (e.g., levels of automation, task complexity, and the number and configuration of system operators). The outcome of the proposed effort will provide NASA a set of evidence-based, empirically-validated guidelines and a measurement toolkit for mitigating the risk of inadequate design of human and automation/robotic integration as it pertains to the development of safety and efficiency metrics for human automation systems.

The third year of the project focused on completion of the subject matter expert (SME) interviews in order to determine best practices for measurement of human-automation systems within the specific context of spaceflight. The information gained during these interviews served as a supplement to the data gained through the qualitative literature review and during the 2014 HFES (Human Factors and Ergonomics Society) Measurement Workshop by providing a greater understanding of the specific context of interest -- spaceflight. Additionally, during Year 3, greater focus was placed on actual development of the toolkit as a web-based decision tree tool for guiding users in selecting metrics and applying appropriate measurement techniques. Currently, work is continuing on the development of the metrics toolkit prototype and preparation for the usability studies have begun. Usability studies will be conducted through the summer and will be followed by completion of final revisions to the toolkit.

Specific aims of this proposal are threefold: (1) develop a framework for human-systems integration requirements, (2) identify and develop a metrics criteria in which safety and efficiency can be characterized in human-automation teams, and (3) design, develop, and validate a theoretically-driven, empirically-based metrics toolkit that characterizes the safety and efficiency of human automation interactions. This proposal meets NASA goals and objectives by mitigating the risk of inadequate design of human and automation/robotic integration through the development of safety and efficiency metrics for human-automation systems. The proposal is divided into three primary phases. Phase 1 will consist of synthesizing and translating findings from the extant literature relevant to human automation/robotic integration. The result of this effort will be the development of objective metrics generalizable to individual and team levels that characterize the safety and efficiency of a human automation interaction. The final outcome of Phase 1 will be the development of a human automation interaction metrics (HAIM) toolkit. Phase 2 will involve in-depth preparation for scientifically sound experiments. Phase 2 ensures adequate time and methodology for meaningful outcomes for Phase 3. The central outcome of Phase 2 will be the final development of the experimental testbed and experimental protocol. Phase 3 will involve preparation for, and execution of, experiments. This will include the design and execution of a set of multi-level empirical studies aimed at validating the metrics toolkit. The validation studies will focus on testing different aspects of human automation interaction (e.g., levels of automation, task complexity, and the number and configuration of system operators). The outcome of the proposed effort will provide NASA a set of evidence-based, empirically-validated guidelines and a measurement toolkit for mitigating the risk of inadequate design of human and automation/robotic integration as it pertains to the development of safety and efficiency metrics for human automation systems.