NOTE: End date changed to 9/30/2014 (from 7/31/2016) per M. Whitmore/JSC (Ed., 3/24/14)

The research has an analytic and empirical strand. The purpose of the analytic strand is to provide methods and tools for measuring automation-to-work (ATW) alignment, for use guiding development and evaluation of HARI designs. The purpose of the empirical strand is to assess whether measured ATW alignment of HARI designs predicts the learnability of those designs, an important aspect of robustness. By ATW alignment we mean the correspondence between the elements and structure of interaction with the elements and structure of the work. Needs analysis identifies elements and structure of the work. In the analytic strand we will develop a scoring method for measuring alignment. This approach draws on and integrates a wide set of observations and proposals in HARI, Human Computer Interaction (HCI), Work Domain Analysis (WDA), and related disciplines. In the empirical strand we test the prediction that HARI designs that align with work more strongly will be easier to learn, particularly, easier to master using the automation for novel problem solving. We test this hypothesis by identifying and measuring designs that differ in ATW alignment and then comparing the designs with high versus low scores for how easily they are learned and how flexibly they can be used. The development of methods, tools, and data is intended to guide design and evaluation of HARI systems to ensure that such systems are fit-for-purpose, that is, it solves the correct problem. The research approach is based on identifying work needs in a manner that can guide design and evaluation, and consists of the following inter-related strands:

• Representation & Analysis Method. The team developed a method that represents a work domain in terms of the required functions within that domain, and which enables evaluating technology with respect to how well and in what respects the technology supports that body of work.

• Case Study. The team developed the method in concert with applying the analysis to a safety critical, highly automated work domain.

• Tool Prototypes. The team developed initial prototype tools to aid application of the method.

• Test environment. The team developed a medium fidelity test environment that was capable of performing the empirical assessment. The test environment (simulator is rapidly changeable and system design may be guided by those user preferences that are excessively shaped by familiarity and historical methods for accomplishing work rather than rigorous analysis of the current work needs).

• Preliminary Empirical Assessment: The team conducted an empirical assessment, which produced supporting though preliminary data supporting our approach.

The Needs Analysis method represents work as a matrix, which identifies the work functions in the domain of interest and the variables that affect or are affected by those work functions. The work functions in a work domain specify a high-level task or goal to be accomplished, in terms of the variables that are needed as input (such as information or resources) and the variables, which are affected as output of the function. For example, checking a proposed route might be a work function in supervisory control of a robot; input variables might include the proposed route, data about characteristics of the route such as terrain and distance, characteristics about the robot such as its loads and power, and context information such as other tasks slated for the robot; output variables might include modifications to the route, setting time and approval for the robot to begin the trip, and setting check points for robot to verify continuing or human to monitor robot performance.

It is expected that for many complex domains there will be clusters of variables that support a set of work functions and conversely clusters of work functions related because they draw on a related set of variables. Input and output variables provide a common language that can be used to represent device affordances as well as the work domain. Technology components or devices can also be represented in terms of these same variables: the display variables provide input to the human (input needed by the work function) and the control variables enable action by the human (output the result of the work function). Because both work and technology components can be represented in the “common language” of these variables, this enables evaluating alternative technology designs or implemented systems with respect to how well it supports the work functions in the target work domain.

As the team developed the method and applied it to the case study, the simple template work matrices were built in a spreadsheet. However, the process was not well supported by using a spreadsheet. As a result, the team developed a tool using an SQL database with a custom browser interface, which was “above and beyond” the year one deliverable specified in the research plan. The tool is designed to enable a domain expert to enter the work functions, the work variables, and their mutual relevance more easily. A second tool “above and beyond” development project concerned identifying the clustering structure implicit in a work matrix to make that structure more systematically identified and more directly presented to a user. The team also extended the bi-clustering statistical methods developed in genomics and also recruited and extended visualization methods developed there. These above and beyond tool development projects were “bonuses,” drew heavily on contributed time, and greatly benefited from collaborators, and have not been validated at this date.

The investigation team predicted that technology that is better aligned with the work domain will provide performance benefits. For example, technology that is more closely aligned with work should show better performance (e.g., easier learning), than technology that is not well aligned with work. The team tested this hypothesis in a preliminary experiment comparing learning to use two alternative designs of MCP, by pilots who were knowledgeable about much of the work domain and selected to be unfamiliar with either design. Due to a reduction in budget, the team was only able to run six participants; however, the data from these users did seem to confirm the hypothesis by showing faster learning for the better aligned MCP, and better transfer to similar but new situations.

Assessing alignment of technology with the work it is intended to support, or its fitness-for-purpose is critical to ensure sound HARI (Risk of Inadequate Design of Human and Automation/Robotic Integration) design. The method developed in this task for representations in work matrices and device matrices, using a “common language” of input and output variables has promise. The approach can be applied early in the design process, e.g., in requirement specification, and carried through to evaluation. The preliminary results provide suggestive empirical support.

NOTE: End date changed to 9/30/2014 (from 7/31/2016) per M. Whitmore/JSC (Ed., 3/24/14)

The proposed research has an analytic and empirical strand. The purpose of the analytic strand is to provide methods and tools for measuring automation-to-work (ATW) alignment, for use guiding development and evaluation of HAI designs. The purpose of the empirical strand is to assess whether measured ATW alignment of HAI designs predicts the learnability of those designs, an important aspect of robustness. By ATW alignment we mean the correspondence between the elements and structure of interaction with the elements and structure of the work. Needs analysis identifies elements and structure of the work. In the analytic strand we will develop a scoring method for measuring alignment. Our approach draws on and integrates a wide set of observations and proposals in HAI, Human Computer Interaction (HCI), Work Domain Analysis (WDA), and related disciplines. In the empirical strand we test the prediction that HAI designs that align with work more strongly will be easier to learn, particularly, easier to master using the automation for novel problem solving. We test this hypothesis by identifying and measuring designs that differ in ATW alignment and then comparing the designs with high versus low scores for how easily they are learned, how well retained, and how flexibly they can be used.

The process of developing the matrix has produced a need for database and statistical analysis development to deal with the complexity of mapping description of work activities to descriptions of the behavior of specific technologies for completing the work. In particular the team has worked on the development of bi-clustering methods that allow sorting and grouping of related work functions and technologies. The team is currently preparing an empirical study to test the ability to use the method to assess alignment between two proposed automation interfaces.

The proposed research has an analytic and empirical strand. The purpose of the analytic strand is to provide methods and tools for measuring automation-to-work (ATW) alignment, for use guiding development and evaluation of HAI designs. The purpose of the empirical strand is to assess whether measured ATW alignment of HAI designs predicts the learnability of those designs, an important aspect of robustness. By ATW alignment we mean the correspondence between the elements and structure of interaction with the elements and structure of the work. Needs analysis identifies elements and structure of the work. In the analytic strand we will develop a scoring method for measuring alignment. Our approach draws on and integrates a wide set of observations and proposals in HAI, Human Computer Interaction (HCI), Work Domain Analysis (WDA), and related disciplines. In the empirical strand we test the prediction that HAI designs that align with work more strongly will be easier to learn, particularly, easier to master using the automation for novel problem solving. We test this hypothesis by identifying and measuring designs that differ in ATW alignment and then comparing the designs with high versus low scores for how easily they are learned, how well retained, and how flexibly they can be used.