Reporting for the period 11/7/16-11/6/18
To date, project efforts have centered on clarifying program focus with research and operational sponsors, and building the evidence-base for assessment and selection practices through literature review.
Task 1.1. Finalize competencies, tasks, and timescales and performance criteria for each analog. Per discussions with NASA research and operational sponsors, this work focuses on three main LDSE competencies and their sub-competencies: teamwork (including team orientation, team care, communication), leadership / followership, and operational problem solving (including judgment, adaptability).
Task 1.2. Finalize traditional competency measures. This task is largely complete. We have reviewed the literature for team orientation, team care, communication, leadership / followership, and adaptability. Some of these competencies map clearly on to constructs reported in the literature (e.g., team orientation, communication) while others are more complex (e.g., team care). For these more complex competencies, we have generated a list of component constructs related to this competency and preformed literature reviews on those. The goal of these reviews was to identify specific scales or measurement practices used for focal constructs and their associated validity evidence. We are in the process of generating white papers detailing best measurement and assessment practices for each of these areas.
Task 1.3.1 Map LDSE competencies to existing sociometric evidence and theory. This task is largely completed--we have conducted two literature reviews to synthesize available validity evidence and measurement practices for unobtrusive measures. The first focuses on the use of physiological measurement within teams. This review was systematic, as search terms are relatively definable and the literature is reasonably well organized. The second review focused on unobtrusive measures of team communication including content-based analysis methods (lexical analysis, supervised learning, and generative modeling techniques) and paralinguistic features of speech (communication flow, vocal features, gesture and posture, facial expression, and gaze behavior). This review is narrative as the literature is not well organized and spread across multiple literatures. Each of these reviews has informed study design and measurement planning for this work and will be submitted as white papers to NASA and developed as peer reviewed articles.
Task 1.3.2 Reactive systems task analysis method. We had proposed to develop a method for team task analysis that mapped unobtrusive measurement practices to a given team’s configuration and workflow. After discussion with operational team, it was decided that a scenario design method would be more helpful. We are reframing the deliverables of this task to include guidance on how to design scenarios that tap targeted LDSE competencies by generating scenario events (task conditions) representing opportunities to enact competencies linked to traditional and unobtrusive measurement practices. We are currently developing this approach by linking the findings of literature reviews described above.
Task 1.3.3 Sensor pilots. As unobtrusive measurement methods are relatively new, and there are unanswered questions about their psychometric properties, we are designing and conducting a series of studies to rigorously assess the error structure of data generated with these methods. We have identified (through literature review and task analysis) four categories of measurement facets that could systematically influence data: 1) device, equipment, and processing factors, 2) environmental and physical layout factors, 3) team characteristics, and 4) task and work process factors. We will be conducting a G-study in the upcoming months to determine the magnitude of variance associated with each of these measurement facets.
Task 4. Develop open architecture assessment system. One of the final project deliverables includes an open ‘middle layer’ system for extracting unobtrusive measures of LDSE competencies from multiple sensor systems. We have developed the first iteration of database linking data from physiological, communication, and location-detection systems. This is necessary to enable upcoming data collection efforts and sensor pilot studies. This version of the database is implemented in SQLite for rapid prototyping. The next major iteration will be developed using PostgreSQL to improve scalability for data collection across multiple sites and afford the ability to include complex data extraction methods (e.g., pre-processing of physiological signals, generating measures of synchrony across data streams for a team) within the database itself.