Preliminary Findings for HERA Campaign 6 Missions 1-3
The Virtual Intelligent Task Assistant (VITA) study to be conducted in HERA Campaign 6 started on September 2021. At the time this report was submitted, the VITA sessions for three missions in the NASA Human Exploration Research Analog (HERA) Campaign 6 (C6) were completed, and the fourth mission was in progress. We worked with our HERA Experiment Support Scientists (ESSs) Michael Merta and Reinhold Polvilaitis to prepare for each mission. Prior to each mission, we train the crew on how to interact with the VITA. After each mission, we debrief each crew member about their experience using VITA during the mission. Data collection and preliminary data analysis for the VITA study in HERA C6 were performed on data from the first three missions during the reporting period as well. We met with Millenia Young, the NASA biostatistician working on VITA, to define statistical analysis that will be performed once data from C6 Mission 4 is available.
We report preliminary results from 12 participants in HERA C6 Missions 1-3 in the remainder of this section.
Situation Awareness (SA)
Two types of situation awareness data are collected, situation awareness during performance of the task and situation awareness after completion of the task. We report preliminary descriptive statistics for each.
Situation Awareness during the Task
Two assessments are collected for each session, one mid-way through the task (interim) and one near the end of the task (final). Each assessment consists of the user indicating their situation awareness as a value ranging from 1 (lowest SA) to 7 (highest SA). Users are given the following definition of situation awareness to help make this assessment:
Situation awareness is the perception, understanding, and anticipation of factors impacting the effective completion of the tasks. • Are you getting the information that you need to do the task? • Do you understand what you are doing? • Do you have a good feel for what is coming next?
The descriptive statistics for situation awareness during the task by condition for three HERA C6 missions indicate the Team Tablet condition received the highest SA ratings (M = 6.14, 6.33), followed closely by the Solo Tablet (M = 6.04) while HoloLens was rated lower (M = 5.04).
Situation Awareness after the Task
The second type of SA data was collected using a questionnaire administered at the end of the task. This questionnaire uses the Situation Awareness Rating Technique (SART) to measure SA. The SART questionnaire has 10 questions grouped into the following categories:
• Task Demand. The amount of attention needed to perform the task and the task complexity and dynamics that affects this. A lower number indicates a less demanding task. • Supply of Crew Resource. Crew alertness, spare mental capacity, and the ability to focus on the task. A higher number indicates more crew resource is available for the task. • Availability of Information for Crew Understanding. How much information is provided to the crew, how easy it is to access this information, and the crew’s familiarity with the tasks. A higher number indicates a better task understanding.
The descriptive statistics for situation awareness after the task by condition for three HERA C6 missions indicate the task was considered the least demanding for the Team Reader (M=3.14) while other conditions were slightly more demanding. The crew resources available were slightly less when using Tablet (M=4.68) and slightly more for Team (M=5.02, 5.11). Crew resources for HoloLens condition were between Tablet and Team conditions. Crew information for understanding was the highest for the Team Reader (M=5.39) and the lowest for HoloLens (M=4.78).
Workload
The Bedford workload scale is used to assess the workload for each condition. Workload varies from 1 (least workload) to 10 (highest workload). Workload is measured at the end of each assembly session (condition). The descriptive statistics for workload by condition for three HERA C6 missions suggest that workload for HoloLens (M=5.58) is higher than other conditions, all of which have a mean workload of less than 3. A plot of the mean workload during missions 1-3 for each rover assembly procedure indicates the mean workload for HoloLens varies between a mean of 3-7 for the different rover assembly procedures.
Usability
The System Usability Scale (SUS) is used to assess usability of the different conditions. The SUS scale is rated from 1 (“Strongly Disagree”) to 5 (“Strongly Agree.”) Values are then scaled to be between 0 and 100. Usability is measured at the end of each rover assembly session.
For Campaign 6 Missions 1-3, preliminary descriptive statistics indicate the Team condition (M=80.42, 84.55) was considered the most usable, closely followed by the Tablet condition (M=77.29). The HoloLens condition (M=48.96) was considered less usable than other techniques.
Discussion of Preliminary Findings
Descriptive statistics for mean workload and usability for Missions 1-3 indicate the VITA augmented reality interface on the HoloLens 1 requires more workload and is less usable than either the Team or Tablet conditions. There are some observations about this augmented reality interface that might have affected these findings.
First, system reliability may have influenced the users’ ratings of augmented reality. Occasionally the HoloLens 1 would stop working and require a system restart. This would usually happen after 50 minutes or more of use and is suspected to be due to overheating. Our protocol required users to take a 5-minute break after 50 minutes of use to give the HoloLens time to cool. Even with this protocol, typically, one user per mission experienced a system crash. This system restart potentially affected both the users’ perception of the technology as well as the workload and SA measures during the session.
Second, the augmented reality system usability ratings may be affected by a few observations. First, most crew are not familiar with augmented technology, meaning it requires some users more practice to master. Additionally, some crew experienced fatigue when using the augmented reality headset. Also, gaze control was reported as fatiguing to some users.
Finally, the augmented reality system performance was observed to become less responsive at times. This likely was due to a combination of occasional slowness in the VITA software and intermittent network slowness due to loading.
To address some of these issues and improve the use of augmented reality in later VITA studies, the VITA augmented reality interface is being moved to the HoloLens 2 [ https://www.microsoft.com/en-us/hololens/ ] running Unity [ https://unity.com/ ] on the HoloLens 2 headset.
In addition to these findings based on data that were collected at the end of each condition (session), a questionnaire was administered after the final VITA session (called the end of mission questionnaire). This questionnaire was intended to collect crew's subjective assessment of the different techniques after they had used all of them.
One question in this survey asked users to indicate whether they preferred figures or text instructions for rover assembly:
“For all methods, to what extent did you rely on / prefer text vs. figure instructions to successfully complete the procedure actions?”
Through Mission 3, most participants (N = 8) reported preferring figures to text instructions (N = 2), while only a few had no preference (N =2). Thus, in HERA Campaign 7, we will investigate a style of procedures that relies more on figures.
|