This research aims to test the following hypothesis: Modifying haptic sensation alleviates the subjective perception of time delays and expedites operator’s adaptation to stochastic delays in robot teleoperations. Human sensorimotor controls rely on multimodal sensory feedback, such as the visual, auditory, and tactile cues, to make sense of the consequence of the initiated action. Any latency between the action and the consequence creates a mismatch in motor perception and thus leads to perceptual-motor dysfunction. Literature has already found that sensory manipulation, i.e., providing additional sensory modalities as reinforcement cues, can modulate the effectiveness of motor learning and rehabilitation. The rationale of the proposed approach is that by simulating virtual force of physical interactions on the operator end, the delayed visual cues of teleoperation are reinforced by multimodal sensory feedback, mitigating the perception of time delays and improving performance.

This research originally proposed three aims. The fourth aim was added for exploring longer delays:

Aim 1: Develop a haptics-based sensory augmentation system for robot teleoperation with varying delays. It develops a Virtual Reality (VR) and haptics-based simulator to support robot teleoperation with varying levels of delays (i.e., teleoperation latency). The system is expected as a potential human-robot interface in the future missions, as well as a testbed for supported the proposed human-subject experiments.

Aim 2: Perform Experiment I to explore the impact of the sensory manipulation system up to 1s delay. It collects experimental data how modified haptic stimulation expedites operator’s adaptation to varying delays in teleoperation up to 1s. The haptic simulation refers to reproducing the contact dynamics of the remote robotic system (e.g., resistance, torque, and nominal weight etc.) for operator via haptic devices.

Aim 3: Data analysis to quantify the impacts of the proposed sensory manipulation method on teleoperation performance and human function. This aim proposes to analyze the experiment data to better understand if the proposed system can improve teleoperation performance while reducing the perceived delays.

Aim 4 (added to the original proposal): Perform Experiment II to explore the impact of the proposed sensory augmentation system up to 5s delay. This aim proposes to collect experimental data how modified haptic stimulation expedites operator’s adaptation to varying delays in teleoperation up to 5s.

The deliverables of this research include: (1) proof of concept evidence about the use of sensory manipulation in reducing the sense of time delays and expediting human adaptation to time-delayed robot teleoperations; (2) multimodal sensory feedback system design suggestions for human-robot interaction (HRI) in time-delayed teleoperations; and (3) quantitative models of functional and performance improvements in a variety of delay scenarios.

This research proposes an innovative sensory manipulation approach to help reduce risks related to teleoperation delays. The neural, perception, and performance evidence contributes to the formulation of effective space teleoperation designs. The quantitative human models of perceptual and performance provide predictive models for NASA to perform risk and opportunity assessment for yet-to start missions that involve robot teleoperations. Lessons learned in this research will also inform a new training paradigm for both crewmembers and ground supports as for adapting to the changing environments in future deep space exploration with adaptive and assistive sensory augmentation. The data can also be transferred to other domains such as aviation and manufacturing industry with automation controls.

1. THE DEVELOPED SYSTEM

We developed a sensory manipulation system for providing additional sensory cues, especially haptic feedback, for robot teleoperation. The system includes the following units: a) Robot commanding unit. This system connects a robot arm with the Unity game engine for digital twinning and for haptic controls. Robot operating system (ROS) is used as the main platform for exchanging data between the ROS system and Unity; 2) Digital twinning unit. Unity game engine is used to create a digital twin model of the remote robot and the workplace. The human operator can use a Virtual Reality (VR) headset to visualize the remote workplace and the robot for coordinating the hand-picking tasks in an immersive way; 3) Haptic interface unit. It includes haptic feedback and control systems. A total of seven types of physical interactions, including weight, texture, momentum, inertia, impact, balance, and rotation, are simulated via a physics engine and then are played via a high-resolution haptic controller. As such, the human operator can feel the enriched physical processes pertaining to the hand-picking task. In this system, we also programmed the system to intentionally add nine levels of latencies – 0ms, 250ms, 500ms, 750ms, 1000ms, 1250ms, 2500ms, 3750ms, 5000ms – to the visual or haptic feedback; and 4) Human assessment unit. The last component of the system includes a set of neurophysiological sensors embedded in the VR system for real-time human assessment, including eye trackers, motion trackers, and functional near-infrared spectroscopy (fNIRS) to examine the hemodynamic activities in 24 brain areas.

2. HUMAN SUBJECT EXPERIMENTS

To test the impact of the proposed sensory manipulation method on the performance in a robot teleoperation task, two human-subject experiments were performed where Experiment I focused on delays up to 1s and Experiment II focused on delays up to 5s. The task was a replacement and repair (R&R) task in a low gravitational environment. The task involves picking up, moving, and placing four cubes with different masses as fast as and as accurately as possible. The experiments consisted of four conditions:

Control condition: haptic and visual feedback are in real-time (haptic feedback=0; visual feedback=0).

Anchoring condition: haptic feedback is in real-time, and visual feedback has varying delays (haptic feedback=0; visual feedback=250ms, 500ms, 750ms, 1000ms, 1250ms, 2500ms, 3750ms, and 5000ms).

Synchronous condition: both haptic feedback and visual feedback are delayed at the same amount (haptic feedback=visual feedback=250ms, 500ms, 750ms, 1000ms, 1250ms, 2500ms, 3750ms, and 5000ms).

Asynchronous condition: haptic feedback and visual feedback are delayed at different amounts (haptic feedback=250ms; visual feedback=250ms, 500ms, 750ms, 1000ms, 1250ms, 2500ms, 3750ms, and 5000ms).

The experiment was designed as a within-participant experiment, i.e., each participating subject experienced four conditions. The sequence order was shuffled for each subject to avoid learning effects. The performance data (time and accuracy), motion data (moving trajectory), eye tracking data (gaze focus and pupillary size), and neurofunctional data (measured by Functional Near-Infrared Spectroscopy or fNIRS) were collected. Participating subjects were also requested to report their perceived delays to compare with the actual delays. Final measurement metrics used in the analysis included: 1) performance (time on task and positioning accuracy); 2) perception (perceived delays versus actual delays); 3) cognitive load (NASA TLX and eye tracking); and 4) fNIRS.

3. RESULTS

We successfully recruited 43 healthy subjects to participate in Experiment I, and another 51 healthy subjects in the Experiment II. The following sections introduce the results of both experiments.

3.1. Performance The result showed that sensory manipulation improved the teleoperation performance in terms of time on task when the delay was up to 1s. We focus on presenting the result in the anchoring condition in this report, i.e., providing haptic cues coupled with an action, because this condition represents using simulated haptic feedback to augment a person’s motor action despite visual delay levels. The result shows that time on task was significantly reduced given the anchoring method independent of the visual delays. It could be because human subjects could rely more on haptic feedback when it was available to coordinate the teleoperation actions. The benefit of providing a real-time haptic stimulation boosted the performance to a level similar to the control condition (i.e., no delay). We did not see a significant improvement in the anchoring condition when the delay was up to 5s. It suggests that there may be a cutoff for our proposed sensory manipulation method to be useful. In addition, we did not observe significant differences across the four conditions in terms of picking and dropping accuracy. This could be due to the fact that the overall level of difficulty of the designed task was not challenging enough.

3.2. Perception It was also found the proposed sensory manipulation method could also reduce subjective feelings of teleoperation delays up to 5s. For Experiment I (delays up to 1s), the data shows that under the anchoring condition, the overall average perceived visual delay in teleoperation was significantly lower than the synchronous condition. In addition, 18% of test subjects reported a perceived visual delay that was smaller than the actual one under the anchoring condition. For Experiment II (delays up to 5s), 20% and 15% of test subjects reported a perceived visual delay that was smaller than the actual one under the anchoring condition and asynchronous condition, respectively. Knowing that both the anchoring condition and synchronous condition feature fixed haptic feedback after a motor action (either in real-time or after 250ms), means that coupling real-time haptic feedback with the action during teleoperation can mitigate the subjective feeling of delays.

3.3. Cognitive Load Our data also showed that the proposed sensory manipulation method also presented benefits in terms of cognitive load for delays up to 1s. First, we analyzed the pupillary size as the literature indicates that increased pupillary size means increased cognitive load levels. We did not see any difference among the four conditions when all data from each trial was aggregated in a holistic analysis. However, after dividing the data of each trial into three stages: the object pickup stage (20s), the object drop-off stage (20s), and the object movement stage (the remaining time), we found that anchoring condition led to lower cognitive load in both the object pickup stage and the object drop-off stage. Interestingly, the NASA Task Load Index (TLX) analysis did show a similar benefit of anchoring condition in terms of mental load. But the anchoring condition led to a higher level of confidence and a lower level of frustration in comparison with the synchronous condition and the asynchronous condition. To be noted, none of the benefits related to cognitive load, perceived frustration or self-confidence level were observed when delays were up to 5s (results not shown). Once again, it suggests that there may be a cut-off time for delays for our method to be useful.

3.4. Neurofunction (fNIRS) Finally, we also found that our proposed sensory manipulation method may present neural functional benefits (for up to 1s). We tracked 35 channels using fNIRS during all trials of Experiment I. We focused on two regions of the brain in our analysis, including the dorsal cortex and the prefrontal cortex. The dorsal cortex is believed to be related to time perception, with a higher activation level meaning more engagement in time perception. The prefrontal cortex is related to activity planning activities. Our result shows that the anchoring condition led to lower activation levels in both regions, see Fig.13. It suggests that providing real-time haptic feedback (which could be simulated haptic feedback based on physics engines) may help reduce the need for focusing on planning motor actions (prefrontal cortex) and focusing on “telling how much it delayed” (dorsal cortex). In other words, subjects should be able to focus more on the actual motor tasks – such as picking up or moving an object.

4. DISCUSSION AND CONCLUSIONS The Experiment I (n=43) confirmed a variety of benefits of the proposed sensory manipulation method in teleoperation tasks with delays up to 1s. It generally confirmed that providing haptic cues along with the initiated action could significantly reduce time on task, no matter how much visual delay is presented. It was also found that participating subjects tended to perceive a smaller visual delay when real-time haptic cues were provided. There are also benefits related to reduced cognitive load, improved perception of self-confidence and frustration levels, and more desired neural functional performance. The findings suggest that the anchoring method, i.e., providing real-time haptic feedback, has multiple performance and functional benefits. However, many of the performance and functional benefits were not observed when we prolonged the delays to up to 5s in Experiment II (N=51). One of the benefits that may still hold in Experiment II is that a significant portion of subjects still reported perceived visual delays shorter than the actual visual delays. It suggests that there may be a cut-off time of delay for our proposed method still working.

As a result, we have started an investigation to figure out the cut-off time point. First, we examined multiple ways of combining visual delay and haptic delay of different amounts, such as 250ms haptic delay plus 1250ms visual delay, showing as the “async_th_250_tv_1250” in our analysis result. In other words, for each label on the X axis, the first value refers to the haptic delay, and the second value refers to the visual delay. Then we ran a pairwise analysis between each of the combinations of haptic and visual delays in terms of performance (such as positioning accuracy) to see, starting at what combination, a difference starts to emerge. As mentioned earlier, under shorter delays, we did not observe any difference in terms of positioning accuracy. However, when visual delay was increased from 3750ms to 5000ms, and meanwhile, when haptic delay was increased from 0ms to 250ms, we started to see a significant p-value (<0.05) and found that the positioning accuracy started to drop. In order words, the cutoff time for our method to still work may be somewhere around haptic delay<250ms and visual delay<5000ms. A future investigation is needed.

Targeting on the time delays issues in robot teleoperation, this research proposes the third way in addition to automation design and training: induced human adaptation. Inspired by the motor learning and rehabilitation literature, this research hypothesizes that modified (time points, frequency, modality, and magnitude) sensory stimulation, paired with the motor actions, helps alleviate the subjective feeling of time delays, and expedite human functional adaptation to time-delayed teleoperation, without the need for excessive training, or sophisticatedly designed automation/robotic systems.

This research aims to test the following hypothesis: Modifying haptic sensation alleviates the subjective perception of time delays and expedites operator’s adaptation to stochastic delays in robot teleoperations. Human sensorimotor controls rely on multimodal sensory feedback, such as the visual, auditory, and tactile cues, to make sense of the consequence of the initiated action. Any latency between the action and the consequence creates a mismatch in motor perception and thus leads to perceptual-motor dysfunction. Literature has already found that sensory manipulation, i.e., providing additional sensory modalities as reinforcement cues, can modulate the effectiveness of motor learning and rehabilitation. The rationale of the proposed approach is that by simulating virtual force of physical interactions on the operator end, the delayed visual cues of teleoperation are reinforced by multimodal sensory feedback, mitigating the perception of time delays and improving performance.

The two aims of this project are:

Aim 1: Perform human-subject experiments to quantify how modified haptic stimulation expedites operator’s adaptation to varying delays in teleoperations. The haptic simulation refers to reproducing the contact dynamics of the remote robotic system for operator via haptic devices. Note, the haptic simulation will be modified (in terms of timing and modes) to search for strategies for minimizing the subjective feeling of delays (primary outcome measure), ensuring accelerated adaptation to delays (secondary outcome measure), and ultimately, improving teleoperation performance (success metrics).

Aim 2: Predict the short-term and long-term benefits and risks to the operators’ functions based on neurobehavioral evidence. Neuroimaging data based on electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), motion data, and performance data will be acquired to build a predictive model of human sensorimotor adaptation and performance with sensory manipulation in teleoperation tasks.

The expected deliverables of this research include: (1) proof of concept evidence about the use of sensory manipulation in reducing the sense of time delays and expediting human adaptation to time-delayed robot teleoperations; (2) multimodal sensory feedback system design suggestions for human-robot interaction (HRI) in time-delayed teleoperations; and (3) quantitative models of functional and performance improvements in a variety of delay scenarios.

This research proposes an innovative sensory manipulation approach to help reduce risks related to teleoperation delays. The neural, perception, and performance evidence contributes to the formulation of effective space teleoperation designs. The quantitative human models of perceptual and performance provide predictive models for NASA to perform risk and opportunity assessment for yet-to start missions that involve robot teleoperations. Lessons learned in this research will also inform a new training paradigm for both crewmembers and ground supports as for adapting to the changing environments in future deep space exploration with adaptive and assistive sensory augmentation. The data can also be transferred to other domains such as aviation and manufacturing industry with automation controls.

Aim 1: Develop a haptics-based sensory augmentation system for robot teleoperation.

This aim proposed to develop a Virtual Reality (VR) and haptics-based simulator to support robot teleoperation with varying levels of delays (i.e., teleoperation latency). All proposed development has been completed, including the following components:

a. A system that connected a real robot arm and a haptic controller (TouchX) was developed. Human operator could use the haptic device to control the remote robot in hand- picking tasks.

b. Unity game engine was used to create a digital twin model of the remote robot and the workplace. Human operator could use a VR headset to visualize the remote workplace and the robot for coordinating the hand-picking tasks in an immersive way.

c. Sensory augmentation system. Physical interactions, such as contact events, weight, texture, and momentum, were simulated via a physics engine, and were realized via the haptic controller. As such, the human operator could feel the enriched physical processes pertaining to the hand-picking task. To be noted, our system provides augmented haptic feelings (such as grabbing a weight in hands, or hitting a heavy object) in addition to regular tactile stimulations.

d. Teleoperation delay simulation. We programmed the system to intentionally add three levels of latencies – 250ms, 500ms, and 750ms – to the visual or haptic feedback. As a result, we could test how different levels of latencies affected the teleoperation performance with our sensory augmentation system.

Aim 2: Perform human-subject experiments (n=43) to explore the benefits of the proposed sensory augmentation system.

This aim proposed to collect experimental data on how modified haptic stimulation expedites operator’s adaptation to varying delays in teleoperation. The haptic simulation refers to reproducing the contact dynamics of the remote robotic system (e.g., resistance, torque, nominal weight, etc.) for operator via haptic devices. Based on how the haptic simulation was modified (in terms of timing and modes), four conditions were tested:

Condition 1: Control condition, where haptic feedback and visual feedback happen immediately after an action initiated by the human operator, i.e., in real time without any delay.

Condition 2: Anchoring, where the haptic feedback happens immediately after an action, i.e., in real time, while the visual feedback is delayed for 250ms, 500ms, or 750ms.

Condition 3: Synchronous, where the haptic feedback and visual feedback are both delayed synchronously after an action for 250ms, 500ms, or 750ms.

Condition 4: Asynchronous, where the haptic feedback and visual feedback are both delayed after the action, and the amounts of delays of the two feedbacks are different.

Our hypothesis is that modifying haptic sensation along with the visual feedback alleviates the subjective perception of time delays and expedites operator’s adaptation to stochastic delays in robot teleoperation. We call this method sensory manipulation.

We successfully recruited 43 healthy subjects to test the hypothesis. A VR model was created to simulate a replace and repair (R&R) task in a low gravitational environment. The task involves picking up, moving, and placing four objects with different masses as fast as and as accurately as possible. The experiment was designed as a within-participant experiment, i.e., each participating subject experienced four conditions. To avoid learning effects, the sequence order was shuffled for each subject. The performance data (time and accuracy), motion data (moving trajectory), eye tracking data (gaze focus and pupillary size), and neurofunctional data (measured by Functional Near-Infrared Spectroscopy, or fNIRS) were collected.

Aim 3: Data analysis to quantify the impacts of the proposed sensory augmentation method on teleoperation performance and human function.

This aim proposed to analyze the experiment data to better understand if the proposed system can improve teleoperation performance while reducing the perceived delays. The analysis is still ongoing with the following preliminary findings:

a. The augmented haptic feedback indeed improves the teleoperation performance. The anchoring condition, i.e., providing haptic cues coupled with the action, significantly improved the hand-picking task in terms of time, independent of the visual delays. It could be because human subjects could rely more on haptic feedback when it was available to coordinate the teleoperation actions. The benefit of providing a real-time haptic stimulation boosted the performance to a level similar to the control condition (i.e., no delay). To be noted, we did not observe significant differences across the four conditions in terms of picking and dropping accuracy. This could be due to the overall level of difficulty; e.g., if the designed task was not challenging enough.

b. The augmented haptic feedback can reduce subjective feeling of teleoperation delays. As for the perceived visual delays, the data shows that under the anchoring condition, the overall average perceived visual delay in teleoperation was significantly lower than the synchronous condition. And the perceived visual delay seemed to be the highest under the asynchronous condition. About 18% of the test subjects reported a perceived visual delay that was actually smaller than the actual one under the anchoring condition. For example, when the actual visual delay was 750ms, a subject reported 100ms as the perceived delay. It means that coupling real-time haptic feedback with the action during teleoperation can mitigate the subjective feeling of delays. As for the perceived haptic delays, the data shows a little different pattern. Subjects seemed to report a lower perceived haptic delay under the synchronized condition. This makes sense because the coupled haptic and visual feedback may help a better estimate of the haptic delay. As for the visuomotor delay perception, it shows that under the anchoring condition, a significant amount of subjects reported a delay smaller than the actual one. All these results confirmed the perceptual benefits of having haptic feedback synchronized with the action.

c. The augmented haptic feedback can reduce cognitive load. When analyzing the overall cognitive load over the entire course of the task, the synchronized condition shows the lowest cognitive load. Surprisingly, the anchoring condition and the control condition don’t show a lower load as expected. This might suggest an increased cognitive load does not often lead to a worse performance. It deserves a further investigation. In addition, we analyzed the cognitive load changes from the beginning of the task to the end of the task. It shows that across the four conditions, there is a general trend of cognitive load increasing. It may suggest that participating subjects experienced a continuously increasing load during the task. This trend is confirmed by the preliminary fNIRS analysis.

The human-subject experiment (n=43) confirmed a variety of benefits of the proposed sensory manipulation method in teleoperation tasks with delays. It generally confirmed that providing haptic cues along with the initiated action could significantly reduce teleoperation time, no matter how much visual delay presented. It was also found that participating subjects tended to perceive a smaller visual delay when real-time haptic cues were provided. The preliminary findings suggest that the anchoring method, i.e., providing real-time haptic feedback, has multiple performance and functional benefits. The cognitive load analysis found that the synchronized condition led to the lowest cognitive load, followed by the anchoring condition. A further analysis is needed to explain the divergence between the cognitive load data and the performance data.

In the next reporting period, we expect to complete the analysis of the experimental data and provide concrete human performance and functional data (neural, physiological, and self-reports) to model the implications of multimodal sensory stimulation in teleoperation with varying levels of time delays. In addition, we propose to add some additional experiments with longer delay levels (i.e., 3s) and a more difficult hand-picking task (i.e., a longer moving distance). Although the planned new experiments are out of the proposed scope of this research, we believe that they will help us better understand the boundary of the preliminary discoveries, and the applicable scenarios of the proposed method.

Targeting on the time delays issues in robot teleoperation, this research proposes the third way, in addition to automation design and training: induced human adaptation. Inspired by the motor learning and rehabilitation literature, this research hypothesizes that modified (time points, frequency, modality, and magnitude) sensory stimulation, paired with the motor actions, helps alleviate the subjective feeling of time delays, and expedite human functional adaptation to time-delayed teleoperation, without the need for excessive trainings, or sophisticatedly designed automation/robotic systems.

This research aims to test the following hypothesis: Modifying haptic sensation alleviates the subjective perception of time delays and expedites operator’s adaptation to stochastic delays in robot teleoperations. Human sensorimotor controls rely on multimodal sensory feedback, such as the visual, auditory, and tactile cues, to make sense of the consequence of the initiated action. Any latency between the action and the consequence creates a mismatch in motor perception and thus leads to perceptual-motor dysfunction. Literature has already found that sensory manipulation, i.e., providing additional sensory modalities as reinforcement cues, can modulate the effectiveness of motor learning and rehabilitation. The rationale of the proposed approach is that by simulating virtual force of physical interactions on the operator end, the delayed visual cues of teleoperation are reinforced by multimodal sensory feedback, mitigating the perception of time delays and improving performance.

The two aims of this project are:

Aim 1: Perform human-subject experiments to quantify how modified haptic stimulation expedites operator’s adaptation to varying delays in teleoperations. The haptic simulation refers to reproducing the contact dynamics of the remote robotic system for operator via haptic devices. Note, the haptic simulation will be modified (in terms of timing and modes) to search for strategies for minimizing the subjective feeling of delays (primary outcome measure), ensuring accelerated adaptation to delays (secondary outcome measure), and ultimately, improving teleoperation performance (success metrics).

Aim 2: Predict the short-term and long-term benefits and risks to the operators’ functions based on neurobehavioral evidence. Neuroimaging data based on electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), motion data, and performance data will be acquired to build a predictive model of human sensorimotor adaptation and performance with sensory manipulation in teleoperation tasks.

The expected deliverables of this research include: (1) proof of concept evidence about the use of sensory manipulation in reducing the sense of time delays and expediting human adaptation to time-delayed robot teleoperations; (2) multimodal sensory feedback system design suggestions for human-robot interaction (HRI) in time-delayed teleoperations; and (3) quantitative models of functional and performance improvements in a variety of delay scenarios.

This research proposes an innovative sensory manipulation approach to help reduce risks related to teleoperation delays. The neural, perception, and performance evidence contributes to the formulation of effective space teleoperation designs. The quantitative human models of perceptual and performance provide predictive models for NASA to perform risk and opportunity assessment for yet-to start missions that involve robot teleoperations. Lessons learned in this research will also inform a new training paradigm for both crewmembers and ground supports as for adapting to the changing environments in future deep space exploration with adaptive and assistive sensory augmentation. The data can also be transferred to other domains such as aviation and manufacturing industry with automation controls.