NOTE: Extended to 3/30/2014 per E. Connell/JSC (Ed., 7/3/13)

The DRP will focus on the range of time delays encountered in the ground-based control of the robotic assets on ISS, ranging from 20-50 ms (effectively for line-of-sight communication) up to 2-10 s for multiple satellite ground-station relayed (Tracking and Data Relay Satellite System, or TDRSS, and associated ground network) communication, and, in particular, as delay instantaneously varies because of real-time changes in communication paths and data buffering.

First, we will conduct human-in-the-loop (HITL) performance experiments using visual display of a dynamic simulation representative of a variety of SPHERES operations requiring different movement precision under this range of time-delay conditions. Next, we will examine HITL performance under these conditions employing mitigation techniques for short time delays such as prediction algorithms that generate compensatory in command signal lead and, for longer delays, predictive “feed-forward” graphical overlays that “look ahead” and provide a virtual view showing the future pose and location of the robot. The goal of the second of the studies is to understand the performance trades between these techniques in a wider variety of environmental and latency conditions than is usually achievable during in situ experimentation.

Finally, based on these empirical HITL results, we will design and test a strategy for combining and gracefully switching between mitigation techniques as telerobot system time delays vary across the millisecond to second range.

To conduct the DRP studies, we will build our experiment testbed derived from elements of the HET ground-to-orbit SPHERES task, encompassing ground operator user interfaces as well as computer-based simulations of the SPHERES robots and ground-ISS communication links. This strategy allows us to run HITL tests that will reduce the operating environment to offer sufficient flexibility and control for human performance experiments, yet still maintain salient features of the HET tasks key for face-validity and applicability of the results. The results from our experimental studies will help define more focused and scientifically revealing experiments that could subsequently be conducted on the ISS.

The aims of the proposed work are: 1) to employ human-in-the-loop (HITL) testing to empirically investigate the impact of variable communication delays, with latencies spanning from tens of milliseconds up to approximately five seconds, on coupled human-system performance for telerobotic systems; 2) to evaluate empirically the efficacy of existing time delay compensation schemes for this range of latencies for telerobotic tasks and control modes that have different required movement precision levels; and 3) to use the data resulting from these studies to identify the trade points between latency compensation schemes as a function of time delay and required task precision and then design and test strategies for gracefully switching between mitigation techniques as telerobot system time delays vary.

The guidelines, tools, mitigation techniques, and performance metrics developed from this research will help provide a rational basis for the design of teleoperation tasks to be carried out in the presence of communication delays. These products will in turn assist subsequent task, technology design, and validation experiment decisions regarding acceptable or desirable delay compensation techniques and define at what point to engage more autonomous operational modes.

Under this project, we studied the influence of latency due to transport delays on telerobotic user manual performance as a function of task control difficulty. For this series of experiments we first established a task that allows direct manipulation of the amount of control difficulty by the introduction of generalized rotations of the users’ control frames of reference with respect to their viewing frames. Such rotations are typical of the non-optimal viewing conditions experienced when remote teleoperation cameras are not appropriately oriented with respect to the telerobot’s end effectors or work environment. Employing sets of display-to-control rotations in three experimental investigations has allowed us to impose equivalent levels of difficulty for a highly generalized, three-dimensional, Fitts-like movement task that we studied in a high-fidelity virtual-environment simulation of a telerobotic reaching task.

In Experiment I, a wide range of rotations that varied in twist, i.e., rotation, magnitude, and axis orientation were studied using a wide range of body-referenced movement directions. In this experiment the rotation axes were aligned with the body’s canonical axes. The resulting precise measurement of what we call the Misalignment Effect Function (MEF) has allowed development of a theoretically derived explanation for part of this function’s range. It is based on pure pursuit tracking and is most parsimoniously described in terms of a proposed natural measure called normalized Path Length (nPL). This theoretical explanation, that we named the Secant Rule, has been shown accurate in predicting measured nPL for twists up to ~65 degrees. Using higher-order statistical moments of recorded normalized movement path lengths, we have shown how task performance transitions from our well characterized Secant Rule behavior to a still uncharacterized process for larger input-to-display rotations.

By repeating Experiment I with more generically oriented rotation axes Experiment II provided evidence for initially establishing nPL-based equivalence classes of rotations that could be quantitatively assigned to three known levels of control difficulty. A selection of rotation conditions from Experiment II were in turn used in Experiment III to investigate directly the interaction of latency and imposed control difficulty in a highly generalized way. Both nPL and targeting movement completion times were used in turn in a study of the interaction of control difficulty and latency.

Our analysis of participants’ ability in Experiment III to make movements in a wide variety of body-referenced target directions revealed a purely multiplicative interaction between latency and task difficulty. We showed that this interaction depends upon accounting for an internal human processing latency of ~250 ms, which parallels the interaction derived by E.R. Hoffmann (Ergonomics, 35, 1992) from experiments on Fitts’ Law. Moreover, our data suggest that the interaction varies nonlinearly with rotationally imposed task difficulty in a manner quantitatively consistent with Hoffmann’s results for Fitts’ Index of Difficulty (ID). Further analysis has led us to propose a theoretically based quantitative model in the form of an additive pair of two-way multiplicative interactions that may predict overall task performance, as represented by movement time, for targeting movements in arbitrary directions subject to arbitrary input-to-display rotations, demands on movement precision (i.e., Fitts ID), and communication delays. One of the two-way model interactions is identical to Hoffmann’s product of latency multiplied by Fitts ID. The other two-way interaction is the product of latency and the MEF resulting from our experimentally imposed input-to-display rotation.

Up to this point we have only tested and demonstrated one of the proposed model’s multiplicative two-way interactions, the one between rotation and delay. Assessing whether the pair of two-way interactions indeed does provide a valid description of the interaction between rotation, delay, and precision will require further empirical data from new studies in which all three of these factors are experimentally manipulated. These experimental studies should consider the full span of misalignment rotation angles up to 180 degrees, with specific focus both on angles where the Secant Rule applies and on larger angles where operator movement transitions to still undefined processes. Furthermore, these studies should also consider a more extensive range of time delays beyond the 800 ms we tested in Experiment III as well as a broad range of Fitts ID employed by Hoffmann rather than a single value. The testing of alternative models, including plausible purely multiplicative three-way interactions between these factors, should also be considered in the studies’ design.

Exploring the breadth of the proposed three-factor experimental parameter space would be useful in determining how far the quantitative theoretical model that we proposed could be extended. Data analyses should consider not only average operator responses but also higher statistical moment (variance, skew, kurtosis) to elucidate stochastic “noise” processes underlying teleoperator performance. Even when general closed-form mathematical descriptions cannot be expressed, the resulting experimental data, at minimum, would populate a look-up table that could provide a trade-space to weigh the relative impacts of, and the interactions between, input-to-display rotation, required precision, and communication delays for teleoperation applications as well as afford predictions of potential performance outcomes.

Future work should also include more detailed analyses of movement trajectories of individual users’ responses to stressed operation in the presence of latency. In particular, attention should be paid to different types of trajectory change points such as temporal pauses and spatial discontinuities that delimit submovements and may be indicators of underlying transitions in operator strategy and performance. Potentially, these analyses could lead to more dynamic-process models that would reveal deeper interrelations between MEF measures such as nPL and movement time as a function of teleoperation task conditions such as task difficulty (e.g., input-to-display rotation), required precision (i.e., Fitts ID), and communication latencies.

This project narrowed the gap SHFE-HARI-04 by revealing a multiplicative interaction between latency and task difficulty experimentally imposed by input-to-display rotation in a manner analogous to that reported for task precision imposed by Fitts ID. Thus our results provide a basis for establishing requirements for reference tasks in which latency may be traded off against rotational or other types of difficulty in a manner analogous to what has been shown by Hoffmann for latency versus the task precision tradeoffs modeled by the Fitts ID. The project did not close the gap because it has not yet demonstrated how this new understanding can be applied to improve existing latency mitigation techniques, for example, for teleoperation systems using rate control. Consequently, because of the maturity of this work, new specifications or guidelines for teleoperation latency mitigation strategies cannot yet be offered.

NOTE: Extended to 3/30/2014 per E. Connell/JSC (Ed., 7/3/13)

The DRP will focus on the range of time delays encountered in the ground-based control of the robotic assets on ISS, ranging from 20-50 ms (effectively for line-of-sight communication) up to 2-10 s for multiple satellite ground-station relayed (Tracking and Data Relay Satellite System, or TDRSS, and associated ground network) communication, and, in particular, as delay instantaneously varies because of real-time changes in communication paths and data buffering.

First, we will conduct human-in-the-loop (HITL) performance experiments using visual display of a dynamic simulation representative of a variety of SPHERES operations requiring different movement precision under this range of time-delay conditions. Next, we will examine HITL performance under these conditions employing mitigation techniques for short time delays such as prediction algorithms that generate compensatory in command signal lead and, for longer delays, predictive “feed-forward” graphical overlays that “look ahead” and provide a virtual view showing the future pose and location of the robot. The goal of the second of the studies is to understand the performance trades between these techniques in a wider variety of environmental and latency conditions than is usually achievable during in situ experimentation.

Finally, based on these empirical HITL results, we will design and test a strategy for combining and gracefully switching between mitigation techniques as telerobot system time delays vary across the millisecond to second range.

To conduct the DRP studies, we will build our experiment testbed derived from elements of the HET ground-to-orbit SPHERES task, encompassing ground operator user interfaces as well as computer-based simulations of the SPHERES robots and ground-ISS communication links. This strategy allows us to run HITL tests that will reduce the operating environment to offer sufficient flexibility and control for human performance experiments, yet still maintain salient features of the HET tasks key for face-validity and applicability of the results. The results from our experimental studies will help define more focused and scientifically revealing experiments that could subsequently be conducted on the ISS.

The aims of the proposed work are: 1) to employ human-in-the-loop (HITL) testing to empirically investigate the impact of variable communication delays, with latencies spanning from tens of milliseconds up to approximately five seconds, on coupled human-system performance for telerobotic systems; 2) to evaluate empirically the efficacy of existing time delay compensation schemes for this range of latencies for telerobotic tasks and control modes that have different required movement precision levels; and 3) to use the data resulting from these studies to identify the trade points between latency compensation schemes as a function of time delay and required task precision and then design and test strategies for gracefully switching between mitigation techniques as telerobot system time delays vary.

The guidelines, tools, mitigation techniques and performance metrics developed from this research will help provide a rational basis for the design of teleoperation tasks to be carried out in the presence of communication delays. These products will in turn assist subsequent task, technology design, and validation experiment decisions regarding acceptable or desirable delay compensation techniques and define at what point to engage more autonomous operational modes.

To better understand teleoperation system users’ capabilities under long time delays and, ultimately, to enable examination of compensation techniques for these time delays, we are analyzing the robustness of the classical finding that for delays above ~250 ms operators can no longer maintain smooth coordinated manual control but instead resort to a move-and-wait strategy. To test the generality of this finding and, in particular, to investigate whether the critical latency associated with the onset of move-and-wait behavior is indeed invariant, we have found a quantitatively modulated stressor that directly impacts the difficulty of a manually driven experimental teleoperation task.

The specific task we have selected is a three-dimensional Fitts-like task in which the participants move a cursor from a central starting location to touch a variably sized sphere target. The experiments were conducted in a virtual environment simulation presented in a stereoscopic head-tracked head-mounted display (HMD). The task itself is inspired by the NASA Human Exploration Telerobotics Project’s SPHERES (for Synchronized Position Hold Engage Re-orient Experimental Satellites) element.

The experimental stressor entails interposing a three-axis rotational misalignment between the visual display and manual input coordinates. We have chosen rotational misalignment as a stressor because it represents a problem commonly encountered in teleoperation: that of the remote camera not being optimally aligned with the operator’s input coordinates.

In the zero-latency studies reported here, our primary goals have been to

1) Examine the utility of this stressor in modulating task difficulty level across a broad range, spanning from almost subliminal to levels that make smoothly coordinated three-dimensional manual control almost impossible;

2) Cross-validate several metrics that we will use to quantify the degree of performance disturbance arising from application of the stressor; and

3) Formulate an analytic computational theory that takes into account both the specific geometry of the misalignment and the location of targets to predict the degree of disturbance arising from application of the stressor.

In the previous literature, the problem of telerobot input-to-display rotation has typically been addressed for misalignments about the user’s control yaw axis, but only rarely for pitch or roll misalignment. Moreover, there have been no systematic studies comparing the pattern of the disturbance between these three canonical misalignment axes. The comparison between control-display misalignment axes was a main objective of these initial empirical studies.

We use the term Misalignment Disturbance Function (MDF) for metrics such as movement time and path length that describe the disruption in user performance due to input-to-display rotation. To our knowledge, there has heretofore been no computational theory predicting the MDF based on geometric relation between rotation and target location for this type of task. We believe that the functional form of the MDF will not only be important for developing a theory of user response to control-display misalignment, but also for demarcating the effects that will be attributable to time delay once non-zero latencies are introduced into this specific teleoperation task.

In our first full study with zero latency, we collected data from 20 participants (12 M, 18 F) in order to characterize the control disturbance introduced by our input-to-display misalignment stressor. Each task trial began when the response cursor driven by the participant’s dominant hand was positioned at the central start point and a button press was applied with the nondominant hand. Participants were instructed to “make a smooth coordinated movement as quickly and comfortably as possible” from the start point to the target. Upon cursor contact with the target, the initial center starting point would reappear and the process was repeated for the next trial. Participants practiced this task during an initial familiarization and training phase both with zero input-to-display rotation and with misalignments. The complete movement trajectory for each trial was recorded and time stamped, providing direct measurements of overall movement time and trajectory path length.

The following three independent variables were used in this first study: 1) axis or misalignment rotation in either pitch, yaw or roll with respect to fixed laboratory coordinates; 2) rotation angle about the respective axis at ten discrete levels between zero and 180 degrees with participant-specific randomization; and 3) the rotation axis presentation sequence to balance for the six possible pitch-yaw-roll rotation axis orders. (Two extra participants, yielding the final total of 20, were later included as their data did not affect the planned statistical analyses.) A set of ten trials, each with a different target location, was completed for each rotation angle and axis. All ten rotation angles were completed for one rotation axis before commencing the next axis. Only data from the last seven trials in each set were used; the first three in each set served as practice during which participants could explore the rotation condition. The ratio of target size to distance from the starting point was held constant to maintain a uniform Fitts Index of Difficulty for the seven data trials.

Movement time and trajectory data from all twenty participants, pooled across all experiment conditions, were significantly positively correlated, indicative of a speed-accuracy tradeoff in their performance. Analyses of Variance (ANOVAs) conducted separately on path length and movement time (transformed to correct for data skew) produced identical results. These ANOVAs showed a significant effect of rotation angle and an interaction between rotation angle and axis, but did not reveal a significant effect of the sequence of rotation axis. General features of the data indicate that the MDF, whether expressed in terms of movement time or path length, reaches a peak at or near 120 degrees for all axes and that the peak’s magnitude was significantly greater for rotation about the roll axis.

The results from this study indicate that the MDF, i.e., the effect of the control-display rotation, is not isotropic with respect to axis of rotation: roll produces a distinctly larger nonlinearity and peak magnitude. We conjecture this is due to the displacement arising from the roll misalignment being projecting into the participant’s two key body reference axes, i.e., the lateral (left-right) axis and the vertical axis (usually aligned with gravity), which contrasts with pitch and yaw disturbances being projected into only one of these axes in the frontal plane. Our results for generalized, isotropic three-dimensional motion confirm earlier suggestions (Smith & Smith, 1962) that roll misalignments may be distinctly more difficult.

In addition to the amount of rotation, the orientation of the rotation axis could a key model parameter to predict behavioral consequences of rotations in general. In a second zero-latency experiment, we carefully controlled target location. Data from this experiment (still being analyzed) provide evidence that target direction with respect to the rotation axis predicts aspects of performance.

To estimate spatial discontinuities in recorded movement trajectories, we are developing computational tools to detect “change points” at which spatial and temporal derivatives change abruptly, indicating that participants were not able to locally adhere to our instruction for smooth movement. This discontinuity metric also correlated significantly with movement time and path length MDFs in our experiments.

We have derived a straightforward geometric description of the spiral movement paths experimentally observed for large rotational misalignments. Under the assumption that individual trial movement trajectories lie in a plane perpendicular to the axis of rotation, this geometrically founded model indicates that the MDF is well characterized by a simple secant function for rotations up to ~65 degrees.

The two investigations conducted this year have enabled us to isolate a set of multi-axis rotations and target locations for the design of a non-zero latency version of the experiment that is tractable in terms of expected participant time commitment and effort, while still offering a sufficient span of task difficulty. Pilot tests with these difficulty levels have been completed with three participants, indicating that a small number of discrete latencies settings below one second will enable us to examine with sufficient resolution the hypothesized impacts of interactions between latency and task difficulty in terms of the resultant MDF.

The DRP will focus on the range of time delays encountered in the ground-based control of the robotic assets on ISS, ranging from 20-50 ms (effectively for line-of-sight communication) up to 2-10 s for multiple satellite ground-station relayed (Tracking and Data Relay Satellite System, or TDRSS, and associated ground network) communication, and, in particular, as delay instantaneously varies because of real-time changes in communication paths and data buffering. First, we will conduct human-in-the-loop (HITL) performance experiments using visual display of a dynamic simulation representative of a variety of SPHERES operations requiring different movement precision under this range of time-delay conditions. Next, we will examine HITL performance under these conditions employing mitigation techniques for short time delays such as prediction algorithms that generate compensatory in command signal lead and, for longer delays, predictive “feed-forward” graphical overlays that “look ahead” and provide a virtual view showing the future pose and location of the robot. The goal of the second of the studies is to understand the performance trades between these techniques in a wider variety of environmental and latency conditions than is usually achievable during in situ experimentation. Finally, based on these empirical HITL results, we will design and test a strategy for combining and gracefully switching between mitigation techniques as telerobot system time delays vary across the millisecond to second range.

To conduct the DRP studies, we will build our experiment testbed derived from elements of the HET ground-to-orbit SPHERES task, encompassing ground operator user interfaces as well as computer-based simulations of the SPHERES robots and ground-ISS communication links. This strategy allows us to run HITL tests that will reduce the operating environment to offer sufficient flexibility and control for human performance experiments, yet still maintain salient features of the HET tasks key for face-validity and applicability of the results. The results from our experimental studies will help define more focused and scientifically revealing experiments that could subsequently be conducted on the ISS.

The aims of the proposed work are: 1) to employ human-in-the-loop (HITL) testing to empirically investigate the impact of variable communication delays, with latencies spanning from tens of milliseconds up to approximately five seconds, on coupled human-system performance for telerobotic systems; 2) to evaluate empirically the efficacy of existing time delay compensation schemes for this range of latencies for telerobotic tasks and control modes that have different required movement precision levels; and 3) to use the data resulting from these studies to identify the trade points between latency compensation schemes as a function of time delay and required task precision and then design and test strategies for gracefully switching between mitigation techniques as telerobot system time delays vary.

The guidelines, tools, mitigation techniques and performance metrics developed from this research will help provide a rational basis for the design of teleoperation tasks to be carried out in the presence of communication delays. These products will in turn assist subsequent task, technology design, and validation experiment decisions regarding acceptable or desirable delay compensation techniques and define at what point to engage more autonomous operational modes.

In order to better understand teleoperator system users’ abilities to operate under long time delays and, ultimately, to enable examination of compensation techniques for these time delays, we have begun to analyze the robustness of the classical finding that for delays above ~250 ms operators can no longer maintain smooth coordinated manual control but instead resort to a move-and-wait strategy.

To test the generality of this finding and, in particular, to investigate the invariance of the critical latency associated with the onset of move-and-wait behavior, we have identified a quantitatively modulated stressor that appears to directly impact the difficulty of a manually driven teleoperation task. The stressor entails interposing a three-axis rotational misalignment between the visual display and manual input coordinates. This stressor is particularly useful because it can metrically introduce a wide range of task difficulty levels ranging from almost subliminal to those that make smoothly coordinated three-dimensional manual control almost impossible. We have chosen rotational misalignment as a stressor because it represents a problem commonly encountered in teleoperation: that of the remote camera not being optimally aligned with the operator’s input coordinates.

The specific task we have selected for this investigation is a three-dimensional Fitts-like task in which the participants must move a cursor from a central starting location to touch a variably sized sphere target. This type of task could serve as the basis for an interaction technique or for a motion-planning tool that could support the HET project’s SPHERES robot satellite element with which we are collaborating.

We have recently completed data collection with twelve participants in a first study (without time delay) to characterize the degree of control difficulty our stressor can introduce. Data analysis is currently in progress.

We are beginning pilot investigations to outline the range of latencies that we will employ in our next study in which we will determine whether the point at which users switch to move-and-wait strategy depends upon task difficulty set by the rotational misalignment stressor. Additionally, we plan to identify a second transition point after which even a move-and-wait approach will no longer work. We believe such a second threshold could be associated with limitations of the users’ short-term memory.

The intent of this upcoming study is to help quantify the upper and lower time-delay bounds for which a teleoperation system user would naturally adopt a move-and-wait strategy and to understand the degree to which these bounds vary with task difficulty. These bounds will also guide the design of delay compensation experiments that we plan to conduct during FY13.

Because MIT’s camera and processor upgrade to the SPHERES robots will not be launched to ISS this fiscal year, our stakeholder, the HET-SPHERES team elected instead to consider the NEXUS-S Android smartphone camera (with cellular components disabled) as a provisional solution to provide video and real-time data feedback from ISS. The smartphone was launched on STS-135. We assisted the HET-SPHERES team in pre-flight safety testing of the phones. The HET-SPHERES team plans to use this smartphone to quantify both the bandwidth and time delay characteristics of various elements making up the ground-to-ISS communication network. The delay measurements they make will help inform the design of our time delay studies and validate the significance of our future experiment results.

The DRP will focus on the range of time delays encountered in the ground-based control of the robotic assets on ISS, ranging from 20-50 ms (effectively for line-of-sight communication) up to 2-10 s for multiple satellite ground-station relayed (Tracking and Data Relay Satellite System, or TDRSS, and associated ground network) communication, and, in particular, as delay instantaneously varies because of real-time changes in communication paths and data buffering. First, we will conduct human-in-the-loop (HITL) performance experiments using visual display of a dynamic simulation representative of a variety of SPHERES operations requiring different movement precision under this range of time-delay conditions. Next, we will examine HITL performance under these conditions employing mitigation techniques for short time delays such as prediction algorithms that generate compensatory in command signal lead and, for longer delays, predictive “feed-forward” graphical overlays that “look ahead” and provide a virtual view showing the future pose and location of the robot. The goal of the second of the studies is to understand the performance trades between these techniques in a wider variety of environmental and latency conditions than is usually achievable during in situ experimentation. Finally, based on these empirical HITL results, we will design and test a strategy for combining and gracefully switching between mitigation techniques as telerobot system time delays vary across the millisecond to second range.

To conduct the DRP studies, we will build our experiment testbed derived from elements of the HET ground-to-orbit SPHERES task, encompassing ground operator user interfaces as well as computer-based simulations of the SPHERES robots and ground-ISS communication links. This strategy allows us to run HITL tests that will reduce the operating environment to offer sufficient flexibility and control for human performance experiments, yet still maintain salient features of the HET tasks key for face-validity and applicability of the results. The results from our experimental studies will help define more focused and scientifically revealing experiments that could subsequently be conducted on the ISS.

The aims of the proposed work are: 1) to employ human-in-the-loop (HITL) testing to empirically investigate the impact of variable communication delays, with latencies spanning from tens of milliseconds up to approximately five seconds, on coupled human-system performance for telerobotic systems; 2) to evaluate empirically the efficacy of existing time delay compensation schemes for this range of latencies for telerobotic tasks and control modes that have different required movement precision levels; and 3) to use the data resulting from these studies to identify the trade points between latency compensation schemes as a function of time delay and required task precision and then design and test strategies for gracefully switching between mitigation techniques as telerobot system time delays vary.

The guidelines, tools, mitigation techniques and performance metrics developed from this research will help provide a rational basis for the design of teleoperation tasks to be carried out in the presence of communication delays. These products will in turn assist subsequent task, technology design, and validation experiment decisions regarding acceptable or desirable delay compensation techniques and define at what point to engage more autonomous operational modes.