Keywords

1 Introduction

Vehicle teleoperation is the remote control of a vehicle, typically wirelessly. Automated vehicles (AVs) are promising for future mobility but face challenges in complex scenarios, limiting their effectiveness. Teleoperation can serve as a backup when AVs reach the limits of their Operational Design Domain (ODD) [1]. The goal of teleoperation is to offer a secure and effective method to overcome these limitations.

Key challenges in vehicle teleoperation include:

!:

Human-machine interaction: User experience with input modalities (e.g., touch, gesture, voice), feedback (e.g., haptic, visual, auditory), and cognitive load.

!:

Network latency: Variable delays degrade performance, especially in low-bandwidth or high-latency environments.

!:

Safety and reliability: Robust fault detection, recovery mechanisms, and handling communication failures or malfunctions (Fig. 1).

Fig. 1.
figure 1

Graphical depiction of the SRPT approach utilized for direct vehicle teleoperation. The remote vehicle is provided with successive reference poses as it progresses forward. The control-loop includes a state-estimation block.

Full size image

Various teleoperation concepts [1], including direct control [2,3,4], shared control [5,6,7], and trajectory guidance [8, 9], face challenges such as latency and situational awareness. This paper focuses on mitigating the impact of network latency in vehicle teleoperation using the Successive Reference Pose Tracking (SRPT) approach, which falls under the direct control classification. SRPT transmits poses instead of steering commands. This method aims to improve safety and effectiveness by reducing human overcorrection and oscillations caused by delays.

1.1 Previous Work and Vehicle State Estimation

Our previous research introduced and assessed the SRPT approach [10,11,12]. Compared to predictive display-based teleoperation, SRPT, utilizing a Nonlinear Model Predictive Control (NMPC) block, demonstrated superior performance by dynamically adjusting vehicle speed and steer during rapid maneuvers and maintaining stability under variable network delays. The control-station generates closely spaced waypoints via joystick steering. Prior research works assumed perfect vehicle states; however, real-world scenarios necessitate evaluating SRPT’s robustness against state estimation inaccuracies.

State estimation is crucial for reliable teleoperation. Methods like the Extended Kalman Filter (EKF) and Unscented Kalman Filter (UKF) estimate vehicle states using inputs from IMUs and wheel encoders. Bersani et al. [13] employed a UKF with a kinematic model for pose and velocity estimation, while Doumiati et al. [14] used an EKF with a four-wheel model to estimate side-slip angle and tire forces. Although high-accuracy methods using Lidar and GPS exist, they are costly. Our approach focuses on cost-effective, resilient solutions using only dead-reckoning sensors with EKF based state estimator to support SRPT.

1.2 Contribution of Paper

This paper assesses the performance of SRPT vehicle teleoperation under state estimation inaccuracies and challenging conditions. It highlights SRPT’s effectiveness using only dead-reckoning sensors, enhancing robustness without GPS dependency.

2 Method

The SRPT approach proposed in [11], requires vehicle states. Considering the single-track vehicle model, set of vehicle states are:

$$\begin{aligned} X=\left[ \,\beta \,, \,\dot{\psi } \,, \,\psi \,, \,F_{y, F} \,, \,F_{y, R} \,, \,V_x, \,x \,, \,y \,, \,\delta \,\right] ^T \;\;\;\; \end{aligned}$$

The NMPC block optimizes for vehicle steer-speed commands, to keep the vehicle motion in sync with the received successive reference poses [11]. Figure 3 shows the incorporation of a state-estimator in the form of EKF. Its prediction model consists of the single-track version of the four-wheel vehicle model given in [14] and it is given by

$$\begin{aligned} \dot{X}=\left[ \begin{array}{c} \frac{1}{m V_x}\left( F_{y, F} \cos \delta +F_{x, F} \sin \delta +F_{y, R}\right) -\frac{\beta \cdot a}{V_x}-\dot{\psi } \\ \frac{1}{I_Z}\left[ \left( F_{y, F} \cos \delta +F_{x, F} \sin \delta \right) l_F - F_{y, R}\,l_R\right] \\ \dot{\psi } \\ \frac{V_x}{\lambda }\left[ C_{\sigma , F} \,\sigma _{F}-F_{y, F}\right] \\ \frac{V_x}{\lambda }\left[ C_{\sigma , R} \,\sigma _{R}-F_{y, R}\right] \\ \dot{V_x} \\ V_x \left( \cos \psi - \sin \psi \; \tan \beta \right) \\ V_x \left( \sin \psi + \cos \psi \; \tan \beta \right) \\ \dot{\delta } \end{array}\right] & \end{aligned}$$

\([\sigma _{F}, \sigma _{R}]\) are the tires slips given by

$$\begin{aligned} \begin{array}{l} \sigma _{F}\simeq \tan \delta -\beta -\dot{\psi } \frac{l_F}{V_x} \\ \sigma _{R}\simeq \,\,\,\,\,\,\,\,\,\,\,-\beta +\dot{\psi }\frac{ l_R}{V_x} \end{array} \end{aligned}$$
Fig. 2.
figure 2

Single-track vehicle model.

Full size image

\([C_{\sigma , F}, C_{\sigma , R}]\) are the lumped cornering stiffness of front and rear axles respectively. [\(m_f, m_R\)] are distribution of vehicle mass on front and rear axle based on [\(l_F, l_R\)] respective distances of axles from CG (Fig. 2 and Table 1).

Fig. 3.
figure 3

Simplified block diagram of the simulation platforms set up on Simulink. Unity has no role in simulation, it is to display the manoeuvres while being performed.

Full size image
Table 1. (Simulated) Vehicle parameters for the single-track model.
Full size table

2.1 Measurement Model

A dead-reckoning set of sensors, comprising a virtual IMU, a speed encoder and a steer encoder is considered. Hence the measurement vector and the measurement model are given by

$$\begin{aligned} Z&=\left[ \,a_{y, meas} \,, \,\dot{\psi }_{meas} \,, \,V_{x, meas} \,, \,\delta _{meas}\right] ^T & \\h\left( X\right) &=\left[ \frac{1}{m}\left( F_{y, F} \cos \delta +F_{y, R}\right) , \;\; \dot{\psi }, \;\; V_{x}, \;\; \delta \right] ^T \end{aligned}$$

3 Simulation Setup

A faster than real-time, vehicle teleoperation simulation test platform is developed using Simulink and Unity3D to emulate the network delayed vehicle teleoperation system, as shown in Fig. 3. In addition to environmental disturbances depicted in Fig. 4, measurement noises (Table 2) are also considered. These noise sources are grouped into following sets:

Fig. 4.
figure 4

The track.

Full size image
  1. i.

    Actual states, no EKF

  2. ii.

    Noise set 1 : EKF + Gaussian noises

  3. iii.

    Noise set 2 : EKF + Gaussian noises + gainV + \(bias\delta \) + \(3^{\circ }tiltedImu\)

  4. iv.

    Noise set 3 : EKF + Gaussian noises + gainV + \(bias\delta \) + \(3^{\circ }tiltedImu + C_{F, over} + C_{R, over}\)

  5. v.

    Noise set 4 : EKF + Gaussian noises + gainV + \(bias\delta \) + \(6^{\circ }tiltedImu\)

  6. vi.

    Noise set 5 : EKF + Gaussian noises + gainV + \(bias\delta \) + \(6^{\circ }tiltedImu + C_{F, over} + C_{R, over}\)

Table 2. Sources of measurement noise
Full size table

Additionally, we include a teleoperation mode (under delays) with a look-ahead driver model, representing a typical human operator, steering the vehicle to align with the reference trajectory, serving as a baseline for comparing SRPT modes.

4 Results and Discussion

Simulations revealed significant oscillations with the look-ahead driver model during double lane change (C) and slalom (H) sections due to network delays. In contrast, the SRPT mode effectively eliminated these oscillations in all sections. The NMPC block considers steer-rate limitations and strategically decelerates the vehicle, allowing for more time to maneuver. Fig 5 compares cross-track errors across all modes. Key findings are:

Obs I::

Expectedly, Lookahead-Optimum consistently outperforms Lookahead-Suboptimum.

Obs II::

SRPT reduces error in high steer demand regions like C and H, countering instability in H better due to SRPT’s speed control.

Obs III::

SRPT performs better in regions with low adhesion (D), but shows a slight decrease in performance in regions with extremely low adhesion (G) due to significant changes in the system model. Nevertheless, this doesn’t significantly affect its overall performance.

Obs IV::

In windy conditions (E and F), SRPT maintains stability and outperforms Lookahead driver model in region F despite measurement noise.

SRPT vehicle teleoperation performs equal to or better than the lookahead driver mode in all regions.

Fig. 5.
figure 5

Cross-track error for Lookahead and SRPT vehicle teleoperation modes at \(22\,km/h\) under network delays. SRPT maintains performance even with measurement noise.

Full size image

4.1 How SRPT Is Unaffected from Diverging State Estimation?

Intrinsically, The SRPT approach relies on accurate vehicle pose estimation within a moving time window equal to the round-trip network delay (200–300 ms). Although global pose estimation may diverge over time, the proposed EKF provides accurate estimation within this small time window. This ensures SRPT’s robustness under varying network delays, demonstrating its resilience in adverse environmental conditions.

5 Conclusion

In our previous work, we introduced the SRPT approach for vehicle teleoperation, transmitting successive reference poses instead of steer commands. This study assessed SRPT’s performance under state estimation errors using simulations and an EKF vehicle state estimation using dead-reckoning sensors. The remote vehicle was simulated with a 14-DOF vehicle model, and network delays were modeled with a FIFO nature GEV distribution, ranging from 200–300 ms. We simulated various adverse conditions, including challenging maneuvers, low-adhesion tracks, strong crosswinds, and significant measurement noise with dead-reckoning sensors.

Our findings show that SRPT outperforms the lookahead driver model under variable network delays during both normal and aggressive maneuvers. Especially during prompt maneuvers such as tight cornering, double lane changes, and slalom, SRPT mode exhibits significantly less cross-track error. This is desirable as these are common real-life maneuvers. SRPT achieves this by automatically moderating vehicle speed when needed. SRPT’s resilience to network delay variability is promising for teleoperation systems, making them robust to measurement noise, network delays, and adverse conditions.

However, extreme network discontinuity can still strand the vehicle, and sensor failures require redundancy. Real-time computational resources are also necessary for optimizing the NMPC horizon. Notably, NMPC optimization performs within 5 ms on a standard office computer, which is sufficient given the 50 Hz optimization cycle frequency.

Future work will involve deploying SRPT on a real vehicle and control station, with operators generating reference poses using a steering joystick. Real-world experiments in urban scenarios, including online estimation of cornering stiffness and road adherence, will further enhance robustness.