1 Introduction

Tactile Internet (TI) requires low levels of latency combined with high levels of reliability [1], something which makes traditional transport protocols, such as UDP [2] and TCP [3], less than ideal. Instead, real-time transport protocols have traditionally been used in these networks, e.g., IRTP [4], ETP [5], HMTP [6], STRON [7], and Smoothed SCTP [8]. However, in recent years numerous end-to-end solutions have been developed for wireless and cellular networks, such as network awareness, multi-homed devices, and private network slices, among others [9]; these real-time transport protocols are not suited to take advantage of these current advancements.

Due to this evolution, a growing interest has been in researching novel protocols or extensions more aligned with current trends [10]. Examples of these new protocols and extensions include MPTCP [11], QUIC [12], MPQUIC [13], and MPRTP [14]. However, even if these protocols follow current trends, they are not directly aimed at complying with the requirements of a Tactile Internet. Not least, they lack the flexibility to adapt to the specific requirements of the different Tactile Internet applications [1]. Moreover, they come with extra features not needed for Tactile Internet, which could worsen their performance in these networks due to the added complexity and unnecessary header overhead.

Remote operation in a Tactile Internet requires a lightweight protocol flexible enough to adapt to specific preferences and unexpected network conditions. A transport protocol for remote operations should optimally trade latency and reliability to accommodate application requirements. To this end, we propose the Multi-connection Tactile Internet Protocol (MTIP), a multipath protocol specifically designed for the remote control of a Tactile Internet in current and next-generation wireless networks. MTIP manages available network paths according to application preferences to offer a tunable quality to applications requiring specific latency and high-reliability levels.

This paper presents a proof-of-concept implementation and evaluation of MTIP [15]. It extends the work presented in [16], introducing an evaluation of MTIP in an emulated environment and an additional evaluation of the protocol using an industrial robot in a real 4G/5G testbed located at the University of Malaga [17]. The aim is to assess MTIP’s ability to intelligently select network paths in diverse scenarios and show how MTIP can be configured to offer reliable, low-latency transport services at the cost of sending some redundant packets in an actual application.

The remainder of the paper is organized as follows. Section 2 surveys related work. Section 3 discusses the design of MTIP and Sect. 4 its implementation. Section 5 presents a first evaluation of MTIP’s ability to select network paths and meet the requirements of Tactile Internet applications in an emulated environment, and Sect. 6 shows a proof-of-concept evaluation of remotely controlling an industrial robot in a real 4G/5G testbed, in order to validate our previous results and illustrate the operation of MTIP in a real scenario. The paper concludes in Sect. 7.

2 Related Work

Real-time multimedia protocols have frequently been used to support Tactile Internet-like applications, e.g., RTP [18] and SCTP [19], two commonly used real-time multimedia protocols, are often used for remote control applications [20, 21]. In addition to RTP and SCTP, several specific real-time protocols and extensions have been adapted for haptic control and robotic teleoperations: Smoothed SCTP [22] is an extension of SCTP, which separates different types of traffic to reduce jitter; the Interactive Real-Time Protocol (IRTP) [23] is a protocol that combines connection-oriented and unreliable data to improve performance; and the Real Time Network Protocol (RTNP) [24] is a protocol that reduces internal delays in a specific multitasking operating system. Still, Tactile Internet remote control data possess characteristics distinct from real-time multimedia streams. In particular, remote control packets are usually smaller; they are transmitted at lower data rates, and real-time multimedia protocols operations like flow control could impose extra delay and variations on these packets, severely impeding Tactile Internet control operations.

A more comprehensive survey of Tactile Internet protocols can be found in Kokkonis et al. [25], and Antonakoglou et al. [26], with additional examples of protocols aligned with TI communications such as the Efficient Transport Protocol (ETP) [5], a protocol that adapts the inter-packet gap to network congestion conditions to reduce the round trip time, STRON [7], a protocol that uses forward error correction to reduce the latency of media streams, and HMTP [6], a multicast protocol for synchronous collaboration in a haptic network. While they collect promising solutions, these do not fully align with the potential of novel networks. Current networks offer opportunities such as information exposure and the ability to utilize multiple paths, which is a key trend in novel protocols (further information on novel trends in [9, 10]).

Multi-connectivity is the ability to manage multiple connection paths with a single protocol and opens up for more reliable data transmission without compromising latency. Multi-connectivity has grown in importance lately with the expansion of multi-homed devices and multiple access technologies in 5G networks [27]. Some of the most well-known protocols that support multi-connectivity are Multipath TCP (MPTCP) [11], Multipath QUIC (MPQUIC) [13, 28], and Multipath RTP (MPRTP) [14]. However, these protocols are not aimed at Tactile Internet communications, (e.g., [29,30,31,32]). Mainly, they lack the adaptability necessary to adapt to the varying demands of different Tactile Internet applications [1]. Furthermore, they contain a number of unnecessary features for the Tactile Internet, which could cost latency without any benefit. Tactile Internet applications need a protocol that can adapt to the specific requirements of each use case and which can take advantage of the full potential of the underlying networks. Compared with all this previous proposals, we designed the Multi-connection Tactile Internet Protocol (MTIP), a protocol that exploits multi-connectivity and context awareness to achieve the lowest latency and high (but not full) reliability with the available communication technologies.

3 Description of the Protocol

MTIP is a transport protocol for the remote control of Tactile Internet applications. MTIP exploits the benefits of having several available paths by intelligently sending redundant packets over multiple paths or sublinks, which act as a reliable link (see Fig. 1).

Fig. 1
figure 1

Use of multiple sublinks in MTIP

Full size image

In order to provide adequate transport services to tactile applications with the least redundant network resources, MTIP uses context awareness to manage and schedule traffic over multiple paths flexibly. The protocol considers both application preferences through its Application Programming Interface (API) and information about the network status through collecting network information to select the best sublinks to send data.

In the remainder of this section, we provide an overview of the MTIP protocol, emphasizing its sublink management and how it uses knowledge about application requirements and network conditions to decide which sublinks to transfer data packets.

3.1 The MTIP Packet

The structure of an MTIP packet is illustrated in Fig. 2. MTIP packets use a UDP-like header with a few additional fields. The timestamp is used to control the deadline of the packets; the sequence number controls the order and identifies duplicate packets, while the flags differentiate the two types of packet used on MTIP, i.e., data and control packets.

Fig. 2
figure 2

The MTIP header

Full size image

Data packets carry application data in the payload. These packets can be duplicated over several sublinks depending on the application preferences. On the receiving side, they are processed to filter out late or duplicate packets, confirmed by ACK messages, informing about the reception state. In contrast, control packets are not filtered out but are directly processed and acknowledged. They manage link establishment and additional operations. Control packets of the types Link and Preference are sent to establish the link, share information about the available interfaces on each endpoint, and synchronize application preferences. Other control packets include Keepalive, employed to perform periodic measurements on idle sublinks, and Characterization, used when the application demands more detailed measurements. MTIP uses Finish packets to close a communication session.

3.2 Data Transmission

MTIP can send data using all paths or only a subset of the available paths. This decision can be left to the application, although MTIP includes an internal algorithm, i.e., the MTIP sending algorithm, to make this selection. While using all paths to send data duplicated simultaneously could be the most beneficial by maximizing some Key Performance Indicators (KPI), such as reliability, in most cases, it would also waste unnecessary network resources. The MTIP sending algorithm can adapt to conditions by maximizing the KPI values and minimizing the waste of resources. As shown in Fig. 3, it makes two decisions: (i) the number of sublinks to use and (ii) which sublinks to use.

Fig. 3
figure 3

MTIP sending algorithm

Full size image

Initially, MTIP starts sending data packets on all sublinks. If the receiving endpoints get too many duplicate packets, MTIP reduces the number of sublinks it uses. In contrast, if reducing the number of sublinks results in too many packets arriving late or getting lost, MTIP increments the number of sublinks in use. This decision is based on the application preferences Duplicate Threshold and Loss-Late Threshold, which indicate the percentage of duplicate and late or lost packets considered acceptable. Then, MTIP decides which sublinks to use for the transmission by ranking available sublinks in terms of latency and reliability. The ranking is based on the application preference Latency Weight, which reflects the relative significance of the application’s latency versus reliability. A higher latency weight would rank the sublinks taking the latency KPI more into account, while a lower one would rank them prioritizing the reliability KPI.

On the receiving side, data packets go through the MTIP reception algorithm to determine if they should be sent to the application layer or discarded before being acknowledged. A representation of the MTIP receiving algorithm is shown in Fig. 4. In particular, MTIP keeps a window to control the arrival of packets: packets that are not deemed obsolete enter the reception window. MTIP uses a timestamp in the packet header to control time. MTIP assumes that both endpoints are synchronized, e.g., by using the Precision Time Protocol (PTP) [33]. MTIP features a mechanism that prevents duplicate packets from being delivered to upper layers: duplicate packets share the same sequence number and are filtered out by MTIP. When a packet arrives on the receiving side, its sequence number is checked against a list containing the sequence numbers of packets already received. MTIP also prevents the delivery of out-of-order messages. If a packet arrives out of order, MTIP stores it until i) the missing packet arrives, and they are both sent to the application layer in order, or ii) the missing packet is considered lost, and just the initial packet is sent to the application before it reaches its deadline. The deadline of a packet is calculated using the timestamps and the application preferences on the maximum one-way latency supported. A packet is considered lost when its deadline has passed (refer to [34] for further information about multipath packet expiration).

Fig. 4
figure 4

MTIP receiving algorithm

Full size image

3.3 Context Awareness

To offer an enhanced service, MTIP uses information outside the transport layer’s scope. This information can be divided into network awareness and application awareness.

Through keepalive and characterization packets, MTIP collects network state information which it stores as network KPIs for each sublink (downlink and uplink). This information is synchronized on both endpoints through ACK packets. The most relevant KPIs are latency and reliability. Latency is calculated using the timestamp on the packets and the current time, while reliability is calculated by measuring packet loss through sequence numbers.

MTIP exposes an API similar to the Socket API. In contrast to the Socket API, it includes functions to manage network and application awareness. These functions are listed in Table 1. Application preferences can be separated into communication preferences and protocol configurations. Communication preferences inform MTIP about application transport-service requirements, e.g., the maximum acceptable one-way latency and the parameters duplicate threshold, loss-late threshold, and latency weight (cf. Figs. 3 and 4). Protocol configurations have default values and define how MTIP works internally, such as the interval between keepalive packets, the importance of new measurements over older ones, and the number of relevant packets used for the measurements and the thresholds. The application can also define these protocol configurations.

Table 1 Extension of the Socket API for MTIP
Full size table

4 Implementation

We have developed the first implementation of the MTIP protocol in C++ using the open-source Qt library [35], benefiting from Qt’s interprocess communication and networking modules. The implementation follows the state diagram illustrated in Fig. 5, which shows the different states of an MTIP link during its lifetime. An MTIP link begins in the CLOSED state, and once it has been established, it enters the LINKED state, where the data exchange can start. At the end of a transport session, the MTIP link is closed, which is done similarly to TCP, i.e., via a FIN_WAIT state.

Fig. 5
figure 5

MTIP state machine

Full size image

The implementation of MTIP is depicted in Fig. 6. It exposes an API compatible with C that developers can use to create applications. When a link has been established and both endpoints have synchronized their information about the sublinks, MTIP creates sockets for each sublink. These sockets are separated into Data Sublinks (DS) and Control Sublinks (CS) but are controlled by the protocol as a single link. The DS are used to send the application data following the algorithms presented in Sect. 3.2 with the application preferences presented in Sect. 3.3. The CS, on the other hand, are in charge of sending control packets to take measurements of the network and keep network KPIs up to date. The operation of CS is supervised by the control plane or monitoring entity (MTIMP), which is in charge of storing the characterization information used by the protocol. The information is stored and shared to avoid unnecessary copies of data, using callback references and signals to keep the operation lightweight.

Fig. 6
figure 6

MTIP implementation structure

Full size image

5 Evaluation in an Emulated Environment

As a first step, we study the impact of different configurations of the MTIP algorithm in an emulated test setup. The scenarios considered have been selected to evaluate the extent to which MTIP can deliver a requested service when it experiences packet loss and packet delays. The topology under test comprises two endpoints connected by four independent sublinks, as shown in Fig. 1, where the remote controller sends messages at the typical Tactile Internet rate of 1 kHz [1].

The evaluation platform consists of a Ubuntu computer with 16 CPUs and 64 GB RAM. We use the Mininet tool [36] to create a virtual topology with independent network paths. We apply netem rules with the tc tool [37] to emulate different network impairments, including random packet losses and delays. Each test is executed 10 times to have accurate measurements. The results presented in Sect. 5.3 show the mean values of the executions. We omit all iterations from the figures, since the 95% confidence interval lower and upper bounds rarely exceed ± 5% and, in any case, never affect the tendency of the effect produced by the thresholds under study.

5.1 Scenarios

We have designed three scenarios that differ in the choice of KPI being impaired:

  • Delay scenarios emulate delays in the lower layers of the protocol stack. These delays could be caused by different configurations, such as the acknowledged mode of the Radio Link Control protocol (RLC) [38], which provides higher reliability at the cost of longer delays. Delay scenarios explore cases with significant delay variation and no losses.

  • Loss scenarios emulate lower-layer losses, e.g., losses caused by the unacknowledged mode of RLC. Loss scenarios explore cases with losses but with no significant delay variations.

  • Mixed scenarios are the most realistic of the three types of scenarios. They present the case of having both delay variation and losses caused by balanced configurations of lower layers.

All scenarios change their conditions every second. However, we create three variants of each scenario: one with good conditions on all paths, one with dynamic conditions on all paths, i.e., paths that change from good to poor conditions and vice versa, and one with poor conditions on all paths, in order to showcase a extremely bad situation. To have some reference values on the worst and best-case scenarios, we characterize the scenarios using MTIP over the paths separately, showing the average value, and then using MTIP redundantly on all paths simultaneously. We show the results of this characterization in Table 2. It is important to note that the percentage shown in the table is the amount of late and lost packets that do not reach the application layer. In delay scenarios, the percentage represents the fraction of packets whose transfer time is larger than 10 ms, a typical deadline in Tactile Internet [1]; in loss scenarios, the percentage denotes actual packet losses; while in mixed scenarios, the percentage is a combination of packets that are late and lost.

Table 2 Characterization of the testing scenarios
Full size table

5.2 Measurements

To evaluate the operation of the MTIP algorithm, we study the impact of its primary parameters: the loss-late threshold, the duplicate threshold, and the latency weight (cf. Fig. 3). We measure the trade-off between the reliability achieved and the number of wasted network resources. The reliability is measured by counting the amount of lost or late packets. In contrast, the waste of network resources is measured in terms of the number of unnecessary duplicate packets that arrive at the receiving endpoint. In order to showcase this trade-off in the graphs, we have created the trade-off metric. This metric directly compares duplicates and losses and is introduced to make it easier to compare the outcome from different scenarios. Since the rate of duplicate packets is usually higher and less critical than losses, the weight is set to reflect that.

$$\begin{aligned} {Trade\text{-}off} = \frac{ {Losses} + ({Duplicates} \times {Weight})}{1 + {Weight}} \end{aligned}$$

Our figures show the values of the trade-off metric with Weight = 0.05. This value represents the importance of wasting resources, shown by the number of duplicate packets, set to a 5% compared to the importance of receiving correct messages. This value helps us compare lost, late, and duplicate packets, see where the metric is lower, and determine which case is more beneficial. However, the decision on the actual value of the metric depends on the specific target application and the importance it assigns to reliability versus resource usage.

5.3 Results and Discussion

First, we study the effects of varying the loss-late and duplicate thresholds. In the topmost graphs of Fig. 7, the loss-late threshold is varied while the duplicate threshold is fixed, and in the bottommost graphs, the duplicate threshold is varied while the loss-late threshold is kept fixed. The fixed thresholds have a value of 5%, enabling MTIP to adapt to changes in the network without being too restrictive, allowing us to study the effect of the varied threshold.

Fig. 7
figure 7

Effect of the loss-late threshold (above) and the duplicate threshold (below) on the different scenarios

Full size image

The graphs suggest that a lower loss-late or a higher duplicate threshold causes a higher number of duplicate packets and fewer losses. In these cases, MTIP is more likely to increase than decrease the number of sublinks. The result is the opposite if the loss-late threshold is set to a higher value or the duplicate threshold is set to a lower value, with fewer network resources used at the cost of some packet losses. This is seen more clearly in worse scenarios, like the Loss (poor) one, since in fairly good scenarios, the variation on the lost-late packets is minimum.

The trade-off metric allows us to analyze the trade-off between lost-late and duplicate packets. In general, if the scenario is good, it benefits from reducing duplicates; while in worse scenarios, it depends on the specific case. For instance, in the mixed scenario (poor), both 0% and 5% loss-late threshold configurations are considered better than the 10% configuration. The reason is that with a 10% configuration, lost-late packets increase more than desirable, even if duplicates are reduced.

The last parameter under study is the latency weight used to rank and select the sublinks. This parameter can be evaluated in mixed scenarios where delays and losses could be used to rank the sublinks. In Fig. 8, we set the values of the loss-late and duplicate thresholds to 10% and 90%, respectively, to have scenarios with some losses, and evaluate the effect of the latency weight parameter on lost-late and duplicate packets. The results show that in better scenarios, a selection of paths that takes into account latency measurements (i.e., latency weight higher than 0%) is generally more beneficial than a selection that only considers reliability (i.e., latency weight 0%). This is because reliability is already improved by using multiple paths; so a selection considering latency measurements would also include this KPI in the equation, having richer information on which to perform the selection.

Fig. 8
figure 8

Effect of the latency weight on lost-late and duplicate packets

Full size image

Furthermore, we can see that there is less variation of lost-late packets than there is a variation of duplicate packets. Duplicate packets are usually more affected by an inappropriate ranking. If the sublinks are stable and the demands are not met with the sublinks currently in use, the algorithm decides to use more paths, which causes more duplicates. However, it does not affects lost-late packets that much after stabilizing. In Fig. 9, we can see why this effect significantly impacts better scenarios. We show an example of the typical sublinks selected in the different scenarios. In an exemplary scenario, a proper selection usually uses only one sublink; in a dynamic scenario, the selection includes more sublinks; and in a poor scenario, almost all sublinks are used at all times. Therefore, in this latter case, the specific ranking of the sublinks is insignificant if all of them are in use.

Fig. 9
figure 9

Sublinks selected in the different scenarios

Full size image

6 Evaluation in a Real Environment

After studying the performance of MTIP in an emulated environment, we consider a more realistic evaluation in a 4G/5G testbed. The objective of our testbed evaluation is to check the validity of our emulated tests, i.e., to confirm that the results obtained in the emulation evaluation of MTIP apply to a real-case scenario, and, secondly, to demonstrate that MTIP could be used in an actual use case. For our tests, we remotely control an industrial robot using MTIP on the Morse lab [17], located at the University of Malaga (UMA). This lab offers an infrastructure that deploys a fully private network in collaboration with Telefónica. The network supports LTE. It also supports 5G NR, enabling Non-Stand Alone and Stand Alone 5G tests. In the following subsection, we provide more details of the topology and the application.

6.1 Topology and Use Case

The experimental setup in our 4G/5G testbed is shown in Fig. 10. The setup consists of a computer that acts as a remote controller and a second computer that acts as an onboard device of an Unmanned Ground Vehicle (UGV). The computers are connected using multiple paths, and are also connected to a PTP server for synchronization at a nanosecond level (the PTP server is not shown to avoid cluttering the figure).

Fig. 10
figure 10

Experimental setup overview

Full size image

The computers run the Ubuntu operating system with a low-latency kernel [39]. Four different networks connect them: a 4G network, a 5G NSA network with special QCI (QoS Class identifier) priority, a secondary regular 5G NSA network, and a 5G SA network. The robot used in the tests is the Husky Clearpath robot [40] shown in Fig. 11. The Husky robot is a medium-sized UGV that runs the Robot Operating System (ROS), an open-source set of software libraries and tools created to build robot applications [41]. In order to run ROS applications over MTIP, we create a proxy setup that encapsulates ROS messages over MTIP. Figure 12 presents the resulting communication setup.

Fig. 11
figure 11

The Husky Clearpath robot used in the testbed tests

Full size image
Fig. 12
figure 12

Communication setup between the remote controller and the robot

Full size image

The case study we want to replicate in our tests is that of a Tactile Internet remote control of an industrial robot. In this application, the industrial robot must move within the specific deadline in a coordinated manner with other robots on the shop floor, as in an assembly line. It must also work reliably; however, it is a priority to do so within the deadline or not move at all. Otherwise, it could cause an accident or destroy important devices. Thus, when a deadline is missed, the robot does not move and misses one of its actions at the assembly line. The application should use resources wisely since it could affect both the network and the devices. Resources can be wasted both as a result of the robot missing actions at the assembly line and as a result of redundant communication. In the case study context, our trade-off metric with a weight of 0.05 captures a scenario where a missed action at the assembly line is twenty times more wasteful than sending a redundant message, representing a more efficient overall operation with a lower trade-off metric. In terms of data characteristics, this application is analogous to the one presented in the emulated evaluation, i.e., sending an order to move the robot at a constant rate with a 10 ms deadline. To allow the robot in use to process the commands and the movement to be perceptible, the rate has been reduced to 0.2kHz, a value still within reason for Tactile Internet applications [1].

6.2 Scenarios

To select the scenarios to evaluate MTIP in a real environment, we follow a similar approach as the one presented in the emulated environment. However, we have to keep in mind the characteristics of a real network and its variability in its normal state. For this reason, all the scenarios presented in this environment are dynamic and can be compared with the dynamic variants present in the previous evaluation.

The first scenario presented is the regular scenario. This scenario introduces the normal operation of the network without any additional impairments. In the network under test and any real network, there is always some end-to-end delay due to the physical distance and limitations of the devices. This delay is variable and cannot be controlled by the end user. In terms of losses, we are working with a stable network that has almost no losses when using Tactile Internet data since packets are fairly small and are sent at a relatively low rate. Therefore, the first scenario presented in this environment is comparable to the previous Delay scenarios of the emulated evaluation.

The second scenario under study is the impaired scenario and is comparable to the Mixed scenarios. Since the variability on the end-to-end delay is not something we can remove, adding losses to the scenario will make the scenario seem closer to the previous Mixed scenario than the previous Loss scenario. This scenario will consist of the delay variation created by the network and some loss impairment that will be added to all sublinks in the communication. In order to add these losses to the network, we perform some real measurements of the losses caused in the 4G and 5G modems when using a multi-Probe Anechoic Chamber to create more challenging radio conditions. An example of the real values of the losses captured can be seen in Fig. 13. Nevertheless, physically replicating this exact behavior multiple times is nearly impossible, since losses keep varying on each path through time. Thus, in order to have reproducibility, we add these real values of loss impairments to the network using the network emulation tool netem and the tc tool [37].

Fig. 13
figure 13

Losses caused with the anechoic chamber

Full size image

We characterize the scenarios to show the impact on the losses and the number of packets that usually arrive after the desired 10 ms deadline. The results of the characterization of the scenarios are presented in Figs. 14 and 15. All tests presented in this section show both the boxplot and the individual results of each of the 10 iterations performed in order to analyze with more detail the results in this dynamic scenario.

In the characterization of the regular scenario (i.e., Fig. 14), we can see how the 5G SA and the 5G NSA PRIO usually work better than the rest. The 5G NSA PRIO is a 5G NSA connection with a special priority over other data that is going through the network, so it is less affected by delay and losses. Next, we have the regular 5G NSA and the 4G paths. In general, this regular scenario presents a fairly good network that has almost no losses and usually a small amount of packets arriving later than 10 ms (medians lower than 2%). The left boxplot shows the result of using all paths at the same time, resulting in almost no late packets.

Fig. 14
figure 14

Characterization of the regular scenario

Full size image

In the characterization of the impaired scenario (i.e., Fig. 15), we add the losses presented in the previous Fig. 13. The loss impairments act differently depending on the network but still show a trend similar to the one seen in the previous characterization. This trend means that 5GNSA PRIO and 5G SA have fewer losses, the 5G NSA has more impairments, and the 4G network has the most losses. We can see the measured impact in Fig. 15, resulting in a scenario similar to the regular one in terms of late packets, but with a clear difference concerning the impact of the losses. Nevertheless, when using all paths, we can see a situation similar to the previous one, i.e., a scenario with very few late and lost packets.

Fig. 15
figure 15

Characterization of the impaired scenario

Full size image

6.3 Measurements

We evaluate the performance of MTIP at the transport and application layers. The measurements conducted at the transport layer are comparable to the ones conducted in the emulated environment. We measure the number of duplicates and of late and lost packets when increasing the loss-late threshold, decreasing the duplicate threshold, and evaluating different settings of the latency weight. Moreover, we use the trade-off metric, as presented in Sect. 5.2, to show the most beneficial cases for the environment under study. The graphs presented in Sects. 6.4.1 and 6.4.2 show the percentage of duplicates, the percentage of late-lost packets, and the aforementioned trade-off metric.

At the application layer, we measure the accuracy in terms of the percentage of robot actions that reach the robot and, thus the number of movements correctly processed by it. We also tie these results to the trade-off metric, which can help us assess efficiency, since lower values of the metric show less waste of resources for the target application. The combination of both measurements allows for a more comprehensive analysis of the most favorable cases presented. In the following subsection, we provide the results of the evaluation.

6.4 Results and Discussion

6.4.1 Duplicate and Loss-Late Thresholds

First, we study the impact of the thresholds in the regular scenario. As in the previous evaluation, we assess the impact of each threshold while setting the others to a value of 5%, a value selected to see variations and assess the effect of the threshold that is being modified. Figure 16a and b shows the number of duplicates and lost or late packets when decreasing the duplicate threshold and increasing the loss-late threshold.

Fig. 16
figure 16

Impact of the thresholds in the regular scenario

Full size image

The behavior is analogous to the one presented in the emulated scenario, with an increase in the duplicate packets tied to the decrease of late packets due to the effect of redundancy; and vice versa. Focusing on the trade-off metric when modifying the duplicate threshold, we see that reducing the number of duplicate packets enhances the trade-off as in the 80% duplicate threshold. Since the regular scenario is not highly impaired, reducing the duplicates does not significantly affect the number of late packets. However, focusing on the loss-late threshold variation, we see that having some redundancy has a beneficial effect on the trade-off as in the 0% loss-late threshold. Since the scenario is not ideal, it can benefit from some redundancy. Specifically, both evaluations show a more advantageous case when keeping duplicate packets around 20–25%.

In Fig. 17a and b, we show the results of the impaired scenario. The first impression is the same as in the regular scenario; lower values of the loss-late threshold or higher values of the duplicate threshold cause more duplicates and fewer losses. However, considering the trade-off metric, we can see that reducing the duplicate packets too much in both the duplicate threshold case or the loss-late threshold case results in a higher and less desirable trade-off metric value (see the 80% duplicate threshold or the 20% loss-late threshold). Increasing the number of duplicate packets in the last case, i.e., having around 50% of duplicate packets, seems more beneficial in this impaired case since the scenario is more prone to losses and can benefit from a higher redundancy. Nevertheless, this does not mean that a 100% of duplicate packets would be the optimal case, since the trade-off metric shows that further increasing the duplicate packets does not significantly improve the losses and this could be considered a waste of resources in the scenario under study.

Fig. 17
figure 17

Impact of the thresholds in the impaired scenario

Full size image

6.4.2 Latency Weight

The result of the evaluation of the latency weight is shown in Fig. 18. As in the previous evaluation, we set the values of the thresholds to study the impact from only using latency measurements (100%) to only using reliability measurements (0%) to select the paths from which to send data. We study both scenarios presented since both can have losses, even if this is not likely in the regular scenario.

Fig. 18
figure 18

Impact of the latency weight in the scenarios

Full size image

The results show that, in general, the difference in the configuration of the latency weight does not significantly affect the trade-off metric. In the impaired scenario, we see variations in duplicate and lost-late packets. Specifically, we see that taking into account latency measurements (25%–100%) can reduce the number of duplicates. However, with the current configuration of the trade-off metric, this reduction of duplicates does not significantly affect the metric; since it incurs in few, but some losses that also impact the metric. Moreover, as in the emulated environment, this poor scenario usually makes use of several paths at the same time, so that the specific selection of paths is not as important as the number of paths in use. In the regular scenario, the difference is slightly more noticeable in the trade-off metric. This shows that using just reliability measurements (0%) can be less convenient than using latency measurements as well. However, since the scenario is dynamic and the losses and late packets are not numerous and can occur in any of the paths, once a path is generally stable, the ranking is no longer as critical.

The behaviors mentioned can be better understood with some examples of the sublinks selected to send data in each scenario, as shown in Figs. 19 and 20. In the impaired scenario, we see a tendency to use more paths. In contrast, in the regular scenario, we can see how the protocol adapts to using just one of the paths for the whole conversation. The specific selection depends on the network behavior for the specific iteration.

Fig. 19
figure 19

Example of sublinks used in the impaired scenario

Full size image
Fig. 20
figure 20

Example of sublinks used in the regular scenario

Full size image

6.4.3 Application Layer Measurements

Now, we tie the results to the robot’s performance. Figure 21 shows the accuracy and the trade-off metric of the different configurations in the regular scenario. Focusing on accuracy, we note that the most accurate case is the one with the duplicate threshold 100%, which receives 100% of the actions accurately in most of the iterations. Nevertheless, this case is not shown to be efficient. In fact, it turns out to be the least efficient with a high trade-off metric value. In contrast, the case with a loss-late threshold of 0% shows a good balance between accuracy and efficiency, with more than 99% of mean accuracy and a low trade-off metric, proving to be the most favorable case studied for this scenario.

Fig. 21
figure 21

Accuracy and trade-off metric in the regular scenario

Full size image

Figure 22 shows the same evaluation for the impaired scenario. In this case, we can also point to the duplicate threshold 100% as the best scenario in terms of accuracy. This configuration is also one of the best in terms of efficiency. In fact, it is only worse than the duplicate threshold 90% scenario. However, the duplicate threshold 90% scenario shows better efficiency without sacrificing much accuracy (close to 99% of mean accuracy). Thus, it could be seen as a better configuration for this specific application based on the trade-off metric.

Fig. 22
figure 22

Accuracy and trade-off metric in the impaired scenario

Full size image

7 Conclusion

In this paper, we evaluate the Multi-connection Tactile Internet Protocol (MTIP), a transport protocol for the remote control of Tactile Internet applications over wireless networks. MTIP uses multiple communication paths or sublinks, with the aim to enhance reliability and meet the latency deadlines expressed by the application through the API. MTIP uses application information and network measurements to select the most suitable sublinks for communication.

We have analyzed the algorithm’s behavior to intelligently select sublinks under different configurations of the algorithm parameters. We have also measured the impact of meeting the application demands and the waste of network resources. To validate our results, we have performed a thorough evaluation in diverse scenarios, first in an emulated environment and then in a real 4G/5G lab manipulating an industrial robot.

The results show that more restrictive thresholds reduce the amount of lost and late packets and increase the number of duplicates, while less restrictive thresholds do the opposite. Moreover, regarding the concrete selection of sublinks, we have seen that a proper selection can reduce the number of duplicates, especially in fairly good scenarios. The evaluation in the real environment also shows how MITP can efficiently comply with the demands of a real application in an industrial scenario.

Future work entails exploring new, more expressive API alternatives currently being developed, such as TAPS [42]. Moreover, it will focus on exposing additional parameters to the application, such as protocol configurations, to study the protocol algorithm further and enhance the selection of sublinks.