1 Introduction

Remote applications have significantly increased in the post-COVID era as face-to-face interactions have been limited considerably. A study on teleworking, conducted across eight countries (USA, UK, Germany, Italy, Sweden, China, Korea, and Japan) before and after the pandemic revealed a significant increase in remote workers in all the countries [1]. Furthermore, a significant percentage of respondents expressed interest in continuing to work remotely in the future. Web conferencing and other services accounted for 11% of the internet traffic in 2017, and this number was expected to almost double by 2022 [2]. However, the COVID-19 pandemic has increased the growth rate of IP video traffic [3]. However, remote applications continue to face challenges, particularly in remote collaboration activities requiring video and audio synchronization between remote locations. While video calling applications, such as Zoom are being widely used on smartphones, communication latency limitations have increased the difficulty of realizing remote ensembles and cheers at live concerts and sporting events [4, 5].

Consequently, the implementation of mobile technologies for remote collaboration has proliferated in recent years [6]. Firstly, video calling applications such as Zoom have become widely used on smartphones, and software for remote ensembles has also been introduced on mobile devices as a result of the COVID-19 pandemic. Secondly, the demand for remote collaboration has also increased in automated driving, where low-latency video and audio streaming from mobile environments is necessary for the remote monitoring of automated driving. Additionally, novel services have been developed, which integrate autonomous and remote collaboration such as gaming environments on autonomous vehicles. Remote collaboration may not be optimal even in fixed environments owing to congested or unstable wireless local area networks (LANs). However, these challenges can be mitigated by using mobile networks. Extensive research is being conducted across various domains to overcome the challenges associated with implementing remote collaboration on mobile devices. In particular, efforts have been focused on the implementation of 5 G technology in the mobile communication environment [7], which is expected to significantly improve the quality of service (QoS) of remote collaboration. For example, tests are being conducted on the latency reduction capabilities of an experimental 5 G environment for the remote monitoring of automated driving. Furthermore, in 3GPP Release 18 [8], discussions are underway on the use of web real-time communications (WebRTC) over 5 G to enhance the performance of the IP multimedia subsystem and facilitate advanced real-time communication.

Real-world 5 G environments face several issues that may influence the QoS of remote collaboration [9, 10]. For example, the mixed-band state of 5 G is expected to persist long after its spread, and the handoff between 4 G and 5 G during the transition period may cause QoS to deteriorate, as shown in Fig. 1. Detailed measurements on remote collaboration over real-world 5 G are absent.

Fig. 1
figure 1

Remote collaboration in 4 G/5 G mixed environments

Full size image

This study determined the potential of 5 G technology for remote collaboration and analyzes the challenges that must be addressed to achieve optimal QoS. Figure 2 depicts the situations in which pedestrians, trains, and automobiles utilize WebRTC through various radio environments, such as 4 G, repurposed 5 G [11], 5 G Sub6, and 5 G millimeter wave (mmWave). In this context, "repurposed 5 G" refers to the utilization of frequency bands originally allocated for 4 G, which are repurposed for the 5 G technology. 5 G was launched globally in 2019, featuring new radio technology with Sub6 and mmWave frequencies allocating ranges from 450 MHz to 6 GHz and 24.250 GHz to 52.600 GHz, respectively [12]. Although the specific frequency allocations for 5 G may vary slightly across different countries, this technology can transform communication networks. The frequencies assigned previously to 4 G are repurposed to 5 G, emphasizing the expansion of population coverage rather than the enhancement of communication performance. Consequently, the expected performance of the repurposed 5 G network is similar to that of the conventional 4 G networks.

Fig. 2
figure 2

How does handoff impact QoS during remote collaboration

Full size image

We conduct experiments to measure the QoS of remote collaboration over real-world 5 G environments and identify the challenges that influence the QoS. Our findings can contribute to the development of mobile technologies for remote collaboration and help in improving the QoS of remote activities in the post-COVID era. Comprehensive quantification regarding remote collaboration over real-world 5 G systems has remained limited. Therefore, we utilized WebRTC, an established open-source software communication protocol commonly used for remote collaboration to address this gap. We streamed video and audio through actual 5 G systems to collect temporal data on numerous key performance indicators. Our contributions are as follows:

  • We proposed and implemented a WebRTC measurement platform, which served as a tool to assess the quality of video and audio streaming over WebRTC.

  • We measured the QoS of remote collaboration under real-world 5 G conditions using our WebRTC measurement platform.

  • We evaluated the correlation between various environmental factors (signal quality, band, handoff, geographic conditions, and mobility) and remote collaboration QoS using the collected data.

  • The measurement data will be made publicly available upon the publication of this research.

The remainder of this paper is organized as follows. Section 2 discusses the related studies and highlights their differences from this study. Section 3 summarizes the key requirements of this study. Section 4 describes the measurement platform used for the remote collaboration. Section 5 presents the results of the field measurements and analysis. Lastly, Sect. 6 summarizes the findings of this study and outlines the future research implications.

2 Related Studies

Table 1 presents a comparison between the findings of previous studies and this study. Although some studies are being conducted on the use of 5 G and WebRTC, only a few have explored these technologies in a moving environment. This study is the first to conduct extensive analyses focusing on movement.

Table 1 Comparison between previous studies and this study
Full size table

2.1 Research on the Performance of 5 G Cellular Networks

The research conducted on 5 G cellular networks is two-fold: general communication performance of cellular networks and analysis of the communication performance of particular cellular networks in a particular use case.

Firstly, Xu et al. [13] conducted measurements on 5 G cellular networks on a university campus to analyze the general communication performance of cellular networks when commercial 5 G deployment in China was in its infancy and observed that the performance of cellular networks was not as good as the performance observed in the past. They then suggested protocol improvements and coexistence measures with legacy infrastructure for effective utilization. Narayanan et al. [14] measured the mmWave and Sub6 throughput of three carriers using smartphones when commercial 5 G deployment in the U.S was in its infancy and analyzed the throughput fluctuations during handoff, the correlation between the angle and distance to the antenna, and throughput. The correlation between the throughput, angle, distance to the antenna, and performance variation when moving in a vehicle was clarified. Fezeu et al. [15] analyzed the performance of mmWave radio in the physical layer corresponding to the end-to-end latency for downlink and uplink transmissions. They observed that increased delays in the physical layer can lead to congestion and jitter on the application layer. Pan et al. [16] collected radio wave information, such as reference signal received power (RSRP) and signal-to-noise ratio of 5 G during high-speed travel, the cumulative distribution of handoff intervals, and performance variation during handoffs on a Chinese high-speed railroad line covered by 5 G Sub6.

2.2 Remote Collaboration Using WebRTC

WebRTC is an open-source technology that provides real-time communication for browsers and mobile applications through a simple API [22]. Therefore, all the specifications are open, and communication details can be defined; various studies are being conducted on WebRTC. Consequently, several studies are focused on QoS and the quality of experience (QoE) of communication using WebRTC.

Kaneko et al. [17] proposed a method to reduce the influence of handoffs when using WebRTC for media streaming in an in-vehicle environment by bundling multiple cellular internet and forwarding UDP packets with multipath control. Moulay et al. [18] measured the real-world performance of WebRTC streaming over 4 G LTE cellular networks. Sathyanarayana et al. [19] created an Open Radio Access Network (RAN) environment to analyze the characteristics of WebRTC with multi-access edge computing in a LAN. Particularly, the evaluation focused on measuring jitter and latency. McClellan et al. [20] proposed a Wi-Fi communication quality measurement tool that utilizes WebRTC and can be run on a web browser. Garcia et al. [21] assessed the QoE of video streaming over WebRTC while considering different packet loss and jitter conditions. They used the video multi-method assessment fusion developed by Netflix and structural similarity to evaluate the correlation between the actual network performance and human-perceived QoE. Additionally, they employed objective video QoE metrics, such as peak signal-to-noise ratio, and objective audio QoE metrics, such as perceptual speech quality evaluation and objective listening quality analysis to provide a centralized evaluation of the QoE.

3 Platform Requirements

This section outlines the requirements to measure remote collaboration in cellular networks.

Versatility A system must support cross-platform functionality to that measurement data can be obtained from various devices. Web-based measurement tools, such as fast.comFootnote 1 and iNoniusFootnote 2 have been predominantly used in recent years; therefore, this system must be implemented as software that can run on a web browser. WebRTC-based measurements have been employed in various domains and their efficacy has been determined through various studies. For example, Chodorek et al. [23] conducted measurements in a real environment to evaluate the feasibility of air-to-ground communication between a drone and ground station using WebRTC streaming over a public wireless LAN during a disaster. Furthermore, Chodorek et al. [24] proposed a hardware implementation to enable AI computing at the edge of a WebRTC-based UAV system and demonstrated the use of LSTM-based temperature prediction. Tanskanen et al. [25] used WebRTC for web-based drone remote control system operations. The system comprised a Raspberry Pi edge device, Raspberry Pi camera, WebRTC SFU, and a monitor device that can be used to view video images. We aim to evaluate the end-to-end latency of WebRTC streams on a system similar to that presented by Tanskanen et al.

End-to-end Real-time Performance Technologies such as WebRTC, remote ensemble, and first-person shooter (FPS) games have gained significant attention in end-to-end peer-to-peer (P2P) remote collaboration. However, performance measures beyond network address translation (NAT) must be developed because the performance evaluation is performed using mobile networks. Specifically, within mobile networks, GTP is the tunneling protocol between RAN and the core network, and local addresses are assigned to the devices, which further emphasizes the importance of surpassing NAT. Gutterman et al. [26] proposed a real-time method, in relation to real-time performance measurement, to estimate the QoS from encrypted YouTube traffic. Beigbeder et al. [27] highlighted the influence of communication latency on the FPS game performance. Long et al. [28] analyzed the effects of artificially induced delay on-screen pointing tasks in a gaming context. Their results indicated that the subjective quality degraded when the latency exceeded 100 ms. Consequently, we employed software that facilitates real-time performance evaluation via P2P connections, capable of transcending NAT. A server-client approach was adopted to evaluate the uplink and downlink performance.

Real-time Measurement and Analysis Conventional methods of measuring the network performance, such as ping and iperf, may not accurately reflect the quality of remote collaboration because of the differences in the packet generation frequency, data size, congestion control algorithms, and other factors. Therefore, depending only on these measurements may not provide reliable information to assess the QoS in telecollaboration. Furthermore, remote collaboration requires low-latency transmission of video and audio as well as high-quality output that is perceptible to humans. Therefore, measuring the real-time performance of streamed video and audio and analyzing the raw data to ensure optimal quality is essential.

Mobile Data Collection This study aimed to measure the remote collaboration using an actual mixed 4 G/5 G environment to analyze the correlation between communication quality and the QoS in remote collaboration. Detailed information, such as the frequency bandwidth, radio wave strength, and connection information to 4 G/5 G networks, must be obtained during the measurement. Additionally, time and location information must be acquired in real time to account for variables, such as location and travel speed, during the analysis.

4 Measurement Platform for Remote Collaboration

We herein present a real-time remote collaboration system for QoS evaluation and implementation. The existing measurement methods, such as ping, iperf, packet generation frequency, data size, and congestion control algorithms, must be revised to evaluate the QoS for remote collaboration. Therefore, we proposed a system that satisfies the requirements discussed in Sect. 3 and can accurately evaluate the QoS for video and voice-based remote collaboration, as shown in Fig. 3. The proposed system utilizes WebRTC to measure the real-time QoS because video and audio quality must be human-like. Our measurement platform, which was stored remotely, streamed the video and audio data locally, enabling analysis of raw video and audio data. Figure 3a presents the schematic diagram of the measurement platform. We employed smartphones as the user equipment (UE) and area testers to evaluate the verifiable use cases in mixed 4 G/5 G environments. Furthermore, we implemented software that can evaluate the radio wave information and WebRTC. We also implemented software to evaluate the remote collaboration information over the internet. Cloud/server roles can be mutually assumed by deploying the WebRTC software on devices and the Internet.

Fig. 3
figure 3

Remote collaboration QoS with WebRTC over 5 G

Full size image

Figure 3b depicts the proposed measurement platform, which performed two types of measurements: WebRTC and radio wave data. The platform comprised a device side that performed the measurement and a management side that supported device control and data collection/analysis.

On the management side of WebRTC measurement, a signaling server required for P2P connections over NAT [29,30,31], data collector, and data analyses modules are all implemented in the same virtual machine (VM). The WebRTC measurement part, i.e., the WebRTC Page, is implemented in JavaScript on the Apache Web Server in the VM. The signaling server was built in Node.js on the backend, and coding was performed to enable coordination with the signaling server and Flask API in data analysis on the backend. Furthermore, the acquired data were analyzed and visualized using software. SmartViser [32] is a software that logs and displays active mobile connection information on the devices, such as bandwidth and signal strength. It was installed on the client’s smartphone and remained active during the measurement to accumulate logs. In addition to streaming audio and video, the data channel in WebRTC enabled virtual TCP or UDP-like communication to send and receive data between the devices. The communication method for the data channel of WebRTC is specified in RFC 8831 [33].

As discussed in Sect. 3, web browser-based applications, such as fast.com and iNonius, are increasingly prevalent and are designed to function within a web browser. Furthermore, the initiation and conclusion of real-time P2P connections across NAT can be configured freely. This implementation is primarily employed to gauge the communication quality within PC and mobile environments. However, the server-client method can conduct measurements by altering the starting point of the measurement to the server. Table 2 lists the software versions utilized in the implementation.

Table 2 Software version
Full size table

5 Measurement and Discussions

Here, we first described the setup environment used for the measurements, including the parameters corresponding to frequencies. Secondly, we analyzed the environment and frequencies connected to the UE in the experiments in the actual field. Subsequently, we evaluated the application performance achieved through WebRTC. Lastly, we examined and discussed the handoff performance. Figure 4 presents the experimental configurations, which show that each mobile network operator (MNO) was provided with a terminal and SIM card. For this study, we utilized two UEs, each of which accommodated multi-carriers for two MNOs. We connected a control PC to the two terminals via a USB cable to streamline the measurement process, which was operated using the Android debug bridge command. Each UE was equipped with a single SIM card, and the measurement data from two MNOs was obtained simultaneously. We then used an area tester that can measure Sub6 and mmWave to assess the radio wave condition. Table 3 lists the frequencies measured in this study. For Sub6, we measured n77 (TDD, 100 MHz) and for mmWave, we measured n257 (TDD, 400 MHz) for both 2MNOs. We determined the radio conditions in the field by measuring the coverage of each frequency. Notably, mmWave combined four frequencies with one component carrier. Therefore, in the described scenario, the data acquired from the UE and the data obtained by the area tester were aligned to the GPS information. This step was necessary since the GPS data installed on the area tester was more accurate than that of the UE’s GPS. Once the GPS data were aligned, frequency identification was performed using the ARFCN (Absolute Radio Frequency Channel Number) [34] and the frequency bandwidth obtained on the terminal side. This identification process helped in determining the specific frequency to which the UE was connected.

Fig. 4
figure 4

Measurement results

Full size image
Table 3 Measured 5 G frequencies
Full size table

We used a conventional in-home environment for a particular system to determine the influence of 5 G on remote collaboration. Table 4 lists the measurements conducted of all the equipment. Firstly, we used a high-end gaming PC and a commonly available broadband connection using IPv6 and IPoE. The home premises used in this study are situated in the same area in which the mobile devices were deployed. On the control side, we installed a VM at the Hongo campus of the University of Tokyo, Japan, where the Apache Web Server, Node.js, and data analysis described in Sect. 5 were built and deployed.

Table 4 Hardware summary
Full size table

The measurements were conducted within a radius of approximately 3 km around the lively Shibuya district in Tokyo, Japan, a bustling metropolitan city. A vehicle was driven around this area, collecting data for an average duration of five hours. During the measurement, the WebRTC performance metrics such as video latency, voice latency, and voice jitter were collected at 1-second intervals. Furthermore, pre-recorded videos with accompanying voice were used for WebRTC measurements.

5.1 Radio Field Condition

Figure 5 presents the measurement results for MNO#1. Figure 5a,  b, c depict the coverage status of Sub6, mmWave, and the frequency information of the UE’s connection, respectively. Additionally, Fig. 5a, b were captured using an area tester, while Fig. 5c was captured using a UE with the SmartViser application installed. Based on these figures, Sub6 had a wide coverage area, with some areas displaying strong radio wave strength, although not over a broad area. Conversely, only a few areas were covered for mmWave; however, the radio wave strength was approximately -100 dBm, and the likelihood of connecting to other frequencies was relatively high. The UEs were connected to the repurposed 5 G network (a 5 G version of the existing 4 G network) over a wide area, as shown in Fig. 5c. Furthermore, Sub6 was commonly available sometimes with strong RSRP, except for a few areas, as shown in Fig. 5a. Additionally, mmWave was limited to small areas, as shown in Fig. 5b, indicating that mmWave connections could be utilized for mobile connections only in small areas owing to the limited coverage of mmWave.

Fig. 5
figure 5

Radio field environment in MNO#1

Full size image
Fig. 6
figure 6

Radio field environment in MNO#2

Full size image
Fig. 7
figure 7

MNO#1: WebRTC performance results

Full size image
Fig. 8
figure 8

MNO#2: WebRTC performance results

Full size image

Figure 6 presents the measurement results for MNO#2, with data similar to that of Fig. 5. Figure 6a presents the coverage results for Sub6, with all areas within the coverage area. The data acquisition method for Fig. 6 is identical to that of Fig. 5. Furthermore, several areas exhibited strong RSRP of −70 dBm\(\tilde{\text {-}}\)80 dBm. However, some areas were below -110 dBm, which could be recognized by the UE but were unlikely to cause handoff. The coverage area was narrower than for MNO#1 (Fig. 5b), based on the mmWave results in Fig. 6b. In some areas, the signal strength was -100 dBm; however, the possibility of handoff was low because the signal strength must exceed the other frequencies for handoff to occur. Lastly, Fig. 6c depicts the frequencies to which the UEs were connected. The UE was connected to Sub6 over a wide area, with a higher signal strength when compared to other frequencies in Fig. 6a. The repurposed 5 G was deployed in some areas, and handoffs with Sub6 occurred in several areas repeatedly. Regarding mmWave, although a few areas were connected, the signal strength was low compared with those of other areas. Thus, the likelihood of UEs connecting was low because the signal strength for a necessary condition for connection even if mmWave coverage is available.

Therefore, Sub6 was deployed extensively, and mmWave depended on each MNO’s policy regarding the coverage deployment. Additionally, the conversion of existing 4 G frequencies to 5 G was likely to accelerate further because the deployment area of the repurposed 5 G was validated and the 4 G connection area was small.

Fig. 9
figure 9

Influence of handoff on the QoS of remote collaboration

Full size image

5.2 WebRTC Performance

Figures 7 and 8 present the results of the QoS of remote collaboration using WebRTC, where the cumulative distribution functions are shown for WebRTC video/voice round trip time (RTT) and voice jitter between the UE and home PC for both MNOs. The mmWave provides the best performance, whereas Sub6 exhibited a performance degradation of approximately 80%, indicating that repurposed 5 G provided optimal results, as shown in Fig. 7a. This could be further validated in Fig. 7b, c, where the performance degradation was approximately 60% and 30% for RTT and voice jitter, respectively.

Owing to the limited number of connection samples available for mmWave in Fig. 8, we focused on other features (Sub6, repurposed 5 G, 4 G). For video RTT, Sub6 was worst over the 90th percentile when compared to 4 G, whereas voice RTT was relatively stable. Additionally, the performance of Sub6 degraded above the 80th percentile for voice jitter. Based on these results, voice was more susceptible to performance degradation than video.

5.3 Handover Analysis

Figure 9 depicts a heatmap visualization of the RTT for each frequency combination during a handoff is shown in. The representative value in the heatmap was the median of the five entries after the handoff, averaged for each combination. Only the point of switching to another frequency was considered a handoff in this study. The un-handoff state refers to the scenario wherein the pre and post-handoff frequencies were in the same band and was used as a reference value. Figure 9a, b depict MNO#1 and MNO#2, respectively, and blank cells indicate undetected observations in the experimental site.

According to Fig. 9a, b, the RTT of Sub6/mmWave was lower than that of the 4 G and repurposed 5 G at the same frequency when no handoff was performed. The repurposed 5 G had a lower RTT when compared to 4 G. We compared the handoff processes from (A) Sub6 or mmWave to 4 G or repurposed 5 G and (B) 4 G or repurposed 5 G to Sub6 or mmWave. The RTT for (A) was lower than that of (B). This indicates that leaving the 5 G area resulted in degraded characteristics and larger overhead than entering the area. However, when handoffs were compared without 5 G, that is, Sub6 to mmWave or mmWave to Sub6, the RTT values were comparable, indicating a small overhead. Figure 9a, b depict the variations in RTT from 4 G to the repurposed 5 G and from the repurposed 5 G to 4 G, respectively, indicating degraded performance and large overhead. The overhead was small for handoff between Sub6 and mmWave. The overhead increased when multiple frequencies, such as 4 G and the repurposed 5 G, were used. Therefore, handoff overheads can be avoided, and the performance can be enhanced by using multiple MNOs.

6 Conclusion

This study devised a novel WebRTC measurement platform for remote collaboration and conducted extensive measurements of real-time 5 G communication performance in a real-world environment. In contrast with previous studies that employed ideal network quality measurement tools, we assessed the QoS of remote collaboration in a real-world 5 G environment. The main contribution of this study is the specific knowledge required to resolve problems when creating use cases in real-world environments, which is achieved through our unique measurement approach. Additionally, evaluating and analyzing a vehicle in a mixed environment of multiple 5 G frequencies, rather than focusing solely on traditional single Sub6 or millimeter wave frequencies, indicates various noteworthy implications for future commercial deployments. This approach presents a more comprehensive understanding of real-world scenarios, providing insights into the performance and challenges associated with managing diverse 5 G frequency bands. The findings of this study provide a basis for the optimization and enhancement of future 5 G deployments in commercial settings. Our results, which involved vehicle movement, mmWave outperformed the others, and the performance of Sub6 was unstable. Furthermore, the combination of the 4 G and the repurposed 5 G increased the overhead, despite the small overhead between Sub6 and mmWave during handoff.

A potential research direction involves the implementation of the WebRTC measurement tool used in this study into open-source software. The vast dataset acquired in our experiments will be released for different use cases, such as machine learning. Furthermore, the use of the multipath technology for in-vehicle WebRTC, which was effective in this study, presents considerable potential for future research.