Aspects of testing when introducing 5G technologies into industrial automation

. This paper deals with test aspects for the introduction of 5G technologies into industrial communication. In this context, test types, test objectives and test objects need to be re-evaluated. For the information and communication industry it is new to consider speciﬁc ﬁelds of application. For the automation industry, the type of network technology is fundamentally changing. Therefore, the role of validation, demonstration, conformance tests and tests to prove the fulﬁlment of application requirements for both industries will be reconsidered. Furthermore, essential test objects are described with their possible test interfaces. The example of the test for proving the fulﬁlment of application requirements of communication links shows how closely the views of the information and communication industry and the automation industry are interconnected. On the basis of an assignment of test types to test objects, recommendations are made on how the topic of testing industrial 5G technologies can be further developed in a systematic way.

The cooperation in 5G-ACIA includes aspects of testing communication solutions. This subject area includes conformity and interoperability tests as well as reliability and timing tests with regard to the requirements of industrial automation. An important aspect here is the specification and design of interfaces. In addition to the user data transmission (User Plane), mobile communications technology also knows separate levels for control (Control Plane) and management (Management Plane) of networks. For each of these levels, in addition to radio transmission, various higher protocols such as IEEE 802.3, in future TSN standards, Internet Protocol (IP), Message Queuing Telemetry Transport (MQTT), Representational State Transfer (RESTful), etc. are used. This is the basis that will be available for industrial communication. Based on this situation, a number of questions arise for testing automation devices with 5G communication as well as for network installations.

Test within a product life cycle
The general term test denotes a methodical experiment or check to determine the suitability, characteristics or performance of an object. The test method depends on the life cycle phase of the object and on the test objective. Figure 1 provides an overview over types of tests. Validation, demonstration and trial are important stages on the way towards implementing technologies into products. If a communication standard has been developed the implementations shall be tested for conformance to the standard and for interoperability with other implementations of the communication standard. Within the production process of the developed implementations product evaluations are advisable in different productions steps. When a communication product is installed and configured within an application system approvals of installation and commissioning ensure the readiness for use and can provide documentations of the initial state. Periodic audits can discover dangers deviations of the initial or expected state. The same task take diagnosis functions. If even so an error occurs dedicated tests may discover the fault. Tests can be performed by using a simulation platform or a software test bed, or by using a hardware test bed.

Validation
Newly developed methods, algorithms or models are typically validated against the requirement specification. This can be done by simulation or by measurements of exemplary implementations in a hardware test bed. This kind of test is performed in research and in early phases of product development. With respect to industrial applications this kind of tests are necessary for example to support the development of new radio technologies considering the industrial propagation conditions. Validation procedures often focus very strongly on the specific research object and its degree of maturity. The specification of a uniform procedure is hardly possible here. However, application-related parameters and influences and conditions affecting them should already be taken into account in this phase of product development.

Prototype demonstration
Prototype demonstration is an effective way to present the key features of a new product development. That means not every aspect of the planned product have to be in the focus. The prototype is put into an operational environment according to the intended objective of the demonstration. The operational environment should represent the relevant influences and conditions. The main focus of the test is the proof of certain functions and not the in-depth metrological analysis of the behaviour of the prototype. One example of prototype demonstrations is a so called plug fest. Different vendors of network hardware and software demonstrated the interoperability of their products within a communication network. Since ICT and OT are involved in building industrial communication networks with 5G technologies, such kind of demonstrations are strongly recommended.

Test of conformance
If there is a standard for a communication technology, devices that refer to it must be tested. With a number of well specified test cases the conformity of an implementation to the related standard has to be proven. Successful test results of conformance tests are the necessary condition to build an error-free network. If 5G technologies are used for industrial communication, most likely additional communication standards such as IP or industrial application layers may be necessary. Experience has shown that implementations of communication layers and an application are usually not retroactive. This has to be considered when specifying conformance tests for industrial 5G implementations. While 5G standards are developed by 3GPP, further relevant standards are developed by IEC or by other organisations. So, it has to be clarified who is responsible for test case development and how and by whom industrial 5G implementations should be tested.

Test of application requirements
Industrial applications have a wide variety of requirements regarding network size and coverage, data volume, time and error behaviour, and passive and active environmental conditions. To avoid having to test a radio solution for every application, application profiles were defined and partially specified in [1]. A similar procedure is followed with [2]. In [1] it is also defined which parameters can be used to specify requirements. This provides the basis for testing radio solutions in an application-oriented manner to ensure that the requirements are met. This approach should be considered for the development of applicationoriented 5G test beds.

System under test
A system to be tested can be of different shape in relation to industrial 5G implementations. A system under test can be an entire communication solution or even a set of communication solutions. The latter is relevant if for example wireless coexistence issues are addressed. Furthermore, physical entities such as hardware or or software implementations or logical entities such as a link between to well-defined endpoints can be test objects. The core network and its components such as the radio access network or base station are not the subject of these considerations. We assume that these objects are part of demonstrations and proof of compliance with application requirements. However, they do not represent an extra object of investigation, for example for conformity tests.

Concept, method or algorithm
Concepts, methods or algorithms of data transmission functions or communication functions are developed during the first phase of a live cycle. Examples are concepts for plug & work, medium access methods or security algorithms. For it, models are developed or prototypes are implemented. The tests can be carried out by simulation of model instances or by investigation of prototype implementations. The concepts, methods or algorithms are validated with respect to defined functional requirements. Depending on the particular test object the reference interfaces can be very different. Therefore, these interfaces are clearly to be specified. Standardised characteristic parameters and measuring units shall be used.

Wireless communication solution
A wireless communication solution is a specific implementation or instance of a wireless communication system. It consists of wireless (automation) devices, infrastructure devices, and communication relations between them. A wireless communication solution provides the infrastructure to implement a set of logical (communication) links between distributed application functions. While in industrial communication the focus is on productive data transmission, with the introduction of 5G technologies additional functions for network management and network control will have to be considered. Such functions will become more and more important as production processes become more flexible. This results in further interfaces between application and communication in addition to the productive data transmission. If applications rely on the functionalities and behaviour of the network, these must also be the subject of tests. A wireless communication solution can be object of several types of tests. Examples are tests of installation and commissioning, periodic audits, diagnosis and failure analysis. The tests are performed with respect to the requirement specification. The requirement specification may be adapted during the life cycle for example due to plant extensions. Depending on the hardware and software implementation, and the degree of integration, the reference interfaces can be very different.

Communication link
A communication link is a relation between two endpoints of a communication solution. Several shapes of communication links are possible. If the endpoints are assumed above the MAC layer, this can be considered as an end-to-end connection of the 5G network. An IP connection is defined by the endpoints of the IP service. Most relevant for industrial automation is the logical link. It is the communication relation between two logical end points. The set of logical endpoints forms the reference interface between the application function and the communication function. User data, for example measurement data of a sensor or set points for drive control, are provided to the logical source end point for transmission. The identical user data are expected at the logical target end point within an anticipated time interval or time frame, or according to an anticipated data throughput.
For reproducible tests of a logical link and for comparable test results the requirements and conditions that influence the transmission should be standardised. This standard should be independent of any communication technology. The standard should allow to describe use cases formally. Alternatively, application profiles of connected industry and automation can be developed. Logical links are in the first place object for tests of application requirements. For it requirement specifications shall specify target conditions and characteristics in a quantifiable manner. Standard definitions should be available for relevant parameters. Depending on the hardware and software implementation, and the degree of integration, the reference interfaces of a logical link can be very different.

Wireless device
From the point of view of industrial wireless communication, a wireless (automation) device comprises both wireless communication functions and automation functions [4]. Examples of wireless devices in this context for example I/O devices, temperature transmitters, programmable logic controllers, tablets, or smart phones. A wireless device can be subject of prototype demonstration, conformance test or of test of application requirements. Depending on the hardware and software implementation, and the degree of integration, the reference interfaces can be very different. An I/O device with 5G technologies may only provide digital inputs and outputs for test purposes. temperature transmitters may provide an Ethernet or RS585 interface. In any case the hardware and software interface (IP, fieldbus, etc.) to be used for test purposes are clearly to be specified. Standardized characteristic parameters and measuring units shall be used. A specific wireless device in this context is a wireless module.

Wireless communication module
We call a wireless module an implementation, which contains wireless communication functions but normally no application functions. It is typically not developed specifically for industrial applications, but must meet their requirements. Therefore, industrial 5G implementations do not require any additional conformity tests for radio modules. Proof of compliance with application requirements is, however, a relevant test task. In addition, radio modules can also be part of application-related demonstrations. Wireless modules typically provide for the application computer interfaces such as PCI express, SPI or USB . In some cases, the application can also be implemented on a processor used for communication.

Characteristic parameter values and interfaces
If parameters that characterise communication are to be determined on a standardized basis, a uniform understanding of the relevant interfaces, the measurement endpoints, is of decisive importance. However, there are frequently misunderstandings about the relevant interfaces and the associated terms. Here, a clear distinction must be made between the application view and the 5G network view. The biggest problem is when the application requires values promised by the 5G network. But the 5G network is not the complete communication stack.
In figure 2 is a communication stack with the different layers pointed. The different layers lead to different responsibilities. A stack have a application layer 1 which we count among the higher communication layers (HCL). The application layer provides the data for the application. The bottom layer is the physical layer. The physical layer and the MAC layer 2 pertain to the 5G network. In some respects, the IP layer 3 is part of the considerations to. The layers of the 5G network we count to the lower communication layers (LCL). For the assessment of the communication it is important to differentiate between the 5G network performance and the whole communication up to the application layer. In figure 2 the different parts to be assessed are marked with different coloured arrows. The red arrow depicts the application point of view. The light violett shows the 5G network with the IP layer and the back arrow without the IP layer. It is to see, it is not likely that the the same values will be reached by the application layer interface compared to the 5G network interface. The section of Quality of Service will go deeper in the topic of characteristic parameters. In this section the clear distinction of parameters regarding to the interface is shown. Figure 3 shows the transmission of messages. The source application hand over a message to the reference interface of the communication stack. In the Higher Communication Layers (HCL) the data is processed. From the HCL the data is transferred to the Lower Communication Layers (LCL) or rather the 5G network. After the transmission via the media and the LCL of the second stack, the data goes to the HCL and the target application. The figure shows various characteristic parameters using the time of the transmission as an example. For the application view the transmission time is measured, for the 5G network the end-to-end-latency. End-to-end means in this context the end of the 5G network [3]. Not to associating end-to-end with the application interface is the challenge.  Figure 4 shows a selection of characteristics for the QoS assessment at the application level. The QoS assessment uses the characteristics of the dependability. General dependability parameters [5,6] which are often used are availability, reliability, up time and down time. Characteristics for the communication are for example the transmission time and the update time ( Figure 3) as well as parameters of the status of the message (e. g. Number of Lost Messages). From all parameters can be calculated statistical values (e.g. mean up time, Message Loss Ratio). The measurement and calculation of them is explained and standardized in [1,7].

Jitter is out
If you are looking for the term jitter, you will find a large selection of definitions and interpretations for this term. Jitter is the "the maximum deviation of a time parameter relative to a reference or target value" [3]. This means jitter is not a stand-alone parameter, it comes every time together with a time parameter. One example can be at the application level the transmission time or the update time and at the 5G network level the end-to-end latency. This is one interpretation of jitter. Because of the ambiguity of jitter, for the variation of a time parameter the timeliness is given.

Timeliness
Timeliness (TL) is described by a time interval (see Figure 5). The interval is restricted by a lower bound (t LB ) and an upper bound (t UB ). This interval contains all values t A that are in time with respect to the required value t R . A message reception is considered in time or valid, if it is received within the timeliness interval. It is received outside the timeliness interval, the message reception is considered invalid. In equation 1 are the lower and the upper-bound in the interval, in equation 2 are they out. A mix of the condition is possible.

Accuracy, earliness and tardiness
Accuracy (AC) describes the discrepancy between the actual time value (t A ) and the desired time value (t R ). The value range of Accuracy can assume positive as well as negative values.
AC(t A ) < 0 for t A < t R , the actual time value is too early (4) AC(t A ) = 0 for t A = t R , the actual time value is on time (5) AC(t A ) > 0 for t A > t R , the actual time value is too late (6) Earliness (E) describes how much the actual time value is before the required value. The earliness function is positive for actual time values smaller than the required time values.
Tardiness (T) describes how much the actual time value is after the required value. The tardiness function is positive for actual time values greater than the required time values.

Survival time
According to [3] and [2] the survival time is "the time that an application consuming a communication service may continue without an anticipated message". This means that the 5G network is in the up-state for the entire survival time, even if no message has been received correctly. If a message was received correctly before the survival time expires, the 5G network will remain in the up-state, otherwise it will go to the down-state. A message is considered available when the conditions for timeliness and correctness are met. The survival time is a important parameter for the requirement specification. The number of violations of the survival time is in inverse proportion to the communication service availability. A thorough explanation can be found in Annex A of [3].

Tasks for IT and OT
It should be explicitly noted that the following assessments are made from the perspective of the application of 5G technologies for industrial automation applications. It is undisputed that beyond this, tests of 5G technologies and their implementations are necessary, but do not have to consider any specific aspects of an application. Table 5 summarizes which tests we recommend for which object to test. Concepts, methods and algorithms are to be validated in the context of research and development, whereby application-related parameters should be considered. Within the framework of the introduction of 5G technologies, coordinated demonstrations of industrial radio solutions should be offered. Standards should be developed to prove conformity to the communication standards used. A standard procedure based for example on [1] should be developed for demonstrating compliance with the application requirements. The properties of implementations of logical links will be the subject of validations and demonstrations within the framework of research and development. A standard approach should be developed to demonstrate compliance with the application requirements. From the point of view of industrial 5G implementations, test standards should be developed for wireless devices, but this is not necessary for wireless modules. Proof of compliance with the application requirements should for both objects based on a standard. The 5G Core network and its components such as the radio access network or base station are not covered in this paper. Like wireless devices and wireless modules, they will be part of demonstrations. These systems are also required to demonstrate compliance with application requirements but will not be considered separately as test items. In the tests to be standardized, the challenge is to consider the different components of an industrial wireless solution and their interactions. This requires close cooperation between ICT and OT. In addition to testing pure data transmission functions, future network management (monitoring) and network control functions must also be taken into account. Furthermore, the investigation of the behaviour of transient processes, such as the establishment of a connection or the use of a service after a longer pause, will gain special importance. The aspects of industrial 5G implementations testing discussed here should already be considered in the specification of application requirements. Specified requirements are only of value if they can be verified.