Real-time Production Networks

. A recon(cid:12)gurable production is able to adapt to a new product in a short time. This can be built up from cyber-physical production systems (CPPS) which exchange various data with several devices simultaneously via a convergent network. In these future production systems, convergent networks will connect more participants than real-time production networks do today. In order to enable uninterrupted operation, recon(cid:12)gurations must not a(cid:11)ect existing communication. Due to this limitation, fragmentation of the con(cid:12)guration can occur, restricting future operations. This paper investigates fragmentation caused by recon(cid:12)gu-rations using a simulation to determine whether fragmentation can lead to problems in future production networks.

used for control, but are also stored and evaluated by applications, for example in order to improve the early detection of errors. Instead of separation into levels, this results in a network of different applications that communicate with each other. Real-time communication is no longer limited to local networks and spans the entire network from CPPS instead, as shown in Fig. 1 [15]. Fig. 1: The automation pyramid dissolves and separated RTPNs turn into one large RTPN. Adapted from [15].
Two possible network architectures that realize a CPPS-based automation are explained in [7] and [8]. Both topologies integrate all participants into one RTPN, which can be implemented with the vendor-independent time-sensitive networking (TSN) extensions to the IEEE 802.1 standard, for example [10]. The control is done in local cloud platforms, which execute applications for control. [8] also explains that this control can be executed on external cloud platforms. The network architecture is shown in Fig. 2. The main difference to the previous architecture of different, isolated RTPNs is that all network participants are contained in a shared RTPN. This results in a RTPN with more participants than before, which can span an entire shop floor. Due to the local separation of the sensors/actuators and the controller, the data must be transmitted over several switches, which must be configured in such a way that all real-time critical data reach their destination on time.
Due to its increased number of devices and new flexibility, changes may occur more frequently to this RTPN that require a new configuration of the switches. Examples for changes are the addition or removal of devices from the network. Furthermore, when starting a new application, such as real-time analysis of manufacturing data, additional data may need to be exchanged between the sensors/actuators and the execution platform. This must not affect the current communication. So, the rest of the production is not interrupted by the reconfiguration. In contrast, [9] only takes care during reconfiguration so that as few packets as possible fail and a modification of existing communication is tolerated.
The paper is structured as follows: In Section 2, RTPNs with minimum guaranteed latencies are introduced, followed by a description of the reconfiguration of these networks without interruption of the existing communication and the possibility of fragmentation in Section 3. In Section 4, a simulation of fragmentation is done and evaluated. Afterwards, a summary is given and possible further work is explained in Section 5 and Section 6.

Real-time Ethernet
RTPNs require minimal latency and jitter, which can be achieved by transmission at specified time slots. In addition, the CPPS architecture requires support for multiple senders and receivers. This can be accomplished, for example, by using TSN with 802.1Qbv [6]. It is assumed that all participants have a synchronized time, for example through the Precision Time Protocol (PTP) [5]. Within TSN, data are classified into traffic classes. The real-time capability is realized by defining that only data of a certain traffic class are transmitted in a time slot. Data of other traffic classes are not transmitted in this time slot. The communication runs cyclically, whereby the synchronized clocks ensure that all participants use the same synchronized cycle. This cycle is divided into time slots and provides an exclusive transmission path for the data of a traffic class. The division can be defined per port at the switch and does not have to be identical for the entire network.
In Fig. 3 this distribution is visualized in an example. A cycle from t to t + 1 is shown. From t to t+t 1 only data of the traffic class 1 can be sent, from t+t 1 to t + t 2 only traffic class 2 can be sent. The same applies to the third slot. Finally, there is a slot from t + t 3 to t + 1 which allows to send all data. This cycle is repeated continuously.
In the following, a common cycle time is assumed for the entire network. In case of different transmission cycles a hypercycle with cycle time of the least common multiple of all cycle times can be considered equivalent [3]. Also, only the traffic sent in a defined time slot is considered as real-time traffic in this article. Prioritized traffic, which is sent only prioritized to other traffic, is not considered separately and is seen as part of non-real-time traffic.

Reconfiguration of real-time production networks
As mentioned in Section 1, due to the larger RTPN and new possibilities to start/stop applications that access real-time data across the network, changes in the configuration of the RTPN are more likely. A reconfiguration adds and/or removes any number of time slots for real-time data. Existing time slots for real-time data cannot be modified (e.g. moved or split), as otherwise guarantees regarding latency and jitter could be violated [12]. This section points out which conditions must be met to reconfigure a network without disturbance to the existing real-time traffic. Afterwards, the problem of fragmentation is addressed as a consequence of the reconfiguration.

Execution of a reconfiguration in a real-time production network
When executing a reconfiguration that is not intended to restrict the functionality of the entire network, the possibilities of reconfiguration are limited in contrast to a fresh configuration without existing time slots. Time slots without real-time data do not have any requirements. These can be moved at will, shortened by time slots with real-time data (inserting a time slot at the edge) or interrupted (inserting a time slot in the middle). In general, changing the configuration can be done in two steps. First, all time slots that are no longer required are removed from the current configuration. Afterwards, the newly added time slots are inserted into the available time ranges in the cycle that are not yet occupied with real-time data. Possible incremental algorithms that support the placement of new time slots are, for example, [12] and [13].

Fragmentation
When removing and inserting time slots of different sizes, gaps may occur between time slots because existing time slots remain in the cycle at their previous time during reconfiguration. Within the context of RTPNs we define fragmentation in a communication cycle when the reserved time slots have gaps between each other so that the remaining free space in the communication cycle is not available as a single block. This can cause problems when adding a larger time slot and there is no sufficiently large gap available even though the total sum of time slots is less than the cycle time. This is similar to external fragmentation of storage systems, when files have been removed and added over time causing separated blocks of free memory [14]. However, the strategies to reduce external fragmentation (e.g. compaction) on storage systems can not be applied to fragmentation in RTPNs, as moving existing time slots within a cycle may violate guaranteed latencies. Fragmentation in the context of RTPNs is not the same as IP fragmentation, when IP packages are splitted during transmission. An example of fragmentation is shown in Fig. 4, which visualizes a sequence of reconfigurations. At the beginning (step 1), there are five time slots, which occupy the whole cycle. Then, the time slots 1 and 4 are removed and the smaller slots 6 and 7 are inserted at their position (step 3). After removing the time slot 3, there are gaps between all existing time slots in the resulting cycle so that time slot 8 cannot be inserted although there is enough space in total. This demonstrates how fragmentation can cause problems over time, as small gaps between time slots cannot be used for larger time slots. To what extent and after how many reconfigurations this problem occurs is evaluated in the following using a simulation.

Simulation
In the simulation, fragmentation is evaluated based on a real-time connection between two network participants by removing and inserting multiple time slots of different sizes. It is not considered that the time slot in a network must be reserved on the complete path from the sender to the receiver. Fig. 5 shows this conflict. A time slot for data from switch A over B to C must be available in both A → B and B → C. On the A → B route and on the B → C route there is enough space. However, on A → B the new time slot can only be inserted in the middle, while on B → C it can only be inserted at the end. Consequently, there is no time for the new slot that is available on both A → B and B → C.
Since a solution for scheduling in networks is NP-hard [12] and different algorithms produce different scheduling results, only a single connection is con- The simulation considers a cycle that is reconfigured several times. For the simulation a 100 Mbit network equivalent to current field buses is assumed, which exchanges data with a frequency of 1 kHz. Thereby, 12500 Bytes can be transmitted per cycle including the overhead of the protocol (e.g. Ethernet header). Below is a list of all the parameters of the simulation, some of which have been chosen identically to enable a comparison: The simulated reconfiguration sequence is shown in Algorithm 1. Each time a new time slot is added, its size is randomly determined from a parameterized interval. During initialization, X time slots are inserted into a cycle whose size for this simulation is set to 12500Bytes. The time slots are all inserted into the cycle one after another so that there are no gaps between them (as shown in Fig. 5 in B → C with four time slots). Afterwards, the following steps are repeated: First, Y random time slots are removed. Then, Y random time slots are inserted into the existing gaps again. If several gaps are large enough for a time slot, the smallest gap is selected in order to have larger gaps available for any following, larger time slots. The two steps of removing and adding time slots are performed until either the time slots can no longer be inserted or the maximum number of iterations has been reached. Up to 10000 iterations are executed in this contribution.
Insert X time slots next to each other; for i = 1 to 10000 do Remove Y random time slots; Try to insert Y random time slots with random sizes, each time slot uses the smallest available gap; if Insert failed then break; Algorithm 1: Simulation of reconfigurations.  Table 1, 1000 runs with different random sized time slots are performed. Each run contains up to 10000 reconfiguration iterations. The first parameter set (P 1 ) of the simulation uses 40 time slots (X) of a size of [249 : 250] and replaces one time slot per round (Y ) so that the interval of 12500 is occupied between 79.68 % and 80.32 % in total. The result is shown as a histogram in Fig. 6. The histogram uses a bucket size of 100 so that the blue bar on the left represents the 133 simulations that failed within the first 100 iterations. The red line represents the total percentage of simulations that failed until an iteration, e.g. at 10000 it has a value of 93.5 %. Thus, 935 from 1000 runs of P 1 failed before iteration 10000.
As a result, only 65 runs out of 1000 were capable to process 10000 iterations, while 25 % of the runs were not able to reach 200 iterations. Transferred to production, this means that in case of this parameterization, in 25 % of the cases no 200 reconfigurations of the network can be performed without the requirement of moving previous scheduled time slots. But depending on the initial configuration and on the required changes, it might also be possible to reach over 10000 reconfigurations.  Table 1 lists all simulations performed with parameters and results. All simulations have in common that they use 80 % of the available cycle (1000 of 12500) in average. The first column is the number of the parameter set. The next three columns contain the parameters used in that simulation, the number of removed and inserted time slots per iteration (Y ), the range of time slot sizes and the total number of time slots (X).
The other columns show evaluated results of the 1000 runs per simulation. "Min./max. iterations" is the minimum and maximum number of iterations. A value of ≥ 10000 means that the simulation has stopped after 10000 iterations. "Failed" contains the count of the 1000 runs in total that do not reach the 10000 iterations, "passed" contains the runs that reach 10000 iterations. The last two columns show the iteration when 25 %/90 % of the runs have failed. When less Passed iterations before failure than 25 %/90 % of the simulation runs failed in total, there is no value ("−"). In Fig. 6 this value is the number of iterations when the red line reaches 25 %/90 %. Regarding the simulations P 1 to P 5 , the maximum number of iterations decreases from ≥ 10000 to 4059 when the size of time slots varies more ([249, 251] to [245,255]). So, wider range of time slot sizes can therefore reduce the number of successful reconfigurations. This can be explained by the strategy that the smallest possible gap is used when inserting a time slot. When the largest time slot should be inserted, in P 1 , only two other time slot sizes ([249, 250]) might occupy a previous removed largest time slot, while in P 5 ten smaller time slots ([245, 254]) can occupy a largest time slot. Thus, the chance of finding a matching gap between two time slots is the smaller the more possible time slot sizes are available.
Using smaller but more time slots in P 6 and P 7 , compared to P 1 , fewer runs (11 in P 6 and 1 in P 7 ) failed within 10000 iterations. One explanation for this is that more iterations are needed to fill 20 % of the available space if no smaller gap is found for an inserted time slot, as the time slot is smaller. By doubling the size of the time slots (P 8 compared to P 1 ), the maximum number of iterations decreases from ≥ 10000 to 1465.
Comparing P 1 with P 9 to P 13 , the amount of passed runs varies within a range of [35,96]. In P 10 to P 13 the number of iterations till 25 % percent of the runs fail is higher than P 1 . So, increasing the number of changed time slots per reconfiguration (Y ) from one to two firstly causes more failures. From 3 on (P 10 to P 13 ) more reconfigurations are possible before 25 % of the runs failed. Two effects might cause this behavior. At first, the first iteration may remove only time slots of the minimum size and only inserts slots with a greater size so that all inserted slots must be placed at the end of the previously inserted time slots. When this is done for more slots simultaneously, then more gaps are in the cycle and the buffer at the end becomes smaller more quickly. Otherwise, the more time slots are reconfigured, the more likely it is that the size of some removed time slots will match that of some time slots to be inserted. In case of 40 reconfigured time slots per iteration, all time slots are removed and inserted again, so there will be no fragmentation at all.
In summary, the simulation of a single wire between two switches identifies the following behaviors: -The greater the amount of sizes, the lower the average number of successful reconfigurations. -Smaller time slots enable more reconfigurations than larger time slots.
-The reconfiguration of multiple time slots at once has a positive effect on the number of reconfigurations, after a deterioration at the beginning.

Summary
In a reconfigurable production based on CPPS, new network architectures with larger RTPNs are used. The increased number of participants and the common reconfiguration of the production lead to an increased number of reconfigurations of the RTPN in operation, whereby parts of the configuration must be exchanged without affecting the existing communication. If real-time communication is implemented as a cyclic data transport, in which fixed time slots are reserved (e.g. using IEEE802.1Qbv), the reconfiguration cannot move existing time slots. This can lead to fragmentation when time slots are added and deleted during reconfiguration. Within this paper a simulation has been used to check how often a network can be reconfigured for specific sets of parameters. Depending on the parameters, a RTPN can only be reconfigured a few hundred times before gaps between existing time slots prevent a successful reconfiguration.

Further work
On hard disks, the problem of fragmentation has been solved by running defragmentation operations that move existing files on the disk. To solve the problem of fragmentation in RTPNs equivalent concepts need to be developed and methods for modifying existing time slots during a reconfiguration are required. One solution is that some applications can be customized to tolerate scheduling the time slot to a new position during runtime within a certain range. In future publications we will try to move existing time slots repetitively only a tiny bit (e.g. only several nanoseconds) over time. This should not interrupt the existing communication and enables drifting of existing time slots to reduce gaps. When this is done during reconfiguration larger gaps for inserted time slots should be achieved in the progress.
Open Access This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.