Technical Debt as indicator for weaknesses in engineering of automated production systems

The concept of Technical Debt describes a situation in which a technical compromise is made despite better knowledge. The survey presented delivers insights on Technical Debt in 48 German companies supplying automated production systems. The participating companies do have some immediate benefits from taking Technical Debt under time pressure, but encounter a significant higher long-term additional effort to recover from technical debt. However, awareness for Technical Debt at these companies is low. Therefore, the automated production system manufacturers need to keep a closer eye on expenditure for Technical Debt. The developed survey can be used as a self-assessment method for other companies to compare their results with the average results from this survey.


Introduction and motivation
Technical Debt (TD) describes a situation in which a technical compromise is made, e.g., delivering not-quite-right code in order to meet an urgent deadline [1]. The technical compromise chosen can yield a short-term benefit (TD benefit) but may cause a long-term negative impact (TD interest) on the system quality or the productivity of engineers [2]. The concept of TD can be transferred to the development of automated Production Systems (aPS), which are used nowadays to create products in various sectors, including automated packaging, pharmaceutical production or food processing. aPS are specific classes of mechatronic systems. Typically, aPS are designed by engineers from the three disciplines mechanical, electrical and software. Other disciplines such as hydraulic, pneumatic, sensor or drive technology may also get involved whenever needed. There are strong interdependencies between all those disciplines. Analysing TD in the software discipline at one aPS company, Besker et al. [3] suggested to include mechanical and electrical disciplines additionally to software.
A survey is conducted to address uncovered aspects in prior work [4] such as cost, as well as consciousness of TD in the aPS domain. The study considers the views from management and specialists as well as different types of aPS manufacturers including inputs from software, electrical, and mechanical hardware personnel.
The main contribution of this paper is the first quantitative survey to determine the state-of-the-practice regarding TD in the aPS domain. The paper thus provides a sound basis for TD management in order to reduce costs in engineering and manufacturing. cost overruns in many software development projects [6]. TD might have influences on the planned, reported, and actual product delivery date of a project [7][8][9], profitability of the organization [10][11][12]. However, sometimes the software engineers request investments to remove TD, but after inquiring about their business value, the executives might decline [13]. In addition, the individuals choosing to incur TD (e.g., designers) could be different from those responsible for recovering from the debt (e.g., maintenance staff) [14] [15].
In the domain of software engineering for embedded systems, which is more similar to the aPS domain, Martini et al. [16] investigated reasons for TD. They identified, e.g., Time pressure, Lack of knowledge and Parallel development. However, Parallel development as reason for TD did not yet cover the factor of cross-disciplinarity. For a specific case, Martini et al. [17] reported that improving modularity can reduce architectural TD in the software.
In the aPS domain, Besker et al. [3] studied the work of software developers at one aPS company in Scandinavia and identified that they spend "… quite a lot of resources in paying the interest on Technical Debt, on average 32% of the development time". The estimated amount of TD was confirmed by managers. Vogel-Heuser et al. [18] identified that, on average, plant manufacturers have larger software projects compared to the projects of machine manufacturers. Taking this difference into account, the ways of coping with TD should be analysed, depending on the company type -plant manufacturer or machine manufacturer (RQ1). Furthermore, it is unclear how project characteristics affect TD and the awareness for TD (RQ2). Motived by the interdisciplinarity of aPS development [14], it is necessary to study which discipline and phase of the life cycle are affected by decisions of taking TD (RQ3). In prior work [4], some important aspects such as cost saved, long-term additional cost and consciousness of TD were not covered. In addition, it is important to analyse how the management and specialists rank the amount of TD with regard to the disciplines involved and the causes of TD (RQ4).
The survey by Schuh et al. [19] revealed some major challenges for manufacturing companies. In another survey, also with German companies from the manufacturing industry, Schuh et al. [20] identified some factors contributing to the performance of the systems developed. However, TD aspects have not yet been addressed, since those two surveys focused either on upcoming technologies [19] or on information exchange between service and development [20]. According to Vogel-Heuser et al. [21], maintainability is a prerequisite for the evolution of aPS, which have an especially long lifetime. In order to improve maintainability, high modularity of aPS software is a critical factor [18]. Being good at modularity would enable reusability and consequently higher efficiency in development as well as higher software quality. However, insufficient management of TD may destroy modularity. Unfortunately, in available surveys, TD was either not considered [19] [20] [21] or TD was only analysed regarding modularity of a specific software at one company close to the aPS domain [17]. As far as we are aware, no other studies have been undertaken to explore reusability, modularity, and TD in different disciplines of the aPS domain. Thus, the effect of reusability and modularity on TD in different disciplines of aPS companies should be analysed (RQ5).
In summary, the aPS domain lacks a quantitative study regarding TD. Moreover, the study shall consider the compounds of TD occurrence such as (1) project types/scopes; (2) types of aPS manufacturers; and (3) the perspectives of management and specialists. This research aims to fill this gap in order to gain broader knowledge about TD and the state of the practice in the aPS domain. Based on that, future work can develop proper TD management activities while considering multiple disciplines. Thus, this work forms the basis for a potential increase in cost effectiveness in the aPS domain.

Research questions and hypotheses
Following the idea of Li et al. [2], we identify typical settings, in which TD occurs and which constraints, situations, and people or disciplines are involved. The in-depth analysis explores whether specific types of companies or projects are more in danger to encounter TD than others. We also investigate the effect of TD on various disciplines. Saved cost and disciplines having the largest benefit from taking TD are found. Subsequent to the findings from Vogel-Heuser et al. [18], the study includes TD on different levels of modularity and reusability. It is assumed that a high level of modularity or reusability leads to low TD.
Twelve hypotheses were formulated for five research questions in different contexts (e.g., company/project to discipline).
At company/project level, TD can be expected to affect larger projects more, which prevail in plant manufacturing. This is because larger projects are more complex, less transparent, and harder to manage. Nowadays, to serve diverse customer wishes, companies may need to work on a wide range of project scopes. However, typically, plant manufacturer projects are larger in scope than those of machine manufacturers (H1.1). This is expected to lead to different TD awareness levels for plant manufacturers and machine manufacturers (H1.2). As Besker et al. [3] pointed out, significant additional cost is required due to insufficient attention and management of TD. However, independent of the project's kind, it is unclear what shortterm costs are saved, which long-term additional cost are caused, and what the awareness level is. The general projects, which are not adapted to customer specific requirements, require solutions that are more general, in order to satisfy all customers' requirements. Hence, different kinds of projects might differ in TD. On the one hand, general projects might have less TD benefit (H2.1) and more TD awareness (H2.2). On the other hand, customer specific projects suffer more from TD interest (H2.3). In contrast to Besker et al. [3], who conducted their survey at one company, this survey studies 48 companies.
At the discipline level, besides the complex dependencies, the start time of each discipline on the engineering timeline is different [14]. Usually, the development starts with the mechanical engineering discipline, which creates the construction plan of the mechanical parts. Based on this construction plan and component lists of sensors, actuators and valves, the electrical engineers design the electrical system. This includes the corresponding circuit diagram, the connection of sensors and actuators to the programmable logic controller and the task of distributing power. Documents from both disciplines are then forwarded to the software engineers, who use them to develop the control software for the soft or hardware programmable logic controllers [14]. Due to the sequence, the departments responsible for TD might choose sub-optimal solutions, which TD interest is induced to other departments. Thus, the departments responsible for TD are affected less by their own decisions (H3.1). In case one department has lots of influence (e.g. is larger than the others) in a project, it might be able to force TD on other departments (H3.2). Up to now, it is unclear how management and specialists perceive the amount of TD benefit and interest at disciplines (H4.1). In different interviews, software engineering is the often blamed department for not finishing their task and that we want to study with this survey whether software is the most benefited as well as the most affected discipline (H4.2). Furthermore, from the views of management and specialists, Time pressure is the most common reason for TD (H4.3). Besides the reasons found in Martini et al. [16], the research would include the reasons relating to cross-disciplinary development such as equipment unavailability or missing transdisciplinary.
At different disciplines, there might be different experience in development methods and achieve different modularity and reusability level (H5.1) in order to cope with TD. Hence, different modularity and reusability levels might lead to different levels of TD (H5.2).

Method and threats to validity
In this section, the method is described and potential threats to validity are reported.

Method
A survey was conducted with German machine and plant manufacturers from June until October 2017. The survey was distributed by an independent publishing service (Vogel Business Media GmbH & Co. KG). Eighty complete responses were collected from 48 companies. The participating companies operate in various markets such as water treatment, medicine, automotive, printing, et cetera. The percentages of participants involved in electrical, software, and mechanical disciplines are 69, 49 and 17% respectively. Details of the questions used in this study are available online [22]. The originally German questionnaire was translated for this paper. Hereafter, Q#[number] denotes a question, which has the according ordinate number in the translation. Based on the discussions between authors, the answers of each question were ranked from zero to five (maximum score), where applicable. For example, the scores at Q#2.6 (short-term cost saved) are zero (for "0%" answer), 1.25 (for "1-15%" answer), 2.5 (for "16-30%" answer), 3.75 (for "31-60%" answer), 5 (for "> 60%" answer). Another example is the scoring for Q#2.9 (TD awareness), which ranges from zero (for "No, not at all"), to 1.25 (for "In selected departments"), to 2.5 (for "At the project lead"), to 3.75 (for "Within the management") to 5 (for "In the whole company"). To ensure validity of the results, "could not determine" or "not applicable" (n.a.) answers were not scored and excluded from the analysis.
In this paper, besides the three disciplines mentioned (i.e. mechanical, electrical and software parts), the studied systems also include hydraulic, pneumatic, sensor or drive technology. Hereafter, the term mechanics includes mechanical, hydraulic and pneumatic disciplines and the term electrical includes electrical, sensor and drive technology.
Besides TD, the survey contains questions aimed at other aspects such as tools for version/variant management or maturity in software exchange. As the scope in this paper is limited to TD, only questions related to the TD perspective are considered. In total, thirteen questions provide valuable information in the context of TD: one question about the profile of participants (Q#1.4), four questions about characteristics of projects (Q#1.6, Q#1.7, Q#1.8, Q#1.14), one question about reusability (Q#1.15), one question about modularity (Q#2.5) and six questions about TD itself (Q#2.6, Q#2.6.1, Q#2.7, Q#2.8, Q#2.8.1 and Q#2.9). To answer a research question, it might be necessary to select and combine the results of individual questions. For example, Q#1.7, Q#1.14 and Q#2.9 are used to answer RQ1. A correlation coefficient analysis was performed with MATLAB in order to assess the strength of relations between the scores in TD, modularity or reusability questions.

Threats to validity
First, there might be a bias when deriving the research questions and hypotheses. To reduce this bias, the research questions follow the ideas and suggestions from recent publications as well as feedback from discussions and workshops with experts from industry.
Secondly, there might be a bias when developing questions and their answers in the survey, especially questions related to the TD perspective, since the concept of TD is not yet well known in the aPS domain. To mitigate this threat, two interviews were conducted to test the survey with experts who already had some knowledge about TD and who are working with aPS. These interviewees were a developer lead and a quality assurance engineer from two different companies. To reduce the time to fill in the survey, the terms used in six questions about TD were carefully revised so that the participants could provide the answers quickly. For example, the term "long term additional effort" was selected instead of "effort to pay TD interest". The term TD only appears at the last question (Q#2.9) for TD aspect. In Q#2.9, textual options are preferred to score/percentage options since the textual ones can describe the TD consciousness level better.
Third, the participants and their companies were not selected by the authors, but by an independent business media partner. Thus, we do not know the company names.
There is a threat that if participants did not provide exact answers, we will not be able to figure this out. Nevertheless, diversification of the responses can be reached as the survey was sent to a broad range of plant and machine manufacturers, which are in the network of the partner. In addition, the method using such a survey has been proven to show reasonable results as it has been compared in prior studies to interview results.
Fourth, during the analysis phase, the survey results were analysed independently by various employees of our institute, were consolidated, and discussed with domain experts in order to reduce errors in the results' interpretation.
In Sect. 5.6, the validity of each finding is made clear by use of a traffic light indicator.

Results
This section presents results of the research questions individually and closes with a summary of our findings and an assessment of the findings' validity.

How do plant manufacturers and machine manufacturers cope with Technical Debt? (Research Question 1)
The amount of plant engineering, special purpose machinery or serial machinery projects is asked in Q#1.7 as a pre-processing step. If a participant could not determine the amount, the response is excluded from analysis. Machine manufacturers create individual machines. Plant manufacturers build plants, which include individual machines. Companies were categorized as machine manufacturers or plant manufacturers according to the majority of their projects. Question Q#1.14 addresses the scopes of the primarily created projects (see Fig. 1). As can be seen in Fig. 1, only a miniscule number of plant manufactures provides Single modules (2) and a small number produces Single modules (2) and Parts of a machine (3). This means, that several plant manufactures also offer parts of machines. Unsurprisingly, the typical scope of plant manufacturers' projects (4.22) is bigger than the mean scope of machine manufacturers' projects (3.29) and the upper quartile of machine manufactures creates Complete machine (4). We conclude that H1.1 is true.
The different scopes of projects might lead to different TD attention because managing large scope projects is more complex. It can be observed that plant manufacturers score better TD awareness level than machine manufacturers (mean 2.50 > mean 1.12) (Fig. 2). H1.2 is true. Nevertheless, TD at both plant manufacturers and machine manufacturers might have been a good choice because values for shortterm cost savings are better than the values of long-term additional cost (plant manufacturers: 2.14 > 1.94; machine manufacturers: 2.60 > 1.82).

How do project characteristics affect Technical Debt and the awareness for Technical Debt? (Research Question 2)
The amount of specific work (e.g. design of a new machine or adaptation of an existing one to a specific customer wish) might vary in different project kinds. aPS from Projects individually constructed at customer's option (i.e. customer specific projects) are designed individually and developed for specific customer requests. aPS from Projects not adapted at customer's option (i.e. not-adapted projects) are designed and developed for multiple customers. The scope of Projects adapted at customer's option (i.e. partially adapted projects) stays in between the two above project kinds. The shares of the different kinds of projects in the total amount of projects are obtained at question Q#1.6. The shares are used to identify the project kind, which has majority at the company of each participant. Figure 3 presents benefit, interest as well as attention of TD at different project kinds. Regarding TD benefit, customer specific projects (2.86) and partially adapted projects (2.88) have better scores than general (not-adapted) projects (2.57). H2.1 is partially true as the score gap is small. One might expect that the customer specific ones score much more on TD benefit, because they allow less standardization. However, the result is not exactly as hypothesized.
Customer specific projects and partially adapted projects have quite low TD awareness levels (1.48 and 1.59 point in respectively). Not-adapted projects have higher TD awareness (2.5 point) compared to the two kinds of projects above. H2.2 is true. It could be explained that customer specific projects would require solutions, which are more general for all customers and, thus, might affect TD recovery strategies and levels of consciousness for TD.
For TD interest, customer specific projects (1.60) have the lowest score compared to general projects (1.97) and partially adapted projects (2.02). Hence, H2.3 is true. It seems that there is a significant amount of TD at customer specific projects.

Which discipline and which phase of the life cycle are affected by decisions of taking Technical Debt? (Research Question 3)
RQ3 analyses whether the departments responsible for TD are not affected by their own decisions (inverse relation between Q#2.6.1 and Q#2.8.1). Question Q#2.6.1 asked the participants, which discipline often takes TD benefit (i.e. effort is saved) and Q#2.8.1 asks for the discipline which has to pay the respective TD interest (i.e. long-term additional effort). Question Q#1.8 enquires the number of engineers from each discipline that are involved in an average project. The number of engineers is used to check whether a discipline has a majority in the project. The distribution of TD interest when one department takes TD benefit is illustrated in Fig. 4. When the discipline mechanics takes TD benefit, 89% TD interest occurs at mechanics itself (Fig. 4). The majority of employees in 50% of these projects are mechanical engineers. The remaining TD interest (11%) has to be paid by the software discipline. Mechanical engineers are majority in none of the projects, which put TD interest on the software discipline. Overall, mechanics is affected only by its own decisions and mechanical engineers are not felt to force TD on others. A similar situation occurs in the software discipline.
An interesting result is collected regarding the electrical discipline. When the electrical discipline takes TD benefit, 43% of TD interest is shifted to the software discipline and electrical engineers form the majority in 33% of these projects (cf. (4), Fig. 4). 43% of TD interest occurs in electrics and electrical engineers are the majority in 66% of these projects (cf. (3), Fig. 4). 14% of TD interest occurs at mechanics and electrical engineers are the majority in none of these projects (cf. (5), Fig. 4). Overall, the electrical discipline is only partially affected by its own decisions. When electrical engineers form the majority in a project, sometimes (33% of projects which electrical engineers are majority) they force TD interest on the software discipline. The electrical discipline causes significant TD interest on software discipline. In conclusion, H3.1 is partially true.
In summary, mechanics, and software take both TD benefit and TD interest. The electrical discipline takes TD benefit and causes significant TD interest on the software discipline. When mechanics or software engineers form the majority in a project, they do not force TD on others. When electrical engineers are the majority in a project, sometimes they do force TD on software engineers. In addition, as aPS software development highly depends on the electrical development (and mechanics development as well), it could be the main reason that the software discipline often takes TD interest when the electrical discipline takes TD benefit.

How do management and specialists rank the amount of Technical Debt at disciplines and the causes of Technical Debt? (Research Question 4)
Question Q#1.4 is used to classify the respondents. Nearly half of respondents (45%) are in leadership or management positions (e.g., director, head or group leader). The percentage of respondents as specialist is 40%. The remaining respondents (15%) did not reveal their positions. Thus, the results can roughly be divided into two views: management and specialists. In this part of the analysis, only the responses from the respondents who revealed their positions are counted. Question Q#2.6 studies the average effort that can be saved in the short term by implementing a sub-optimal solution in comparison to implementing the most reasonable one. The most votes are for 16-30%, which has 39% votes from management and 38% votes from specialist (Fig. 5).
A wise Gaussian distribution can be observed that management and specialist have similar rates for the benefit of TD, except at > 60% which has more votes from management (17%) than from specialist (9%).
Question Q#2.8 studies the long-term additional effort caused by the implementation of a sub-optimal solution compared to the implementation of the most reasonable solution. Both management and specialist have high number of votes at large TD interest (31-60% and > 60%) (Fig. 6). H4.1 is true. It should be noted that specialists have higher ratings (focusing on 16-30%, 31-60% and > 60%) than management.
The result of question Q#2.6.1 indicates disciplines with the biggest potentials for savings. Both management (31%) and specialists (34%) agree on the software discipline (cp. (1) in Fig. 7). There is still a significant gap between the votes of management and specialists at each discipline. For example, project lead/sales has 19% of management and 25% votes from specialists, which result in a delta of 6. The average delta is 9 and yet there is a significant number of specialists (23%) who could not determine the discipline. Regarding the discipline that has to pay the most TD interest (Q#2.8.1), the software discipline gets the most votes from both management (38%) and specialists (52%) (Fig. 8). Comparing the results of mechanics, electrics and software, the electrical discipline has the least votes (14% from management and 0% from specialist). Overall, the software discipline has the most votes for TD benefit (Fig. 6) as well as the most votes for TD interest (Fig. 8), from both management and specialists. Overall, H4.2 is true.
Question Q#2.7 studies the reasons for the choice of a sub-optimal solution instead of the most reasonable one. One participant can vote for multiple reasons as several causes of TD might apply at the same time. Overall, both management and specialists vote similarly for all reasons (Fig. 9). The highest votes at Time pressure from both management (83%) as well as specialists (88%) confirm the finding of the work from Martini et al. [16] in the embedded software engineering domain, where pressure to on time delivery is a significant TD trigger, H4.3 is true.

How do different reusability and modularity levels affect Technical Debt? (Research Question 5)
Different reusability levels are obtained from question Q#1.15. Scores of modularity (Q#2.5) vary from 2 (Project specific share), 3 (Libraries/templates), 5 (Both) and 0 (Neither nor). Could not determine responses are excluded from the analysis. Figure 10 presents the modularity at different reusability levels. It can be observed that at each discipline, the different ratings on modularity lead to different reusability levels. Hence, H5.1 is true. It can be explained that, most of the time, system needs to be well modularized in order to be good at reusability. In addition, a medium correlation (r = 0.693) is identified between modularity of the software discipline and modularity in electrics/electronics. TD benefit, interest, and consciousness at different reusability levels are illustrated in Fig. 11. An interesting result is that companies, which score low reusability also score low TD interest (1.07). Thus, different levels of reusability might lead to different levels of TD. H5.1 is partially true since TD benefit and attention are similar at different reusability levels. One explanation could be that TD interest is low with low proficiency to reuse, not because the system is not reusable, but because the company does not reuse it and therefore they do not pay long-term interest on reused system with TD.

Summary of Findings and Their Validity
The research questions, related hypotheses, and validity are summarized in Fig. 12. The result from H1.1 confirms the result in Vogel-Heuser et al. [18], with a larger group of companies. Although larger project scopes lead to higher awareness for TD (H1.2), it might be argued that larger projects might have developed better TD recovery strategies; however, the TD recovery strategies remain unclear at these companies and should be checked in future research.
Regarding H3.2, besides majority of engineers, the influences "power" might be enrooted in other factors such as prestige or management support. It would be interesting to check the influences from those factors in future research, too. As aPS software development often starts after mechanics and electrical development (due to dependencies), "quick-and-dirty" solutions might be implemented in software, in case the project deadline is close.
As a medium correlation between modularity of electrical and software disciplines is identified, it seems that mechanical engineering just does not have as many dependencies with electrics as electrics have with software engineering. Furthermore, bad mechanical engineering decisions might not be perceived as TD, but rather as a boundary condition.
Regarding H4.1 to H4.3, management and specialists rank the amount of TD quite similar. This confirms the findings of Besker et al. [3]. For H4.2, although the software discipline has the highest votes from both management (30.56%) and specialists (34.38%) for disciplines taking TD benefit, it is assumed that this perception might be from bad bug fixes from the software engineers. It could be also the case because most people crossing this box are not from the software department. Therefore, the reasons for this perception should be taken into account in future research.
Regarding H5.2, with similar TD consciousness, the factor modularity might play an important role. Mature modularity solutions lead to higher reusability (H5.1), which in turn reduces TD interest (mean 2.03 and 2.22) in comparison to those cases with low reusability (mean 1.07).

Conclusion and outlook
This study uncovered that the decision of taking TD benefit by the electrical discipline causes significant TD interest (43%) in the software discipline. The results reveal important details to leverage the transparency of unscheduled cost between different disciplines in a TD perspective. TD has a significant impact on the overall cost for aPS. However, TD awareness at these companies is low. Therefore, both plant manufacturers and machine manufacturers should monitor costs for TD more closely. The developed survey can be used as a self-assessment method for other companies. Thereby, the average results from this study can serve as a benchmark.
Future work should study TD recovery strategies, which aPS manufacturers use to cope with cross-disciplinary TD since no existing TD management approach has deeply discussed or investigated a utilization in actual cases [2], and furthermore, in a cross-disciplinary environment such as aPS development.