Technical Debt as indicator for weaknesses in engineering of automated production systems
- 123 Downloads
The concept of Technical Debt describes a situation in which a technical compromise is made despite better knowledge. The survey presented delivers insights on Technical Debt in 48 German companies supplying automated production systems. The participating companies do have some immediate benefits from taking Technical Debt under time pressure, but encounter a significant higher long-term additional effort to recover from technical debt. However, awareness for Technical Debt at these companies is low. Therefore, the automated production system manufacturers need to keep a closer eye on expenditure for Technical Debt. The developed survey can be used as a self-assessment method for other companies to compare their results with the average results from this survey.
KeywordsTechnical debt Cross-disciplinary engineering Automated production systems Management
1 Introduction and motivation
Technical Debt (TD) describes a situation in which a technical compromise is made, e.g., delivering not-quite-right code in order to meet an urgent deadline . The technical compromise chosen can yield a short-term benefit (TD benefit) but may cause a long-term negative impact (TD interest) on the system quality or the productivity of engineers . The concept of TD can be transferred to the development of automated Production Systems (aPS), which are used nowadays to create products in various sectors, including automated packaging, pharmaceutical production or food processing. aPS are specific classes of mechatronic systems. Typically, aPS are designed by engineers from the three disciplines mechanical, electrical and software. Other disciplines such as hydraulic, pneumatic, sensor or drive technology may also get involved whenever needed. There are strong interdependencies between all those disciplines. Analysing TD in the software discipline at one aPS company, Besker et al.  suggested to include mechanical and electrical disciplines additionally to software.
A survey is conducted to address uncovered aspects in prior work  such as cost, as well as consciousness of TD in the aPS domain. The study considers the views from management and specialists as well as different types of aPS manufacturers including inputs from software, electrical, and mechanical hardware personnel.
The main contribution of this paper is the first quantitative survey to determine the state-of-the-practice regarding TD in the aPS domain. The paper thus provides a sound basis for TD management in order to reduce costs in engineering and manufacturing.
2 Technical Debt in automated production systems
A classification of TD types according to different causes for general software systems was presented by Li et al. . Some significant causes are improper architecture, test, documentation, and, most studied, code TD. The work of Avgeriou et al.  indicates that TD always relates to cost. Failure to monitor TD has resulted in unexpectedly large cost overruns in many software development projects . TD might have influences on the planned, reported, and actual product delivery date of a project [7, 8, 9], profitability of the organization [10, 11, 12]. However, sometimes the software engineers request investments to remove TD, but after inquiring about their business value, the executives might decline . In addition, the individuals choosing to incur TD (e.g., designers) could be different from those responsible for recovering from the debt (e.g., maintenance staff)  .
In the domain of software engineering for embedded systems, which is more similar to the aPS domain, Martini et al.  investigated reasons for TD. They identified, e.g., Time pressure, Lack of knowledge and Parallel development. However, Parallel development as reason for TD did not yet cover the factor of cross-disciplinarity. For a specific case, Martini et al.  reported that improving modularity can reduce architectural TD in the software.
In the aPS domain, Besker et al.  studied the work of software developers at one aPS company in Scandinavia and identified that they spend “… quite a lot of resources in paying the interest on Technical Debt, on average 32% of the development time”. The estimated amount of TD was confirmed by managers. Vogel-Heuser et al.  identified that, on average, plant manufacturers have larger software projects compared to the projects of machine manufacturers. Taking this difference into account, the ways of coping with TD should be analysed, depending on the company type – plant manufacturer or machine manufacturer (RQ1). Furthermore, it is unclear how project characteristics affect TD and the awareness for TD (RQ2). Motived by the interdisciplinarity of aPS development , it is necessary to study which discipline and phase of the life cycle are affected by decisions of taking TD (RQ3). In prior work , some important aspects such as cost saved, long-term additional cost and consciousness of TD were not covered. In addition, it is important to analyse how the management and specialists rank the amount of TD with regard to the disciplines involved and the causes of TD (RQ4).
The survey by Schuh et al.  revealed some major challenges for manufacturing companies. In another survey, also with German companies from the manufacturing industry, Schuh et al.  identified some factors contributing to the performance of the systems developed. However, TD aspects have not yet been addressed, since those two surveys focused either on upcoming technologies  or on information exchange between service and development . According to Vogel-Heuser et al. , maintainability is a prerequisite for the evolution of aPS, which have an especially long lifetime. In order to improve maintainability, high modularity of aPS software is a critical factor . Being good at modularity would enable reusability and consequently higher efficiency in development as well as higher software quality. However, insufficient management of TD may destroy modularity. Unfortunately, in available surveys, TD was either not considered    or TD was only analysed regarding modularity of a specific software at one company close to the aPS domain . As far as we are aware, no other studies have been undertaken to explore reusability, modularity, and TD in different disciplines of the aPS domain. Thus, the effect of reusability and modularity on TD in different disciplines of aPS companies should be analysed (RQ5).
In summary, the aPS domain lacks a quantitative study regarding TD. Moreover, the study shall consider the compounds of TD occurrence such as (1) project types/scopes; (2) types of aPS manufacturers; and (3) the perspectives of management and specialists. This research aims to fill this gap in order to gain broader knowledge about TD and the state of the practice in the aPS domain. Based on that, future work can develop proper TD management activities while considering multiple disciplines. Thus, this work forms the basis for a potential increase in cost effectiveness in the aPS domain.
3 Research questions and hypotheses
Following the idea of Li et al. , we identify typical settings, in which TD occurs and which constraints, situations, and people or disciplines are involved. The in-depth analysis explores whether specific types of companies or projects are more in danger to encounter TD than others. We also investigate the effect of TD on various disciplines. Saved cost and disciplines having the largest benefit from taking TD are found. Subsequent to the findings from Vogel-Heuser et al. , the study includes TD on different levels of modularity and reusability. It is assumed that a high level of modularity or reusability leads to low TD.
Twelve hypotheses were formulated for five research questions in different contexts (e.g., company/project to discipline).
At company/project level, TD can be expected to affect larger projects more, which prevail in plant manufacturing. This is because larger projects are more complex, less transparent, and harder to manage. Nowadays, to serve diverse customer wishes, companies may need to work on a wide range of project scopes. However, typically, plant manufacturer projects are larger in scope than those of machine manufacturers (H1.1). This is expected to lead to different TD awareness levels for plant manufacturers and machine manufacturers (H1.2). As Besker et al.  pointed out, significant additional cost is required due to insufficient attention and management of TD. However, independent of the project’s kind, it is unclear what short-term costs are saved, which long-term additional cost are caused, and what the awareness level is. The general projects, which are not adapted to customer specific requirements, require solutions that are more general, in order to satisfy all customers’ requirements. Hence, different kinds of projects might differ in TD. On the one hand, general projects might have less TD benefit (H2.1) and more TD awareness (H2.2). On the other hand, customer specific projects suffer more from TD interest (H2.3). In contrast to Besker et al. , who conducted their survey at one company, this survey studies 48 companies.
At the discipline level, besides the complex dependencies, the start time of each discipline on the engineering timeline is different . Usually, the development starts with the mechanical engineering discipline, which creates the construction plan of the mechanical parts. Based on this construction plan and component lists of sensors, actuators and valves, the electrical engineers design the electrical system. This includes the corresponding circuit diagram, the connection of sensors and actuators to the programmable logic controller and the task of distributing power. Documents from both disciplines are then forwarded to the software engineers, who use them to develop the control software for the soft or hardware programmable logic controllers . Due to the sequence, the departments responsible for TD might choose sub-optimal solutions, which TD interest is induced to other departments. Thus, the departments responsible for TD are affected less by their own decisions (H3.1). In case one department has lots of influence (e.g. is larger than the others) in a project, it might be able to force TD on other departments (H3.2). Up to now, it is unclear how management and specialists perceive the amount of TD benefit and interest at disciplines (H4.1). In different interviews, software engineering is the often blamed department for not finishing their task and that we want to study with this survey whether software is the most benefited as well as the most affected discipline (H4.2). Furthermore, from the views of management and specialists, Time pressure is the most common reason for TD (H4.3). Besides the reasons found in Martini et al. , the research would include the reasons relating to cross-disciplinary development such as equipment unavailability or missing transdisciplinary.
At different disciplines, there might be different experience in development methods and achieve different modularity and reusability level (H5.1) in order to cope with TD. Hence, different modularity and reusability levels might lead to different levels of TD (H5.2).
4 Method and threats to validity
In this section, the method is described and potential threats to validity are reported.
A survey was conducted with German machine and plant manufacturers from June until October 2017. The survey was distributed by an independent publishing service (Vogel Business Media GmbH & Co. KG). Eighty complete responses were collected from 48 companies. The participating companies operate in various markets such as water treatment, medicine, automotive, printing, et cetera. The percentages of participants involved in electrical, software, and mechanical disciplines are 69, 49 and 17% respectively. Details of the questions used in this study are available online . The originally German questionnaire was translated for this paper. Hereafter, Q#[number] denotes a question, which has the according ordinate number in the translation. Based on the discussions between authors, the answers of each question were ranked from zero to five (maximum score), where applicable. For example, the scores at Q#2.6 (short-term cost saved) are zero (for “0%” answer), 1.25 (for “1–15%” answer), 2.5 (for “16-30%” answer), 3.75 (for “31–60%” answer), 5 (for “> 60%” answer). Another example is the scoring for Q#2.9 (TD awareness), which ranges from zero (for “No, not at all”), to 1.25 (for “In selected departments”), to 2.5 (for “At the project lead”), to 3.75 (for “Within the management”) to 5 (for “In the whole company”). To ensure validity of the results, “could not determine” or “not applicable” (n.a.) answers were not scored and excluded from the analysis.
In this paper, besides the three disciplines mentioned (i.e. mechanical, electrical and software parts), the studied systems also include hydraulic, pneumatic, sensor or drive technology. Hereafter, the term mechanics includes mechanical, hydraulic and pneumatic disciplines and the term electrical includes electrical, sensor and drive technology.
Besides TD, the survey contains questions aimed at other aspects such as tools for version/variant management or maturity in software exchange. As the scope in this paper is limited to TD, only questions related to the TD perspective are considered. In total, thirteen questions provide valuable information in the context of TD: one question about the profile of participants (Q#1.4), four questions about characteristics of projects (Q#1.6, Q#1.7, Q#1.8, Q#1.14), one question about reusability (Q#1.15), one question about modularity (Q#2.5) and six questions about TD itself (Q#2.6, Q#2.6.1, Q#2.7, Q#2.8, Q#2.8.1 and Q#2.9). To answer a research question, it might be necessary to select and combine the results of individual questions. For example, Q#1.7, Q#1.14 and Q#2.9 are used to answer RQ1. A correlation coefficient analysis was performed with MATLAB in order to assess the strength of relations between the scores in TD, modularity or reusability questions.
5.1 Threats to validity
First, there might be a bias when deriving the research questions and hypotheses. To reduce this bias, the research questions follow the ideas and suggestions from recent publications as well as feedback from discussions and workshops with experts from industry.
Secondly, there might be a bias when developing questions and their answers in the survey, especially questions related to the TD perspective, since the concept of TD is not yet well known in the aPS domain. To mitigate this threat, two interviews were conducted to test the survey with experts who already had some knowledge about TD and who are working with aPS. These interviewees were a developer lead and a quality assurance engineer from two different companies. To reduce the time to fill in the survey, the terms used in six questions about TD were carefully revised so that the participants could provide the answers quickly. For example, the term “long term additional effort” was selected instead of “effort to pay TD interest”. The term TD only appears at the last question (Q#2.9) for TD aspect. In Q#2.9, textual options are preferred to score/percentage options since the textual ones can describe the TD consciousness level better.
Third, the participants and their companies were not selected by the authors, but by an independent business media partner. Thus, we do not know the company names. There is a threat that if participants did not provide exact answers, we will not be able to figure this out. Nevertheless, diversification of the responses can be reached as the survey was sent to a broad range of plant and machine manufacturers, which are in the network of the partner. In addition, the method using such a survey has been proven to show reasonable results as it has been compared in prior studies to interview results.
Fourth, during the analysis phase, the survey results were analysed independently by various employees of our institute, were consolidated, and discussed with domain experts in order to reduce errors in the results’ interpretation.
In Sect. 5.6, the validity of each finding is made clear by use of a traffic light indicator.
This section presents results of the research questions individually and closes with a summary of our findings and an assessment of the findings’ validity.
6.1 How do plant manufacturers and machine manufacturers cope with Technical Debt? (Research Question 1)
6.2 How do project characteristics affect Technical Debt and the awareness for Technical Debt? (Research Question 2)
Regarding TD benefit, customer specific projects (2.86) and partially adapted projects (2.88) have better scores than general (not-adapted) projects (2.57). H2.1 is partially true as the score gap is small. One might expect that the customer specific ones score much more on TD benefit, because they allow less standardization. However, the result is not exactly as hypothesized.
Customer specific projects and partially adapted projects have quite low TD awareness levels (1.48 and 1.59 point in respectively). Not-adapted projects have higher TD awareness (2.5 point) compared to the two kinds of projects above. H2.2 is true. It could be explained that customer specific projects would require solutions, which are more general for all customers and, thus, might affect TD recovery strategies and levels of consciousness for TD.
For TD interest, customer specific projects (1.60) have the lowest score compared to general projects (1.97) and partially adapted projects (2.02). Hence, H2.3 is true. It seems that there is a significant amount of TD at customer specific projects.
6.3 Which discipline and which phase of the life cycle are affected by decisions of taking Technical Debt? (Research Question 3)
When the discipline mechanics takes TD benefit, 89% TD interest occurs at mechanics itself (Fig. 4). The majority of employees in 50% of these projects are mechanical engineers. The remaining TD interest (11%) has to be paid by the software discipline. Mechanical engineers are majority in none of the projects, which put TD interest on the software discipline. Overall, mechanics is affected only by its own decisions and mechanical engineers are not felt to force TD on others. A similar situation occurs in the software discipline.
An interesting result is collected regarding the electrical discipline. When the electrical discipline takes TD benefit, 43% of TD interest is shifted to the software discipline and electrical engineers form the majority in 33% of these projects (cf. (4), Fig. 4). 43% of TD interest occurs in electrics and electrical engineers are the majority in 66% of these projects (cf. (3), Fig. 4). 14% of TD interest occurs at mechanics and electrical engineers are the majority in none of these projects (cf. (5), Fig. 4). Overall, the electrical discipline is only partially affected by its own decisions. When electrical engineers form the majority in a project, sometimes (33% of projects which electrical engineers are majority) they force TD interest on the software discipline. The electrical discipline causes significant TD interest on software discipline. In conclusion, H3.1 is partially true.
In summary, mechanics, and software take both TD benefit and TD interest. The electrical discipline takes TD benefit and causes significant TD interest on the software discipline. When mechanics or software engineers form the majority in a project, they do not force TD on others. When electrical engineers are the majority in a project, sometimes they do force TD on software engineers. In addition, as aPS software development highly depends on the electrical development (and mechanics development as well), it could be the main reason that the software discipline often takes TD interest when the electrical discipline takes TD benefit.
6.4 How do management and specialists rank the amount of Technical Debt at disciplines and the causes of Technical Debt? (Research Question 4)
Question Q#1.4 is used to classify the respondents. Nearly half of respondents (45%) are in leadership or management positions (e.g., director, head or group leader). The percentage of respondents as specialist is 40%. The remaining respondents (15%) did not reveal their positions. Thus, the results can roughly be divided into two views: management and specialists. In this part of the analysis, only the responses from the respondents who revealed their positions are counted.
6.5 How do different reusability and modularity levels affect Technical Debt? (Research Question 5)
6.6 Summary of Findings and Their Validity
Regarding H3.2, besides majority of engineers, the influences “power” might be enrooted in other factors such as prestige or management support. It would be interesting to check the influences from those factors in future research, too. As aPS software development often starts after mechanics and electrical development (due to dependencies), “quick-and-dirty” solutions might be implemented in software, in case the project deadline is close.
As a medium correlation between modularity of electrical and software disciplines is identified, it seems that mechanical engineering just does not have as many dependencies with electrics as electrics have with software engineering. Furthermore, bad mechanical engineering decisions might not be perceived as TD, but rather as a boundary condition.
Regarding H4.1 to H4.3, management and specialists rank the amount of TD quite similar. This confirms the findings of Besker et al. . For H4.2, although the software discipline has the highest votes from both management (30.56%) and specialists (34.38%) for disciplines taking TD benefit, it is assumed that this perception might be from bad bug fixes from the software engineers. It could be also the case because most people crossing this box are not from the software department. Therefore, the reasons for this perception should be taken into account in future research.
Regarding H5.2, with similar TD consciousness, the factor modularity might play an important role. Mature modularity solutions lead to higher reusability (H5.1), which in turn reduces TD interest (mean 2.03 and 2.22) in comparison to those cases with low reusability (mean 1.07).
7 Conclusion and outlook
This study uncovered that the decision of taking TD benefit by the electrical discipline causes significant TD interest (43%) in the software discipline. The results reveal important details to leverage the transparency of unscheduled cost between different disciplines in a TD perspective. TD has a significant impact on the overall cost for aPS. However, TD awareness at these companies is low. Therefore, both plant manufacturers and machine manufacturers should monitor costs for TD more closely. The developed survey can be used as a self-assessment method for other companies. Thereby, the average results from this study can serve as a benchmark.
Future work should study TD recovery strategies, which aPS manufacturers use to cope with cross-disciplinary TD since no existing TD management approach has deeply discussed or investigated a utilization in actual cases , and furthermore, in a cross-disciplinary environment such as aPS development.
The authors thank all companies that have participated in the survey for their contributions. Also, we thank Huaxia Li, Iris Weiß, and Juliane Fischer for their support in this research. Finally, we thank the Vietnam International Education Development and Vietnamese-German University for granting Quang Huan Dong a scholarship under Project 911—“Training lecturers of Doctor’s Degree for universities and colleges for the 2010–2020 period” (Decision No. 771/QD-BGDDT dated 14/03/2017).
- 3.Besker T, Martini A, Bosch J, Tichy M (2017) An investigation of technical debt in automatic production systems. In: Proceedings of the XP2017 Scientific Workshops on XP’17, pp 1–7Google Scholar
- 5.Avgeriou P, Kruchten P, Ozkaya I, Seaman C (2016) Managing Technical Debt in software engineering (Dagstuhl Seminar 16162). Dagstuhl Rep 6(4):110–138Google Scholar
- 6.Guo Y et al. (2011) Tracking technical debt—an exploratory case study. In: 2011 27th IEEE international conference on software maintenance (ICSM), pp 528–531Google Scholar
- 9.Holvitie J, Leppanen V (2015) Examining technical debt accumulation in software implementations. Int J Softw Eng Appl 9(6):109–124Google Scholar
- 12.Snipes W, Robinson B, Guo Y, Seaman C (2012) Defining the decision factors for managing defects: a technical debt perspective. In: 2012 Workshop on managing technical debt (MTD), pp 54–60Google Scholar
- 13.Ozkaya I, Kruchten P, Nord R, Brown N (2012) Second international workshop on managing technical debt. In: Proceeding of the 33rd international conference on software engineering—ICSE’11, p 1212Google Scholar
- 14.Vogel-Heuser B, Rösch S, Martini A, Tichy M (2015) Technical debt in automated production systems. In: 2015 IEEE 7th workshop on managing technical debt (MTD), pp 49–52Google Scholar
- 15.Martini A, Bosch J (2015) The danger of architectural technical debt: contagious debt and vicious circles. In: 2015 12th Working IEEE/IFIP conference on software architecture, pp 1–10Google Scholar
- 17.Martini A, Sikander E, Medlani N (2016) Estimating and quantifying the benefits of refactoring to improve a component modularity: a case study. In: 2016 42th Euromicro conference on software engineering and advanced applications (SEAA), pp 92–99Google Scholar
- 22.Survey on Technical Debt in construction—a lever for more transparency for unscheduled efforts during cooperation. https://mediatum.ub.tum.de/1472174?show_id=1472171&style=full_text. Accessed: 22 Mar 2019 (online)
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.