1 Introduction

The demands of the markets have changed the environment for software companies due to emerging technology and a multitude of factors such as business competitiveness, digital disruptions, and digital reinventions (Gupta et al., 2022). As mentioned by Lomio et al. (2022), instead of creating new features, developers spend more time resolving bugs and modifying the code. Durisic et al., 2011 defend that modern cars have a considerable amount of software and, in their words, the increasing complexity of in-car electronics and the rapidly growing amount of automotive software running on embedded electronic control units, places higher demands on quality assurance for the future. In this way, Farkas (2008) says that the software on embedded control units cannot have quality incorporated after development. Thus, in his words, techniques for defect detection must be established to automatically interrupt development so that an issue may be fixed before the defect continues downstream.

The volume of software developed daily in the automotive sector has made it necessary to improve software quality, particularly in fields like autonomous driving where safety is a top priority. To guarantee software quality and prevent the worst-case situations associated with the automobile sector, particularly with autonomous driving, continuous inspection of software is critical, timely, and essential.

This study was inspired by the big number of failures related to low quality software in the context of the Bosch Car Multimedia DevOps Team working on the Lidar project. Due to the bad quality of the software, it is necessary to review and rewrite the software, so the software’s lifecycle is always going around. This results in costs in terms of money, time and resources and can also damage the company’s reputation, if the deadlines are missed and low-quality software is delivered.

The major goal of this work was to provide a solution for continuous inspection of the software quality, to develop and maintain high-quality software. It is critical to emphasize the following goals to reach the major goal:

  • Study the existing scientific knowledge about the research topic.

  • Define specifications for the support solution.

  • Analyze tools and technologies in the market for continuous inspection of software quality.

  • Implement a proof of concept of the support platform.

  • Implement the solution for continuous inspection;

  • Validate the support solution after the implementation.

2 Literature Review

Many companies are adopting modern continuous software development practices and approaches like continuous integration, continuous delivery, or DevOps, to increase the development speed and the time-to-market. So, Paule et al., (2019) tried to identify which vulnerabilities are present in industry of continuous delivery (CD) pipelines and how they can be detected. After conducting a survey and looking at two CD pipelines from industry, they discovered that the security of the pipelines is not given high importance by development teams. According to them, both CD pipelines under investigation included vulnerabilities with an overall risk severity that ranged from potentially high to medium since the project teams were dependent on the infrastructure of the customers.

Debroy et al., (2018) focused on two challenges: wait times for builds/releases to be queued and completed and the lack of support for tooling, especially from a cross-cloud perspective. They said that by using DevOps principles like orchestration, containerization and cross-team cooperation, they were able to overcome these obstacles and create a lean and reliable CI/CD pipeline that has shown to be effective and appropriate.

According to Shahin et al. (2017), it is critical to systematically analyze and synthesize the methodologies, tools, challenges and practices reported for adopting and implementing continuous practices. The authors defend that their study demonstrates how continuous methods have been effectively used on both greenfield and maintenance projects. According to them, the results demonstrated the growing interest among software engineering researchers in the subject of continuous practices and demonstrated that both greenfield and maintenance projects may benefit from its effective use.

Conforming to Ahmad et al., (2021), software workers look for more information to complete the task at hand while participating in the activities that CI/CD consolidate. According to them, a better understanding of the information requirements of software practitioners has a number of advantages, including maintaining competitiveness, raising awareness of the problems that might prevent a timely release, and developing a visualization tool that can assist practitioners in meeting their information requirements. To determine information needs in CI/CD and developers' information needs, they did a multiple-case holistic research. The authors claim that the data generated by CI/CD can offer important insights into things like team performance, potential bottlenecks, and improvement opportunities.

Using the practice for introducing system test activities in the agile team activities every iteration, Sandu and Salceanu (2019) support the idea that companies may enhance the quality of the delivered products, save costs, become more predictable, shorten the time to market, boost employee’s morale, enhance their reputation and increase customer satisfaction. They offered a methodology for reducing the number of open defects, using agile methodologies. They served as an example of a technique for raising system testing awareness in agile teams, incorporating system test activities within the agile team, to get quick feedback and to solve the defects detected dutring development as soon as feasible. In their words, executing the system test activities inside the agile team significantly improved the delivered products and reduced the development time, effort, and costs, compared to outside the agile team.

Given the significance of software quality, Hamdan and Alramouni (2015) argues that every project should include high-quality software and CI reduces the risks associated with software integration while also enhancing the quality. In this manner, they provided a framework for identifying software quality aspects of the development process when CI approaches are applied to the process of developing software projects. The authors claim that after using the CI, the project required less time for development and fault repair, which significantly reduced the risks associated with software integration process.

Durisic et al., (2011) proposed measures used to quantify the size and complexity of software systems with a focus on the automobile industry to detect possible risk caused by modifications required to the existing systems when talking about new releases. Two metrics to evaluate complexity and connectivity of architectural and software system components in automotive software systems were presented to help assess the impact of changes. They claim that doing so improves product quality and lowers manufacturing costs, and that the steps may be taken early in the development cycle, before sending the suppliers' requests for the creation of certain component needs, to cut down on both time as well as costs.

Still in the automotive industry, Barhate (2015) stated that it is well known that electronic content is growing in automobiles on a daily basis. Many functionalities are related to automobile system safety, which is one of the main concerns for future automobile development. System failure risk is high due to growing technological complexity and software content, so the software must be tested thoroughly to catch nearly all defects. In this approach, the author discusses how the Taguchi technique of test case reduction, prioritizing test execution, and automation serve to make testing successful and shows how a significant number of defects are found quickly. He argues that the findings, which included 20% of flaws found in 5% of test cases within six hours of test execution, proved the effectiveness of test case prioritizing logic.

Similarly, Niæetin et al. (2018) contend that numerous open-source technologies and frameworks may be employed in addition to industry-proven automotive development software to decrease time and costs while still maintaining a high quality of software releases. Specifically, they extend the basic development process by using open-source build automation tools and servers to decrease the build and unit test time and incorporate code-based analysis tools for static and dynamic analysis. In their conclusion, they argue that using servers for continuous integration, it is feasible to shorten the time required to test and incorporate fresh modifications into automotive software. According to them, mistakes that may arise in automotive software may be found and documented using static analysis and dynamic analysis tools. Unit testing frameworks can also be used to confirm the accuracy of software at the unit and integration levels. Therefore, they believe that it has been demonstrated that the strategy is feasible and promising for more affordable development environments that are appropriate for new businesses and start-ups trying to effectively engage in the growing automotive industry.

To increase the quality of coding, Guaman et al., (2017) study performs static analysis on the source code of software applications created in various programming languages. Additionally, using SonarQube, a platform for continuous inspection, it helps the detection of features like the proper application of quality attributes, standards, and best practices of programming that guarantee the correct software development in terms of design and coding. As part of the technical debt pyramid, the authors argue that it is beneficial to employ standards, quality models, and best programming practices to improve maintainability, security, changeability, reliability, and testability.

Software quality assurance is a sub-process that makes sure that generated software conforms with clearly stated or standardized quality criteria, according to De andrade Gomes et al., (2017). They developed a tool for evaluating the quality of source code from this to assist students in improving their source code and, as a result, their programming skills. The suggested tool analyzes student source code using quality reports and gives comments on the student coding. Students must assess their code using the suggested tool after each revision in order to get feedback on the quality of the source code. The tool, named SMaRT, based on a continuous inspection tool, aims to assist students with maintenance responsibilities by letting them assess the quality of the code in their local repositories without compromising the project repository. The authors claim that the outcomes have improved the students' access to high-quality teaching–learning software.

According to Armenise (2015), the ongoing transformation of the software industry, which calls for quick response to changes and, as a result, cross-team communication and collaboration, prompted Jenkins to expand its functionalities to act as an orchestrator for all roles involved in the product lifecycle. In order to facilitate the construction of complex workflows, provide traceability, shorten time-to-market, and boost productivity, controlling the whole continuous delivery pipeline requires the deployment of additional features in Jenkins. In this way, he illustrates how Jenkins changed from being a purely CI platform to a CD one, where the build, release, and delivery processes are all automated. Jenkins serves as the orchestrator tool for all the teams and roles involved in the software lifecycle in this case, enabling close collaboration between the development, quality assurance, and operations teams.

3 Requirements

To design the solution for continuous inspection, some stakeholders impose requirements for the solution to guarantee that the implemented support solution meets all the needs. In this way, the requirements are presented in Table 1. For each requirement was assigned an id, the category, and a description. The category is divided into non-functional, functional and domain.

Table 1 Requirements
Full size table

4 Architecture

Sonarqube and Jenkins are two tools that can be used together to manage and automate the software development process. Sonarqube is a self-managed, automatic code review tool that systematically helps developers deliver clean code, that integrates into the existing workflow and detects issues in code to help perform continuous code inspections of projects. In turn, Jenkins is an open-source automation server which enables developers to reliably build, test, and deploy their software. In addition, SonarLint was used to integrate the existing ide workflow and helps commit with confidence, by detecting coding issues in real-time and getting clear guidance on how to fix them. Then, a tool named CIAnalyzer Tool was created to keep the communication between Sonarqube and Jenkins or another external tool, acting as the middleware.

Figure 1 shows the architecture of the support solution, where Jenkins is the midway point control for the process, working as an orchestrator. The process starts when the developer produces software locally and, with the help of SonarLint, connected to the SonarQube Server and the specific project, a local and real-time analysis is performed. Then, the developer delivers the code to the team’s repository that triggers a build in Jenkins. In turn, Jenkins pushes the analysis in CIAnalyzer Tool that pushes a SonarQube analysis, which at the end returns the results and then the CIAnalyzer Tool constructs the report and returns it to Jenkins. Also, the Json analysis happens in CIAnalyzer Tool. Finally, Jenkins returns to the developer the build status, which depends on the quality gate result, as well as the link for the results and analysis report, which contains information about the analysis of that software. There is another way to trigger the build, using a timer, that happens once per night, and the process is the same.

Fig. 1
figure 1

Architecture

Full size image

5 Analysis Types

It was defined that will exist two analysis types: full or lightweight. Each one has different characteristics, as seen in Table 2.

Table 2 Analysis types
Full size table

6 Implementation

6.1 SonarQube

SonarQube was configured and installed, according to the requirements. For each language to be analyzed, a Quality Profile was implemented with many rules to be applied in the analysis, as shown in Table 3.

Table 3 Implemented quality profiles
Full size table

Also, some Quality Gates were implemented, with a careful and thorough investigation of existing quality gates on SonarQube, searching for the most appropriate according to the objectives and specifications of the solution.

After that, two different quality gates were created, one for the Lightweight analysis, and another for the Full analysis, as can be seen in Table 4.

Table 4 Quality gates
Full size table

6.2 CIAnalyzer Tool

According to the requirement with ID 11, “The platform shall be developed adhering to the existing architecture and have a Command Line Tool that can be used by external tools”, a tool called CIAnalyzer Tool was constructed, using Python. For this tool, the Web API of SonarQube was used and some methods were created in the CIAnalyzer Tool to manage the analysis, like create project, start analysis, select quality gate and other ones.

6.3 Jenkins

In Jenkins, a pipeline was constructed to push the analysis and, if it passes, create the release.

As it is possible to see trough Fig. 2, the job has many stages, and all stages were constructed using Groovy programming language. There is one Job for each analysis type, but they are equal, only the name and the analysis type differ. If it is a Full analysis, the stages “Checking for Differences” and “Create Release” are skipped. On the other hand, the focus of the Lightweight analysis is on the new code, so all the stages are fulfilled.

Fig. 2
figure 2

Jenkins job

Full size image

Firstly, it downloads the dependencies and loads the sources (team’s repository). Then, it checks for differences between the current and the last snapshot, prepares the analysis by defining which quality gate is going to be implemented and, after that, it starts the analysis. Afterwards, it gets the results and, if it passes the analysis, defines the branch for the next analysis. After that, define the status of the build: passed if the analysis passed and failed if the analysis failed. If it passes, creates the release.

7 Results

In this section, the results of this project are displayed, always comparing the first analysis with the last one, which provides a simple way to see the results. Throughout Figs. 3, 4, 5, 6, 7, 8, 9, colors and ratings are presented, which are defined by SonarQube. The Maintainability (code smells) rating scale categorizes as follows: A for outstanding remediation costs ≤ 5% of the time invested, B for 6–10%, C for 11–20%, D for 21–50%, and E for anything exceeding 50%. Also, the Reliability rating scale is defined as follows: A for 0 bugs, B for at least 1 minor bug, C for at least 1 major bug, D for at least 1 critical bug, and E for at least 1 blocker bug. Finally, the Security rating scale is as follows: A for 0 vulnerabilities, B for at least 1 minor vulnerability, C for at least 1 major vulnerability, D for at least 1 critical vulnerability, and E for at least 1 blocker vulnerability.

Fig. 3
figure 3

Results overview number of bugs, code smells and vulnerabilities

Full size image
Fig. 4
figure 4

Reliability first analysis

Full size image
Fig. 5
figure 5

Reliability last analysis

Full size image
Fig. 6
figure 6

Security first analysis

Full size image
Fig. 7
figure 7

Security last analysis

Full size image
Fig. 8
figure 8

Maintainability first analysis

Full size image
Fig. 9
figure 9

Maintainability last analysis

Full size image

Through Fig. 3, it is possible to see that there was a 77.7% decrease in bugs, a 64.6% reduction in code smells, and 100% in vulnerabilities, that are very good results.

Then, looking for the reliability, the graphs in Figs. 4 and 5 represents the bugs’ operational risks, sorted by volume of bugs per file. In this graph, the color represents the reliability rating, and each bubble is one file, and the closer a bubble’s color is to red, more severe the bugs are, and if they are green, it means that is better. The bubble size represents the bug volume in each file, so if the file has many bugs, the bubble will be bigger, and the position represents the estimated time to resolve the bugs. Figure 4 represents the reliability in the first analysis and Fig. 5 in the last one. It can be observed an improvement in the max reliability remediation effort, as well as in the last analysis more green and little bubbles, compared with the first analysis.

The graph in Figs. 6 and 7 represents the security, where the represented vulnerabilities’ operational risks, is constructed with the same logic of the ones in Figs. 4 and 5, so the explanation is the same, but the size of bugs represents the number of vulnerabilities, instead of bugs. Figure 6 represents the security in the first analysis and Fig. 7 in the last one. In the last analysis, it’s possible to see that there are no files with vulnerabilities, which is great.

Then, the maintainability represents the code smells’ long-term risks, so the size of the bubbles represents the number of code smells. Figure 8 represents the maintainability in the first analysis and Fig. 9 in the last. It can be observed, through Fig. 9 that the technical debt improved, as well as the number of green bubbles.

In a general way, in Fig. 10 can be observed all the history of the number of issues, from April 12 to June 28.

Fig. 10
figure 10

Results history

Full size image

The chart in Fig. 10 shows that the number of issues was either dropping or remaining constant, meaning that no new issues were ever added to the code once the solution had been implemented.

Through the results, it is possible to say that, with the implementation of the solution, the number of issues in the software has improved, and it is now much better compared with the results of the analysis. In this way, it is feasible to determine that all the objectives for this dissertation have been achieved. It is possible to emphasize that the support solution has contributed to the creation and maintenance of high-quality because there are now fewer issues compared to the initial analysis, because of the implementation of the support solution.

8 Conclusion

To make the development and maintenance of high-quality software easier, this study offers an in-depth overview of how to perform continuous inspection of software quality in an automobile project. For this, a support solution was implemented and any issues that already exist were manually fixed. The results indicate that both the support solution and the overall approach effectively contributed to achieving the major goal. Following implementation, developers are now unable to deliver software with issues, with the team successfully resolving 66.7% of initially identified issues through manual intervention.

Saying that the ideal situation is to resolve every issue is impractical since the team would have to put development on hold for a certain period of time, which is not expected to occur. In this way, while not all the problems were resolved, a significant number of them were within the aim of this work. In general, there was no rise in the number of issues in the repository following the adoption of the solution, and it is evident that the number of issues has decreased, as it’s possible to see through the results, and it is really advantageous for the team, the product, and the company. Additionally, it is now (after this implementation) possible to observe an improvement in the way that all team members produce cleaner code, use best practices, and pay attention to any potential issue they may be entering in the software that they are producing.

In sum, since the requirements and objectives were accomplished and the results are promising, it can be said that this work adds in theory and in practice to the CI/CD area, and the knowledge created during the dissertation was helpful, contributing with a solution capable to provide continuous inspection of the software quality, in an automotive project. However, there is always room for improvement, so in addition to fixing the current issues, we will also apply this solution to other projects, develop new, targeted rules and best practices, and create a dashboard with more comprehensive and practical visuals to help with decision-making.