An experience report on applying software testing academic results in industry: we need usable automated test generation

Abstract

What is the impact of software engineering research on current practices in industry? In this paper, I report on my direct experience as a PhD/post-doc working in software engineering research projects, and then spending the following five years as an engineer in two different companies (the first one being the same I worked in collaboration with during my post-doc). Given a background in software engineering research, what cutting-edge techniques and tools from academia did I use in my daily work when developing and testing the systems of these companies? Regarding validation and verification (my main area of research), the answer is rather short: as far as I can tell, only FindBugs. In this paper, I report on why this was the case, and discuss all the challenging, complex open problems we face in industry and which somehow are “neglected” in the academic circles. In particular, I will first discuss what actual tools I could use in my daily work, such as JaCoCo and Selenium. Then, I will discuss the main open problems I faced, particularly related to environment simulators, unit and web testing. After that, popular topics in academia are presented, such as UML, regression and mutation testing. Their lack of impact on the type of projects I worked on in industry is then discussed. Finally, from this industrial experience, I provide my opinions about how this situation can be improved, in particular related to how academics are evaluated, and advocate for a greater involvement into open-source projects.

Appendix A

1.1 A.1 IntelliJ IDEA

Like for production code, to write tests you need an editor. One of the most popular IDEs for Java is IntelliJ IDEA.¹⁷ Simply put, the autocomplete feature in IntelliJ IDEA is the most important advancement in software development I have experienced since I started to write code in 2000 when I was a student. No longer continuous copy&paste of long variable/method names, or even worse typing them directly. Most of the time, just the first one/two letters are enough to correctly autocomplete. Plus there are all the other kinds of smart autocomplete, like for example autoclosing of XML tags, or showing an autofilled selection of valid data based on the current context (especially useful when dealing with Maven pom files). Other IDEs have some basic support for autocomplete, but they look very basic in comparison.

From a scientific point of view, arguably there is not much “innovation” in an autocomplete feature, as it is a relatively simple idea. However, likely there are many engineering challenges to make it right and user-friendly in a general way applicable in different contexts. Would these engineering challenges be of interest for an academic conference? However, IntelliJ IDEA also provides cutting-edge techniques related to code-refactoring, and code-refactoring has deep roots in software engineering research (Griswold and Opdyke 2015).

Academic research can have large influence on practice (e.g., refactoring), but there are often important topics, with direct impact on software development practice, that have received less attention (e.g., autocomplete). What is important for practitioners does not always match what researchers find interesting to work on.

IntelliJ IDEA is developed by a Czech company called JetBrains, founded by three Java developers in 2000. The community edition is open-source, although the pro version (needed for enterprize programming) requires to buy a license.

1.2 A.2 JUnit

To write and run tests, you need a framework. In the Java world, the de-facto standard is JUnit.¹⁸ There are other popular frameworks as well, like TestNG and Spock, but none of the projects I worked in used them. JUnit was founded by a software engineer and a computer scientist (currently working at Facebook and Microsoft).

1.3 A.3 Maven

To compile a Java project and handle all the third-party libraries automatically, you need a build tool. Currently, the de-factor standard is Maven.¹⁹ Other options are Gradle and Ant.

Of particular interest is the fact that build tools are used to run test cases as well, e.g. by executing the command “mvn test”. Maven has two main official plugins dealing with the running of tests: Surefire (for unit testing) and Failsafe (for integration testing). These plugins provide a lot of extra functionalities, like for example the ability of running tests in parallel or re-executing flaky tests up to a certain number of times.

Maven is developed by the Apache Software Foundation,²⁰ a non-profit corporation with apparently no employee but thousands of volunteers.

1.4 A.4 Jenkins

When several developers work on the same piece of software, it is important to have a remote Continuous Integration server, where the system is built at each new pushed code change. Each time the system is built, all the regression tests are automatically run as well. This is also important for long to execute test suites, which might be cumbersome otherwise to run directly on the developers’ machines.

Arguably, Jenkins²¹ is the most popular Continuous Integration server for Java. Another one is for example Bamboo. Jenkins has several utilities and plugins to help monitor the execution and collect the results of the test cases.

Jenkins is an open-source fork of Hudson, which was originally developed at Sun Microsystems (the company behind Java before being acquired by Oracle). The lead developer of Jenkins/Hudson is an engineer now working as CTO at CloudBees, a company that provides continuous delivery solutions powered by Jenkins.

1.5 A.5 JaCoCo

How many tests one should write? Is the current test suite “good enough” or should it be extended? Of course, one can (and should) have a test for each requirement, but that does not tell you what in the code has been tested. A complementary approach is to look at what is executed by the tests, and then check if any part of the source code has not been exercised. If some parts of the code is not executed, it would be good to add tests to cover them, as those parts might have bugs.

One of the most used tools for Java code coverage is the open-source JaCoCo.²² Others are Cobertura and the commercial Clover. JaCoCo was started by an engineer now working as CTO in a consulting company.

1.6 A.6 Selenium

Writing system level tests for a web application is a challenge, as users interact with it by using a browser (e.g., Chrome or Firefox). The Selenium²³ tool simplifies this task by making possible to programmatically control browsers directly from the test cases. Selenium was originally started at ThoughtWorks as an internal tool to simplify their testing activities.

1.7 A.7 WireMock

Many enterprizes today rely on web services (e.g., SOAP and REST), especially when using microservice architectures. Writing deterministic tests against applications using web services is challenging, as usually the developers of such applications do not have control over those external web services.

WireMock²⁴ is a tool that starts a proxy server, where HTTP responses can be easily configured based on the incoming messages. This can be used to practically mock away those external web services without the need to modify the tested application. WireMock is developed and maintained by an engineer currently working as a consultant.

1.8 A.8 REST-Assured

When testing a RESTful web service, one has to craft an HTTP message, open a TCP connection, send the message, read back the answer, and finally evaluate assertions on such response. To greatly simplify this sequence of operations, one can use a library like REST-Assured.²⁵ REST-Assured was started by an engineer working as a consultant.

1.9 A.9 Mockito

When unit testing code, there is often the problem of dependencies that need complex initialization. At times, some classes cannot be even instantiated, as only interfaces are available. A solution for this kind of problems is to create mock objects for those dependencies.

In Java, there are several libraries that can create mocks, like EasyMock and JMock. The most used one is Mockito.²⁶ Mockito was started by an engineer working at ThoughtWorks. As of 2013, Mockito is among the top 10 libraries used in Java among open-source projects.²⁷

1.10 A.10 ZAP

Security is an extremely important topic when dealing with web applications. It is not rare that security breaches end up in the newspapers, e.g. when hackers steal large number of personal details, like the 500 million user accounts stolen at Yahoo²⁸ in 2014. Often, you just need a silly mistake, like forgetting to secure a single cookie, to compromise a user account or a whole web application.

To help mitigating this type of issues, one can use the ZAP²⁹ tool, which can automatically do different kinds of security attacks against web applications. ZAP is one of the main tools of the OWASP Foundation, a non-profit organization dedicated to web application security. ZAP was started by an engineer now working as a security expert at Mozilla.

1.11 A.11 JMeter

Besides validating the requirements of a web application, there are also non-functional properties that need to be taken into account, like for example response time. Would the application behave properly when under heavy load, like for example hundreds of thousands of users accessing it at the same time?

To help writing performance tests, one can use JMeter.³⁰ JMeter is developed by the members of the Apache Foundation.

1.12 A.12 FindBugs

It is not uncommon that developers make the same typical mistakes, like opening a resource and then forgetting to close it. Or calling methods on immutable objects like strings and then forgetting to save the result in a variable. It is possible to define patterns of common mistakes, and then statically scan source code for those patterns without the need to execute the analyzed code. One great benefit of statical analysis is that it can be easily applied on large code bases and find real errors relatively quickly.

The most famous static analysis tool for Java is FindBugs.³¹ Another popular tool is SonarQube, which also provides a commercial version. FindBugs was originally developed by academics at the University of Maryland (Hovemeyer and Pugh 2004).