Visualizing Testing Activities to Support Continuous Integration: A Multiple Case Study

  • Agneta Nilsson
  • Jan Bosch
  • Christian Berger
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 179)


While efficient testing arrangements are the key for software companies that are striving for continuous integration, most companies struggle with arranging these highly complex and interconnected testing activities. There is often a lack of an adequate overview of companies’ end-to-end testing activities, which tend to lead to problems such as double work, slow feedback loops, too many issues found during post-development, disconnected organizations, and unpredictable release schedules. We report from a multiple-case study in which we explore current testing arrangements at five different software development sites. The outcome of the study is a visualization technique of the testing activities involved from unit and component level to product and release level that support the identification of improvement areas. This model for visualizing the end-to-end testing activities for a system has been used to visualize these five cases and has been validated empirically.


continuous integration software testing visualization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Fowler, M.: Continuous integration (2007),
  2. 2.
    Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R.C., Mellor, S., Schwaber, K., Sutherland, J., Thomas, D.: Manifesto for the Agile Software Development (2001)Google Scholar
  3. 3.
    Holck, J., Jørgensen, N.: Continuous Integration and Quality Assurance: A Case Study of Two Open Source Projects. Australian Journal of Information Systems 11(1), 40–53 (2004)Google Scholar
  4. 4.
    Ståhl, D., Bosch, J.: Modeling continuous integration practice differences in industry software development. Journal of Systems and Software 87, 48–59 (2014)CrossRefGoogle Scholar
  5. 5.
    Beck, K.: Test Driven Development: By Example. Addison-Wesley Professional (2002)Google Scholar
  6. 6.
    Whittaker, J.A., Arbon, C., Carollo, J.: How Google Tests Software. Addison-Wesley Professional (2012)Google Scholar
  7. 7.
    Sturdevant, K.: Cruisin’ and Chillin’: Testing the Java-Based Distributed Ground Data System ‘Chill’ with CruiseControl. In: Aerospace Conference 2007, pp. 1–8 (2007)Google Scholar
  8. 8.
    Downs, J., Hosking, J., Plimmer, B.: Status Communication in Agile Software Teams: A Case Study. In: Proceedings of the Fifth International Conference on Software Engineering Advances, pp. 82–87 (2010)Google Scholar
  9. 9.
    Stolberg, S.: Enabling Agile Testing through Continuous Integration. In: Proceedings of the Agile Conference, pp. 369–374 (2009)Google Scholar
  10. 10.
    Kim, E.H., Na, J.C., Ryoo, S.M.: Implementing an Effective Test Automation Framework. In: Proceedings of the 33rd Annual IEEE International Computer Software and Applications Conference, pp. 534–538 (2009)Google Scholar
  11. 11.
    Kim, E.H., Na, J.C., Ryoo, S.M.: Test Automation Framework for Implementing Continuous Integration. In: Proceedings of the Sixth International Conference on Information Technology: New Generations, pp. 784–789 (2009)Google Scholar
  12. 12.
    Hoffman, B., Cole, D., Vines, J.: Software Process for Rapid Development of HPC Software Using CMake. In: Proceedings of the DoD High Performance Computing Modernization Program Users Group Conference, pp. 378–382 (2009)Google Scholar
  13. 13.
    Ablett, R., Sharlin, E., Maurer, F., Denzinger, J., Schock, C.: BuildBot: Robotic Monitoring of Agile Software Development Teams. In: Proceedings of the 16th IEEE International Symposium on Robot and Human Interactive Communication, pp. 931–936 (2007)Google Scholar
  14. 14.
    Yuksel, H.M., Tuzun, E., Gelirli, E., Biyikli, E., Baykal, B.: Using Continuous Integration and Automated Test Techniques for a Robust C4ISR System. In: Proceedings of the 24th International Symposium on Computer and Information Sciences, pp. 734–748 (2009)Google Scholar
  15. 15.
    Lacoste, F.: Killing the Gatekeeper: Introducing a Continuous Integration System. In: Proceedings of the Agile Conference, pp. 387–392 (2009)Google Scholar
  16. 16.
    Goodman, D., Elbaz, M.: ‘It’s not the pants, it’s the people in the pants’ Learnings from The Gap Agile Transformation. In: Proceedings of the Agile Conference, pp. 112–115 (2008)Google Scholar
  17. 17.
    Downs, J., Plimmer, B., Hosking, J.: Ambient Awareness of Build Status in Collocated Software Teams. In: Proceedings of the 34th International Conference on Software Engineering (ICSE), pp. 507–517 (2012)Google Scholar
  18. 18.
    Yin, R.K.: Case study research: design and methods. Sage-Publications, Newbury Park (1994)Google Scholar
  19. 19.
    Runeson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering 14(2), 131–164 (2009)CrossRefGoogle Scholar
  20. 20.
    Patton, M.Q.: How to Use Qualitative Methods in Evaluation. Sage Publications, Newbury Park (1987)Google Scholar
  21. 21.
    Myers, M.D., Newman, M.: The qualitative interview in IS research: Examining the craft. Information and Organization 17, 2–26 (2007)CrossRefGoogle Scholar
  22. 22.
    Bertolino, A.: Software testing research: Achievements, challenges, dreams. In: Proceedings of Future of Software Engineering (2007)Google Scholar
  23. 23.
    Ries, E.: The Lean Startup: How Constant Innovation Creates Radically Successful Businesses. Portfolio Penguin (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Agneta Nilsson
    • 1
    • 2
  • Jan Bosch
    • 1
    • 2
  • Christian Berger
    • 1
    • 2
  1. 1.Software Engineering Division, Dpmt of Computer Science and EngineeringChalmers University of TechnologySweden
  2. 2.University of GothenburgGothenburgSweden

Personalised recommendations