Software Development as an Experiment System: A Qualitative Survey on the State of the Practice

  • Eveliina LindgrenEmail author
  • Jürgen Münch
Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 212)


An experiment-driven approach to software product and service development is gaining increasing attention as a way to channel limited resources to the efficient creation of customer value. In this approach, software functionalities are developed incrementally and validated in continuous experiments with stakeholders such as customers and users. The experiments provide factual feedback for guiding subsequent development. Although case studies on experimentation in industry exist, the understanding of the state of the practice and the encountered obstacles is incomplete. This paper presents an interview-based qualitative survey exploring the experimentation experiences of ten software development companies. The study found that although the principles of continuous experimentation resonated with industry practitioners, the state of the practice is not yet mature. In particular, experimentation is rarely systematic and continuous. Key challenges relate to changing organizational culture, accelerating development cycle speed, and measuring customer value and product success.


Continuous experimentation Experiment-driven software development Customer feedback Qualitative survey 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bosch, J.: Building products as innovation experiment systems. In: Cusumano, M.A., Iyer, B., Venkatraman, N. (eds.) ICSOB 2012. LNBIP, vol. 114, pp. 27–39. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  2. 2.
    Version One: The 8th Annual “State of Agile” Survey.
  3. 3.
    Holmström Olsson, H., Alahyari, H., Bosch, J.: Climbing the “stairway to heaven”: a multiple-case study exploring barriers in the transition from agile development towards continuous deployment of software. In: 38th EUROMICRO Conference on Software Engineering and Advanced Applications SEAA, pp. 392–399. IEEE Press (2012)Google Scholar
  4. 4.
    Fagerholm, F., Guinea, A.S., Mäenpää, H., Münch, J.: Building blocks for continuous experimentation. In: 1st International Workshop on Rapid Continuous Software Engineering, pp. 26–35. ACM, New York (2014)Google Scholar
  5. 5.
    Ries, E.: The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Business, New York (2011)Google Scholar
  6. 6.
    Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., Pohlmann, N.: Online controlled experiments at large scale. In: 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1168–1176. ACM, New York (2013)Google Scholar
  7. 7.
    Tang, D., Agarwal, A., O’Brien, D., Meyer, M.: Overlapping experiment infrastructure: more, better, faster experimentation. In: 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 17–26. ACM, New York (2010)Google Scholar
  8. 8.
    Steiber, A., Alänge, S.: A Corporate System for Continuous Innovation: The Case of Google Inc. European Journal of Innovation Management 16(2), 243–264 (2013)CrossRefGoogle Scholar
  9. 9.
    Amatriain, X.: beyond data: from user information to business value through personalized recommendations and consumer science. In: 22nd ACM International Conference on Information and Knowledge Management, pp. 2201–2208. ACM, New York (2013)Google Scholar
  10. 10.
    Adams, R.J., Evans, B., Brandt, J.: Creating small products at a big company: adobe’s pipeline innovation process. In: CHI 2013 Extended Abstracts on Human Factors in Computing Systems, pp. 2331–2332. ACM, New York (2013)Google Scholar
  11. 11.
    Davenport, T.H.: How to Design Smart Business Experiments. Harvard Business Review 87(2), 68–77 (2009)MathSciNetGoogle Scholar
  12. 12.
    Holmström Olsson, H., Bosch, J.: Post-deployment data collection in software-intensive embedded products. In: Herzwurm, G., Margaria, T. (eds.) ICSOB 2013. LNBIP, vol. 150, pp. 79–89. Springer, Heidelberg (2013)Google Scholar
  13. 13.
    Holmström Olsson, H., Bosch, J.: Towards data-driven product development: a multiple case study on post-deployment data usage in software-intensive embedded systems. In: Fitzgerald, B., Conboy, K., Power, K., Valerdi, R., Morgan, L., Stol, K.-J. (eds.) LESS 2013. LNBIP, vol. 167, pp. 152–164. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  14. 14.
    Münch, J., Fagerholm, F., Johnson, P., Pirttilahti, J., Torkkel, J., Jäarvinen, J.: Creating minimum viable products in industry-academia collaborations. In: Fitzgerald, B., Conboy, K., Power, K., Valerdi, R., Morgan, L., Stol, K.-J. (eds.) LESS 2013. LNBIP, vol. 167, pp. 137–151. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  15. 15.
    Fink, A.: Analysis of qualitative surveys. In: Fink, A. (ed.) The survey handbook, pp. 61–78. SAGE Publications, California (2003)Google Scholar
  16. 16.
    Jansen, H.: The Logic of Qualitative Survey Research and its Position in the Field of Social Research Methods. Forum: Qualitative Social Research 11(2), 1–21 (2010)Google Scholar
  17. 17.
    Runeson, P., Höst, M.: Guidelines for Conducting and Reporting Case Study Research. Empirical Software Engineering 14(2), 131–164 (2009)CrossRefGoogle Scholar
  18. 18.
    Robson, C.: Real World Research: A Resource for Users of Social Research Methods in Applied Settings. Wiley, Chichester (2011)Google Scholar
  19. 19.
    Lindgren, E., Münch, J.: Interview guide and codebook for the paper “Software Development as an Experiment System”.
  20. 20.
    Need for Speed research program (N4S).
  21. 21.
    Vinson, N., Singer, J.: A practical guide to ethical research involving humans. In: Shull, F., Singer, J., Sjøberg, D.I.K. (eds.) Guide to Advanced Empirical Software Engineering, pp. 229–256. Springer, London (2008)CrossRefGoogle Scholar
  22. 22.
    ATLAS.ti Scientific Software Development GmbH,
  23. 23.
    Ståhl, D., Bosch, J.: Modeling Continuous Integration Practice Differences in Industry Software Development. Journal of Systems and Software 87, 48–59 (2014)CrossRefGoogle Scholar
  24. 24.
    Basili, V., Heidrich, J., Lindvall, M., Münch, J., Regardie, M., Rombach, D., Seaman, C., Trendowicz, A.: GQM+Strategies: a comprehensive methodology for aligning business strategies with software measurement. In: DASMA Software Metric Congress (MetriKon 2007): Magdeburger Schriften Zum Empirischen Software Engineering, Kaiserslautern, Germany, pp. 253–266 (2007)Google Scholar
  25. 25.
    Easterbrook, S., Singer, J., Storey, M., Damian, D.: Selecting empirical methods for software engineering research. In: Shull, F., Singer, J., Sjøberg, D.I.K. (eds.) Guide to Advanced Empirical Software Engineering, pp. 285–311. Springer, London (2008)CrossRefGoogle Scholar
  26. 26.
    Lindgren, E., Münch, J., Männistö, T.: Exploring Software Development as an Experiment System. Master’s Thesis. University of Helsinki (2015)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of HelsinkiHelsinkiFinland

Personalised recommendations