Data Driven Development: Challenges in Online, Embedded and On-Premise Software

  • Helena Holmström OlssonEmail author
  • Jan Bosch
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11915)


For more than a decade, data driven development has attracted attention as one of the most powerful means to improve effectiveness and ensure value delivery to customers. In online companies, controlled experimentation is the primary technique to measure how customers respond to variants of deployed software. In B2B companies, an interest for data driven development is rapidly emerging and experiments are run on selected instances of the system or as comparisons of previously computed data to ensure quality, improve configurations and explore new value propositions. Although the adoption of data driven development is challenging in general, it is especially so for embedded systems companies and for companies developing on-premise software solutions. Due to complex systems with hardware dependencies, safety-critical functionality and strict regulations, these companies have longer development cycles, less frequent deployments and limited access to data. In this paper, and based on multi-case study research, we explore the specific challenges that embedded systems companies and companies developing on-premise solutions experience when adopting data driven development practices. The contribution of the paper is two-fold. First, we provide empirical evidence in which we identify the key challenges that embedded systems and on-premise software solutions companies experience as they evolve through the process of adopting data driven development practices. Second, we define the key focus areas that these companies need to address for evolving their data driven development adoption process .


Data driven development Online software Embedded systems On-premise solutions Adoption process Challenges 


  1. 1.
    Holmström Olsson, H., Bosch, J.: Towards data-driven product development: a multiple case study on post-deployment data usage in software-intensive embedded systems. In: Fitzgerald, B., Conboy, K., Power, K., Valerdi, R., Morgan, L., Stol, K.-J. (eds.) LESS 2013. LNBIP, vol. 167, pp. 152–164. Springer, Heidelberg (2013). Scholar
  2. 2.
    Olsson, H.H., Bosch, J.: From opinions to data-driven software R&D: a multi-case study on how to close the ‘open loop’ problem. In: Proceedings of EUROMICRO, Software Engineering and Advanced Applications (SEAA), 27–29 August, Verona, Italy (2014)Google Scholar
  3. 3.
    Olsson, H.H., Bosch, J.: Towards evidence-based development: learnings from embedded systems, online games and internet of things. IEEE Softw. 4(5) (2017)Google Scholar
  4. 4.
    Patil, D.J.: Building Data Science Teams, pp. 1–25. Oreilly, Radar (2011)Google Scholar
  5. 5.
    Bosch, J.: Building products as innovations experiment systems. In: Proceedings of 3rd International Conference on Software Business, 18–20 June, Cambridge, Massachusetts (2012)Google Scholar
  6. 6.
    Kohavi, R., Longbotham, R.: Online controlled experiments and A/B tests. In: Encyclopedia of Machine Learning and Data Mining, no. Ries 2011, pp. 1–11 (2015)Google Scholar
  7. 7.
    Fagerholm, F., Guinea, A.F., Mäenpää, H., Münch, J.: Building blocks for continuous experimentation. In: Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering (RCoSE), pp. 26–35 (2014)Google Scholar
  8. 8.
    Fabijan, A., Dmitriev, P., Olsson, H.H., Bosch J.: The evolution of continuous experimentation in software product development: from data to a data-driven organization at scale. In Proceedings of the 39th International Conference on Software Engineering (ICSE), May 20–28th, Buenos Aires, Argentina (2017)Google Scholar
  9. 9.
    Bosch, J., Eklund, U.: Eternal embedded software: towards innovation experiment systems. In: Margaria, T., Steffen, B. (eds.) ISoLA 2012. LNCS, vol. 7609, pp. 19–31. Springer, Heidelberg (2012). Scholar
  10. 10.
    Van Nostrand, R.C.: Design of experiments using the taguchi approach: 16 steps to product and process improvement. Technometrics 44(3), 289 (2002)CrossRefGoogle Scholar
  11. 11.
    Fabijan, A., Dimitriev, P., Vermeer, L., Olsson, H.H., Bosch, J.: Experimentation growth: Evolving trustworthy A/B testing capabilities in oline software companies. J. Softw.: Evol. Process 30(12), e2113 (2018)Google Scholar
  12. 12.
    Bosch-Sijtsema, P., Bosch, J.: User involvement throughout the innovation process in high-tech industries. J. Prod. Innov. Manag. 32(5), 793–807 (2015)CrossRefGoogle Scholar
  13. 13.
    Olsson, H.H., Alahyari, H., Bosch, J.: Climbing the “stairway to heaven”: a multiple-case study exploring barriers in the transition from agile development towards continuous deployment of software. In: Proceedings of the 38th Euromicro Conference on Software Engineering and Advanced Applications, 5–7 September, Cesme, Izmir, Turkey (2012)Google Scholar
  14. 14.
    Ståhl, D., Bosch, J.: Modeling continuous integration practice differences in industry software development. J. Syst. Softw. 87(1), 48–59 (2014)CrossRefGoogle Scholar
  15. 15.
    Humble, J., Farley, D.: Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley, Boston (2010)Google Scholar
  16. 16.
    Fabijan, A., Dmitriev, P., Olsson, H.H., Bosch, J., Vermeer, L., Lewis, D.: Three key checklists and remedies for trustworthy analysis of online controlled experiments at scale. In: Proceedings of 41st International Conference on Software Engineering (ICSE), 25–31 May, Montreal, Canada (2019)Google Scholar
  17. 17.
    Olsson, H.H., Bosch, J.: Make up your mind: towards a comprehensive definition of customer value in large scale software development. CLEI Electron. J. 21(1) (2018)Google Scholar
  18. 18.
    Pohl, K.: Requirements Engineering: Fundamentals, Principles, and Techniques. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  19. 19.
    Ries, E.: The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Business, New York (2011)Google Scholar
  20. 20.
    Dmitriev, P., Frasca, B., Gupta, S., Kohavi, R., Vaz, G.: Pitfalls of long term online controlled experiments. In: Proceedings of IEEE International Conference on Big Data (Big Data), pp. 1367–1376 (2016)Google Scholar
  21. 21.
    Xia, T., Bhardwaj, S., Dmitriev, P., Fabijan, A.: Safe velocity: a practical guide to software deployment at scale using controlled rollout. In Proceedings of the 41st International Conference on Software Engineering (ICSE), 25–31 May, Montreal, Canada (2019)Google Scholar
  22. 22.
    Fabijan, A., Olsson, H.H., Bosch, J.: Customer feedback and data collection techniques in software R&D: a literature review. In: Fernandes, J., Machado, R., Wnuk, K. (eds.) Software Business (ICSOB). LNBIP, vol. 210, pp. 139–153. Springer, Cham (2015). Scholar
  23. 23.
    Issa Mattos, D., Dmitriev, P., Fabijan, A., Bosch, J., Holmström Olsson, H.: An activity and metric model for online controlled experiments. In: Kuhrmann, M., et al. (eds.) PROFES 2018. LNCS, vol. 11271, pp. 182–198. Springer, Cham (2018). Scholar
  24. 24.
    Heath, S.: Embedded Systems Design. EDN Series for Design Engineers, 2nd edn. Newnes, London (2003)Google Scholar
  25. 25.
  26. 26.
    Maxwell, J.A.: Qualitative Research Design: An Interactive Approach, 2nd edn. SAGE Publications, Thousands Oaks (2005)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Computer Science and Media TechnologyMalmö UniversityMalmöSweden
  2. 2.Department of Computer Science and EngineeringChalmers University of TechnologyGothenburgSweden

Personalised recommendations