Abstract
From having been exclusive for companies in the online domain, feature experiments are becoming increasingly important for software-intensive companies also in other domains. Today, companies run experiments, such as e.g. A/B tests, to optimize product performance and to learn about user behaviors, as well as to guide product development and innovation. However, although experimentation with customers has become an effective mechanism to improve products and increase revenue, companies struggle with how to leverage the results of the experiments they run. In this paper, we study the reasons for this and we identify three key challenges that make feature experimentation a difficult task. Our research reveals the following challenges: (1) the impact of experiments doesn’t scale, (2) business KPIs and team level metrics are not aligned and (3) it is unclear if the available solutions are applicable across domains.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Olsson, H.H., Bosch, J.: Towards data-driven product development: a multiple case study on post-deployment data usage in software-intensive embedded systems. In: Fitzgerald, B., Conboy, K., Power, K., Valerdi, R., Morgan, L., Stol, K.-J. (eds.) LESS 2013. LNBIP, vol. 167, pp. 152–164. Springer, Heidelberg (2013). doi:10.1007/978-3-642-44930-7_10
Olsson, H.H., Bosch, J.: From opinions to data-driven software R&D: a multi-case study on how to close the ‘Open Loop’ problem. In: Proceedings of EUROMICRO, Software Engineering and Advanced Applications (SEAA), 27–29 August, Verona, Italy (2014)
Olsson, H.H., Bosch, J.: Towards evidence-based development: learnings from embedded systems, online games and internet of things. To appear in IEEE Software (forthcoming)
Bosch, J., Eklund, U.: Eternal embedded software: towards innovation experiment systems. In: Margaria, T., Steffen, B. (eds.) ISoLA 2012. LNCS, vol. 7609, pp. 19–31. Springer, Heidelberg (2012). doi:10.1007/978-3-642-34026-0_3
Bosch, J.: Building products as innovation experiment systems. In: Cusumano, M.A., Iyer, B., Venkatraman, N. (eds.) ICSOB 2012. LNBIP, vol. 114, pp. 27–39. Springer, Heidelberg (2012). doi:10.1007/978-3-642-30746-1_3
Kohavi, R., Longbotham, R.: Online controlled experiments and A/B tests. In: Encyclopedia of Machine Learning and Data Mining, no. Ries 2011, pp. 1–11 (2015)
Fagerholm, F., Guinea, A.F., Mäenpää, H., Münch, J.: Building blocks for continuous experimentation. In: Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering (RCoSE), pp. 26–35 (2014)
Fabijan, A., Dmitriev, P., Olsson, H.H., Bosch J.: The evolution of continuous experimentation in software product development: from data to a data-driven organization at scale. In: Proceedings of the 39th International Conference on Software Engineering (ICSE), 20–28th May, Buenos Aires, Argentina (2017)
Olsson, H.H., Bosch, J.: So Much Data; So Little Value A multi-case study on improving the impact of data-driven development practices. In Proceedings of the Ibero American Conference on Software Engineering (ClbSE), 22nd–23rd May, Buenos Aires, Argentina (2017)
Dmitriev, P., Frasca, B., Gupta, S., Kohavi, R., Vaz, G. (forthcoming). Pitfalls of Long-Term Online Controlled Experiments. (To appear in IEEE Big Data)
Fagerholm, F., et al.: The RIGHT model for continuous experimentation. J. Syst. Softw. 123, 292–305 (2016)
Bosch, J.: Future trends in software engineering. IEEE Softw. 33(1), 82–88 (2016)
Pohl, K.: Requirements Engineering: Fundamentals, Principles, and Techniques. Springer Publishing Company, Incorporated (2010)
Ries, E.: The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Business, New York (2011)
Fabijan, A., Olsson, H.H., Bosch, J.: Time to say ‘Good Bye’: feature lifecycle. In: 42nd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Limassol, Cyprus. 31 August–2 September, pp. 9–16 (2016)
Kim, M., Zimmermann, T., DeLine, R., Begel, A.: The Emerging Role of Data Scientists on Software Development Teams, no. MSR-TR-2015–30, p. 10 2015
Runesson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empirical Softw. Eng. 14, 131 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Olsson, H.H., Bosch, J., Fabijan, A. (2017). Experimentation that Matters: A Multi-case Study on the Challenges with A/B Testing. In: Ojala, A., Holmström Olsson, H., Werder, K. (eds) Software Business. ICSOB 2017. Lecture Notes in Business Information Processing, vol 304. Springer, Cham. https://doi.org/10.1007/978-3-319-69191-6_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-69191-6_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-69190-9
Online ISBN: 978-3-319-69191-6
eBook Packages: Computer ScienceComputer Science (R0)