Advertisement

More for Less: Automated Experimentation in Software-Intensive Systems

  • David Issa MattosEmail author
  • Jan Bosch
  • Helena Holmström Olsson
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10611)

Abstract

Companies developing autonomous and software-intensive systems show an increasing need to adopt experimentation and data-driven strategies in their development process. With the growing complexity of the systems, companies are increasing their data analytic and experimentation teams to support data-driven development. However, organizations cannot increase in size at the same pace as the system complexity grows. Experimentation teams could run a larger number of experiments by letting the system itself to coordinate its own experiments, instead of the humans. This process is called automated experimentation. However, currently, no tools or frameworks address the challenge of running automated experiments.

This paper discusses, through a set of architectural design decisions, the development of an architecture framework that supports automated continuous experiments. The contribution of this paper is twofold. First, it presents, through a set of architectural design decisions, an architecture framework for automated experimentation. Second, it evaluates the architecture framework experimentally in the context of a human-robot interaction proxemics distance problem. This automated experimentation framework aims to deliver more value from the experiments while using fewer R&D resources.

Keywords

Continuous experimentation Automated experimentation Architectural design decisions 

Notes

Acknowledgements

This work was partially supported by the Wallenberg Autonomous Systems and Software Program (WASP).

References

  1. 1.
    Fabijan, A., Olsson, H.H., Bosch, J.: Time to say ‘good bye’: feature lifecycle. In: 2016 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 9–16 (2016)Google Scholar
  2. 2.
    Olsson, H.H., Bosch, J.: From opinions to data-driven software R&D: a multi-case study on how to close the ‘open loop’ problem. In: Proceedings of 40th EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), SEAA 2014, pp. 9–16 (2014)Google Scholar
  3. 3.
    Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data Mining Knowl. Discov. 18(1), 140–181 (2009)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Fabijan, A., Dmitriev, P., Olsson, H.H., Bosch, J.: The evolution of continuous experimentation in software product development. In: Proceedings of the 39th International Conference on Software Engineering, ICSE 2017 (2017)Google Scholar
  5. 5.
    Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., Pohlmann, N.: Online controlled experiments at large scale. In: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD 2013, p. 1168 (2013)Google Scholar
  6. 6.
    Li, L., Chu, W., Langford, J., Schapire, R.E.: A contextual-bandit approach to personalized news article recommendation. WWW 2010, 10 (2010)Google Scholar
  7. 7.
    Bosch, J., Olsson, H.H.: Data-driven continuous evolution of smart systems. In: Proceedings of the 11th International Workshop on Software Engineering for Adaptive and Self-Managing Systems - SEAMS 2016, pp. 28–34 (2016)Google Scholar
  8. 8.
    Mattos, D.I., Bosch, J., Olsson, H.H.: Your system gets better every day you use it: towards automated continuous experimentation. In: 2017 43th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (2017)Google Scholar
  9. 9.
    Kohavi, R., Deng, A., Frasca, B., Walker, T.: Online controlled experiments at large scale. In: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD 2013, pp. 1–9 (2013)Google Scholar
  10. 10.
    Jansen, A., Bosch, J.: Software architecture as a set of architectural design decisions. In: 5th Working IEEE/IFIP Conference on Software Architecture (WICSA 2005), vol. 2005, pp. 109–120 (2005)Google Scholar
  11. 11.
    Bakshy, E., Eckles, D., Bernstein, M.S.: Designing and deploying online field experiments. In: Proceedings of 23rd International Conference on World wide web - WWW 2014, pp. 283–292, September 2014Google Scholar
  12. 12.
    Tang, D., Agarwal, A., O’Brien, D., Meyer, M.: Overlapping experiment infrastructure. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD 2010, p. 17 (2010)Google Scholar
  13. 13.
    Hevner, A.R., March, S.T., Park, J., Ram, S.: Design science in information systems research. MIS Q. 28(1), 75–105 (2004)CrossRefGoogle Scholar
  14. 14.
    Oskoei, M.A., Walters, M.L., Dautenhahn, K.: An autonomous proxemic system for a mobile companion robot. In: Proceedings of Second International Symposium on New Frontier Human-Robot Interaction, pp. 9–15, April 2010Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringChalmers University of TechnologyGothenburgSweden
  2. 2.Department of Computer Science and Media TechnologyMalmö UniversityMalmöSweden

Personalised recommendations