What should your run-time configuration framework do to help developers?


The users or deployment engineers of a software system can adapt such a system to a wide range of deployment and usage scenarios by changing the value of configuration options, for example by disabling unnecessary features, tweaking performance-related parameters or specifying GUI preferences. However, the literature agrees that the flexibility of such options comes at a price: misconfigured options can lead a software system to crash in the production environment, while even in the absence of such configuration errors, a large number of configuration options makes a software system more complicated to deploy and use. In earlier work, we also found that developers who intend to make their application configurable face 22 challenges that impact their configuration engineering activities, ranging from technical to management-related or even inherent to the domain of configuration engineering. In this paper, we use a prototyping approach to derive and empirically evaluate requirements for tool support able to deal with 13 (primarily technical) configuration engineering challenges. In particular, via a set of interviews with domain experts, we identify four requirements by soliciting feedback on an incrementally evolving prototype. The resulting “Config2Code” prototype, which implements the four requirements, is then empirically evaluated via a user study involving 55 participants that comprises 10 typical configuration engineering tasks, ranging from the creation, comprehension, refactoring, and reviewing of configuration options to the quality assurance of options and debugging of configuration failures. A configuration framework satisfying the four requirements enables developers to perform more accurately and more swiftly in 70% and 60% (respectively) of the configuration engineering tasks than a state-of-the-practice framework not satisfying the requirements. Furthermore, such a framework allows to reduce the time taken for these tasks by up to 94.62%, being slower for only one task.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8


  1. Ansible (2018) Core java preferences api, http://www.ansible.com/. Accessed March 08, 2018

  2. Apel S, Batory D, Kästner C, Saake G (2013) Feature-oriented software product lines: Concepts and implementation. Springer, Berlin/Heidelberg, p 308. ISBN 978-3-642-37520-0. http://www.springer.com/computer/swe/book/978-3-642-37520-0, Tech Rep

    Google Scholar 

  3. Bass L, Weber I, Zhu L (2015) Devops: a software architect’s perspective, 1st edn. Addison-Wesley Professional, Reading

    Google Scholar 

  4. Behrang F, Cohen MB, Orso A (2015) Users beware: Preference inconsistencies ahead. In: Proceedings of the 10th joint meeting on foundations of software engineering, ser. ESEC/FSE’15, New York, NY, USA, pp 295–306

  5. Buse RP, Sadowski C, Weimer W (2011) Benefits and barriers of user evaluation in software engineering research. In: Proceedings of the International Conference on Object Oriented Programming Systems Languages and Applications, ser OOPSLA’11, pp 643–656

  6. Cassoli J (2016) Web application with spring annotation-driven configuration: Rapidly develop lightweight Java web applications using Spring with annotations, 1st edn. CreateSpace Independent Publishing Platform, Scotts Valley

    Google Scholar 

  7. Dong Z, Andrzejak A, Shao K (2015) Practical and accurate pinpointing of configuration errors using static analysis, Bremen, Germany, automated debugging;Configuration errors;Configuration options;Correlation degree;Misconfigurations;Run-time information;Software program;Testing oracles;. Online. Available https://doi.org/10.1109/ICSM.2015.7332463

  8. Dong Z, Andrzejak A, Lo D, Costa D (2016) Orplocator: Identifying read points of configuration options via static analysis. In: Proceedings of the 27th international symposium on software reliability engineering, ser. ISSRE’16, pp 185–195

  9. Huang P, Bolosky WJ, Singh A, Zhou Y (2015) Confvalley: a systematic configuration validation framework for cloud services. In: Proceedings of the tenth European conference on computer systems, ser. Eurosys ’15, NY, USA, pp 1–16

  10. Hubaux A, Xiong Y, Czarnecki K (2012) A user survey of configuration challenges in linux and ecos. In: Proceedings of the 6th Int’l workshop on variability modeling of software-intensive systems, pp 149–155

  11. Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test, and deployment automation, 1st edn. Addison-Wesley Professional, Reading

    Google Scholar 

  12. Jiang Y, Adams B (2015) Co-evolution of infrastructure and source code: an empirical study. In: Proceedings of the 12th Conference on Mining Software Repositories, ser. MSR’15, pp 45–55

  13. Jin D, Qu X, Cohen MB, Robinson B (2014) Configurations everywhere: Implications for testing and debugging in practice. In: Proceedings of the 36th international conference on soft. eng., ser. ICSE’14, pp 215–224

  14. Jin D, Cohen MB, Qu X, Robinson B (2014) Preffinder: Getting the right preference in configurable software systems. In: Proceedings of the 29th international conference on automated software engineering, ser. ASE ’14, pp 151–162

  15. Khan M, Huang Z, Li M, Taylor GA, Khan M (2017) Optimizing hadoop parameter settings with gene expression programming guided pso. Concurrency and Computation: Practice and Experience 29(3):186–197

    Article  Google Scholar 

  16. Ko AJ, LaToza T, Burnett M (2015) A practical guide to controlled experiments of software engineering tools with human participants. Empir Softw Eng 20(1):110–141

    Article  Google Scholar 

  17. Krum S, Van Hevelingen W, Kero B, Turnbull J, McCune J (2013) Pro Puppet, 2nd edn. Apress, New York

    Google Scholar 

  18. Li W, Li S, Liao X, Xu X, Zhou S, Jia Z (2017) Conftest: Generating comprehensive misconfiguration for system reaction ability evaluation. In: Proceedings of the 21st international conference on evaluation and assessment in software engineering, ser. EASE’17, NY, USA, pp 88–97

  19. van der Linden F, Schmid K, Rommes E (2007) Software product lines in action: the best industrial practice in product line engineering. Springer, Berlin. Online. Available: https://books.google.ca/books?id=PC4LyoSNNAkC

    Google Scholar 

  20. Lwakatare LE, Kuvaja P, Oivo M (2016) Relationship of devops to agile, lean and continuous deployment. In: Abrahamsson P., Jedlitschka A., Nguyen Duc A., Felderer M., Amasaki S., Mikkonen T. (eds) Product-focused software process improvement. Springer International Publishing, Cham, pp 399–415

    Google Scholar 

  21. Medeiros F, Kästner C, Ribeiro M, Gheyi R, Apel S (2016) A comparison of 10 sampling algorithms for configurable systems. In: Proceedings of the 38th international conference on software engineering, ser. ICSE ’16. Online. Available: https://doi.org/10.1145/2884781.2884793. ACM, New York, pp 643–654

  22. Meng X, Foong PS, Perrault S, Zhao S (2016) 5-step approach to designing controlled experiments. In: Proceedings of the international working conference on advanced visual interfaces, pp 358–359

  23. Morris K (2016) Infrastructure as code: managing servers in the cloud, 1st edn. O’Reilly Media, Sebastopol

    Google Scholar 

  24. Nadi S, Berger T, Kästner C, Czarnecki K (2014) Mining configuration constraints: Static analyses and empirical results. In: Proceedings of the 36th international conference on Soft. Eng., ser. ICSE’14, pp 140–151

  25. Nadi S, Berger T, Kästner C, Czarnecki K (2015) Where do configuration constraints stem from? an extraction approach and an empirical study. IEEE Trans Softw Eng 41(8):820–841

    Article  Google Scholar 

  26. Pohl K, Böckle G, van der Linden F (2005) Software product line engineering: foundations, principles and techniques. Springer, Berlin. Online. Available: https://books.google.ca/books?id=lsX8_O_TRkEC

    Google Scholar 

  27. Qu X, Acharya M, Robinson B (2011) Impact analysis of configuration changes for test case selection, ser. ISSRE’11, Hiroshima, Japan, pp 140–149

  28. Rabkin A, Katz R (2011) Precomputing possible configuration error diagnoses, ser. ASE’11, Lawrence, KS, United States, pp 193–202

  29. Rabiser R, Dhungana D (2007) Integrated support for product configuration and requirements engineering in product derivation, ser. EUROMICRO’07, Piscataway, USA, pp 193–200

  30. Rahman MT, Querel L-P, Rigby PC, Adams B (2016) Feature toggles: a case study and survey. In: Proceedings of the 13th IEEE working conference on mining software repositories (MSR), Austin, TX, pp 201–211

  31. Sarma A, Bortis G, van der Hoek A (2007) Towards supporting awareness of indirect conflicts across software configuration management workspaces. In: Proceedings of the 22 international conference on automated software engineering, ser. ASE’07, pp 94–103

  32. Sayagh M, Adams B (2018) Videos, http://mcis.polymtl.ca/msayagh/userStudy/Config2Code/

  33. Sayagh M, Kerzazi N, Adams B (2017a) On cross-stack configuration errors. In: Proceedings of the 39th international conference on software engineering, ser. ICSE ’17. IEEE Press, Piscataway, pp 255–265

  34. Sayagh M, Dong Z, Andrzejak A, Adams B (2017b) Does the choice of configuration framework matter for developers? empirical study on 11 java configuration frameworks. In: Proceedings of the 17th international working conference on source code analysis and manipulation, ser. SCAM’17, pp 41–50

  35. Sayagh M, Kerzazi N, Adams B, Petrillo F (2018) Software configuration engineering in practice - interviews, survey, and systematic literature review. Transactions on Software Engineering (TSE). Online. Available: mcis.polymtl.ca/publications/2018/tse_mohammed.pdf

  36. Sedlmair M, Meyer M, Munzner T (2012) Design study methodology: Reflections from the trenches and the stacks. IEEE Trans Vis Comput Graphics 18 (12):2431–2440. Online. Available:

    Article  Google Scholar 

  37. Sharma T, Fragkoulis M, Spinellis D (2016) Does your configuration code smell?. In: Proceedings of the 13th international conference on mining software repositories, ser. MSR’16, pp 189–200

  38. Sommerville I (2010) Software engineering, 9th edn. Addison-Wesley Publishing Company, USA

    Google Scholar 

  39. Wagner S, Fernández DM (2015) Analysing text in software projects. arXiv:1612.00164

    Google Scholar 

  40. Wettel R, Lanza M, Robbes R (2011) Software systems as cities: a controlled experiment. In: 2011 33rd international conference on software engineering (ICSE), pp 551–560

  41. Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in software engineering: an introduction. Kluwer Academic Publishers, Norwell

    Google Scholar 

  42. Xu T, Zhou Y (2015) Systems approaches to tackling configuration errors: a survey. ACM Computing Surveys 47(4):70:1–70:41

    Article  Google Scholar 

  43. Xu T, Jin L, Fan X, Zhou Y, Pasupathy S, Talwadker R (2015) Hey, you have given me too many knobs!: Understanding and dealing with over-designed configuration in system software. In: Proceedings of the 10th joint meeting on foundations of software engineering, ser. ESEC/FSE’2015, NY, USA, pp 307–319

  44. Xu T, Zhang J, Huang P, Zheng J, Sheng T, Yuan D, Zhou Y, Pasupathy S (2013) Do not blame users for misconfigurations. In: Proceedings of the 24 symposium on operating systems principles, pp 244–259

  45. Yin Z, Ma X, Zheng J, Zhou Y, Bairavasundaram LN, Pasupathy S (2011) An empirical study on configuration errors in commercial and open source systems. In: Proceedings of the twenty-third symposium on operating systems principles, ser. SOSP’11, pp 159–172

  46. Zhang S, Ernst M (2013) Automated diagnosis of software configuration errors. Piscataway, NJ, USA, pp 312–21, software configuration error diagnosis;ConfDiagnoser;configuration error identification;static analysis;dynamic profiling;statistical analysis;noncrashing configuration errors;configurable software system;Java;. Online. Available: https://doi.org/10.1109/ICSE.2013.6606577

  47. Zhang S, Ernst M (2014) Which configuration option should i change?. In: Proceedings of the 36th international conference on software engineering, ser. ICSE’14, NY, USA, pp 152–163

Download references

Author information



Corresponding author

Correspondence to Mohammed Sayagh.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by: Romain Robbes

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sayagh, M., Kerzazi, N., Petrillo, F. et al. What should your run-time configuration framework do to help developers?. Empir Software Eng 25, 1259–1293 (2020). https://doi.org/10.1007/s10664-019-09790-x

Download citation


  • Run-time software configuration
  • Configuration engineering
  • Empirical study
  • User study