Empirical Software Engineering

, Volume 23, Issue 5, pp 2734–2763 | Cite as

Program comprehension of domain-specific and general-purpose languages: replication of a family of experiments using integrated development environments

  • Tomaž Kosar
  • Sašo Gaberc
  • Jeffrey C. Carver
  • Marjan Mernik


Domain-specific languages (DSLs) allow developers to write code at a higher level of abstraction compared with general-purpose languages (GPLs). Developers often use DSLs to reduce the complexity of GPLs. Our previous study found that developers performed program comprehension tasks more accurately and efficiently with DSLs than with corresponding APIs in GPLs. This study replicates our previous study to validate and extend the results when developers use IDEs to perform program comprehension tasks. We performed a dependent replication of a family of experiments. We made two specific changes to the original study: (1) participants used IDEs to perform the program comprehension tasks, to address a threat to validity in the original experiment and (2) each participant performed program comprehension tasks on either DSLs or GPLs, not both as in the original experiment. The results of the replication are consistent with and expanded the results of the original study. Developers are significantly more effective and efficient in tool-based program comprehension when using a DSL than when using a corresponding API in a GPL. The results indicate that, where a DSL is available, developers will perform program comprehension better using the DSL than when using the corresponding API in a GPL.


Domain-specific languages General-purpose languages Program comprehension Controlled experiment Replication 


  1. Albuquerque D, Cafeo B, Garcia A, Barbosa S, Abrahao S, Ribeiro A (2015) Quantifying usability of domain-specific languages: an empirical study on software maintenance. J Syst Softw 101:245–259CrossRefGoogle Scholar
  2. Baldassarre MT, Carver J, Dieste O, Juristo N (2014) Replication types: Towards a shared taxonomy. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering. ACM, New York, pp 18:1–18:4Google Scholar
  3. Barišić A, Amaral V, Goulão M (2017) Usability driven DSL development with USE-ME. Computer Languages, Systems & Structures. In: PressGoogle Scholar
  4. Bentley J (1986) Programming pearls: little languages. Commun ACM 29 (8):711–721CrossRefGoogle Scholar
  5. Carver JC (2010) Towards reporting guidelines for experimental replications: A proposal. In: 1st international workshop on replication in empirical software engineeringGoogle Scholar
  6. Carver JC, Syriani E, Gray J (2011) Assessing the frequency of empirical evaluation in software modeling research. In: Proceedings of the 1st international workshop on experiences and empirical studies in software modeling (EESSMOD), pp 28–37Google Scholar
  7. Chiş A, Denker M, Gîrba T, Nierstrasz O (2015) Practical domain-specific debuggers using the moldable debugger framework. Comput Lang Syst Struct 44:89–113Google Scholar
  8. Consel C, Marlet R (1998) Architecturing software using a methodology for language development. In: Proceedings of the 10th international symposium on programming language implementation and logic programming, vol 1490, pp 170–194Google Scholar
  9. Cornelissen B, Zaidman A, Van Deursen A (2011) A controlled experiment for program comprehension through trace visualization. IEEE Trans Softw Eng 37 (3):341–355CrossRefGoogle Scholar
  10. van Deursen A, Klint P (1998) Little languages: little maintenance. J Softw Maint 10(2):75–92CrossRefGoogle Scholar
  11. van Deursen A, Klint P (2002) Domain-specific language design requires feature descriptions. J Comput Inf Technol 10(1):1–17CrossRefzbMATHGoogle Scholar
  12. van Deursen A, Klint P, Visser J (2000) Domain-specific languages: an annotated bibliography. ACM SIGPLAN Not 35(6):26–36CrossRefGoogle Scholar
  13. Erdweg S, Giarrusso PG, Rendel T (2012) Language composition untangled. In: Proceedings of the twelfth workshop on language descriptions, tools, and applications (LDTA’12). ACM, New York, pp 7:1–7:8Google Scholar
  14. Gansner ER, Koutsofios E, North S (2009) Drawing graphs with dot. Tech. rep., AT&T Bell Laboratories, Murray Hill. Google Scholar
  15. Goulão M, Amaral V, Mernik M (2016) Quality in model-driven engineering: a tertiary study. Softw Qual J 24(3):601–633CrossRefGoogle Scholar
  16. Häser F, Felderer M, Breu R (2016) Is business domain language support beneficial for creating test case specifications: a controlled experiment. Inf Softw Technol 79:52–62CrossRefGoogle Scholar
  17. Hermans F, Pinzger M, Van Deursen A (2009) Domain-specific languages in practice: a user study on the success factors. In: Model driven engineering languages and systems, lecture notes in computer science, vol 5795. Springer, Berlin, pp 423–437Google Scholar
  18. Hevner AR, Linger RC, Collins Webb R, Pleszkoch M, Prowell S, Walton G (2005) The impact of function extraction technology on next-generation software engineering. Tech. Rep CMU/SEI-90-TR-21, Software Engineering Institute, Carnegie Mellon UniversityGoogle Scholar
  19. Hoisl B, Sobernig S, Strembeck M (2014) Comparing three notations for defining scenario-based model tests: a controlled experiment. In: Proceedings of the 9th international conference on the quality of information and communications technology (QUATIC), pp 95–104Google Scholar
  20. Hudak P (1998) Modular domain specific languages and tools. In: Proceedings of the 5th international conference on software reuse (JCSR ’98), IEEE Computer Society, pp 134–142Google Scholar
  21. Johanson AN, Hasselbring W (2016) Effectiveness and efficiency of a domain-specific language for high-performance marine ecosystem simulation: a controlled experiment. Empir Softw Eng 22(4):2206– 2236CrossRefGoogle Scholar
  22. Jung HW, Kim SG, Chung CS (2004) Measuring software product quality: a survey of iso/iec 9126. IEEE Softw 21(5):88–92CrossRefGoogle Scholar
  23. Kitchenham B (2008) The role of replications in empirical software engineering—a word of warning. Empir Softw Eng 13(2):219–221CrossRefGoogle Scholar
  24. Kosar T, Oliveira N, Mernik M, Varanda Pereira MJ, Črepinšek M, da Cruz D, Henriques PR (2010) Comparing general-purpose and domain-specific languages: an empirical study . Computer Science and Information Systems 7(2):247–264CrossRefGoogle Scholar
  25. Kosar T, Mernik M, Carver J (2012) Program comprehension of domain-specific and general-purpose languages: comparison using a family of experiments. Empir Softw Eng 17(3):276–304CrossRefGoogle Scholar
  26. Kosar T, Bohra S, Mernik M (2016) Domain-specific languages: a systematic mapping study. Inf Softw Technol 71:77–91CrossRefGoogle Scholar
  27. MacVittie LA (2006) XAML in a nutshell. O’Reilly Media, Inc, SebastopolGoogle Scholar
  28. Mauw S, Wiersma W, Willemse T (2004) Language-driven system design. Int J Softw Eng Knowl Eng 6(14):625–664CrossRefGoogle Scholar
  29. Meliá S, Cachero C, Hermida JM, Aparicio E (2016) Comparison of a textual versus a graphical notation for the maintainability of MDE domain models: an empirical pilot study. Softw Qual J 24(3):709–735CrossRefGoogle Scholar
  30. Mernik M (2013) An object-oriented approach to language compositions for software language engineering. J Syst Softw 86(9):2451–2464CrossRefGoogle Scholar
  31. Mernik M, Heering J, Sloane A (2005) When and how to develop domain-specific languages. ACM Comput Surv 37(4):316–344CrossRefGoogle Scholar
  32. Nugroho A (2009) Level of detail in UML models and its impact on model comprehension: a controlled experiment. Inf Softw Technol 51(12):1670–1685CrossRefGoogle Scholar
  33. Prähofer H, Schatz R, Wirth C, Hurnaus D, Mössenböck H (2013) Monaco – a domain-specific language solution for reactive process control programming with hierarchical components. Comput Lang Syst Struct 39(3):67–94Google Scholar
  34. Reis LV, Di Iorio VO, Bigonha RS (2015) An on-the-fly grammar modification mechanism for composing and defining extensible languages. Comput Lang Syst Struct 42:46–59zbMATHGoogle Scholar
  35. Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211–218CrossRefGoogle Scholar
  36. Sprinkle J, Mernik M, Tolvanen JP, Spinellis D (2009) Guest editors introduction: what kinds of nails need a domain-specific hammer? IEEE Softw 26 (4):15–18CrossRefGoogle Scholar
  37. Storey MA (2005) Theories, methods and tools in program comprehension: past, present and future. In: Proceedings of the 13th international workshop on program comprehension (IWPC’05), IEEE Computer Society, pp 181–191Google Scholar
  38. Umuhoza E, Brambilla M, Ripamonti D, Cabot J (2015) An empirical study on simplification of business process modeling languages. In: Proceedings of the 2015 ACM SIGPLAN international conference on software language engineering. ACM, New York, pp 13–24Google Scholar
  39. Varanda Pereira MJ, Fonseca J, Henriques PR (2016) Ontological approach for DSL development. Comput Lang Syst Struct 45:35–52Google Scholar
  40. Wile DS (2001) Supporting the DSL spectrum. J Comput Inf Technol 9(4):263–287CrossRefzbMATHGoogle Scholar
  41. Williams M (2002) Microsoft visual c# (core reference). Microsoft Press, RedmondGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Tomaž Kosar
    • 1
  • Sašo Gaberc
    • 1
  • Jeffrey C. Carver
    • 2
  • Marjan Mernik
    • 1
  1. 1.Faculty of Electrical Engineering and Computer ScienceUniversity of MariborMariborSlovenia
  2. 2.Department of Computer ScienceUniversity of AlabamaTuscaloosaUSA

Personalised recommendations