Software Quality Journal

, Volume 24, Issue 3, pp 755–795 | Cite as

A systematic approach to evaluating domain-specific modeling language environments for multi-agent systems

  • Moharram Challenger
  • Geylani Kardas
  • Bedir Tekinerdogan


Multi-agent systems (MASs) include multiple interacting agents within an environment to provide a solution for complex systems that cannot be easily solved with individual agents or monolithic systems. However, the development of MASs is not trivial due to the various agent properties such as autonomy, responsiveness, and proactiveness, and the need for realization of the many different agent interactions. To support the development of MASs various domain-specific modeling languages (DSMLs) have been introduced that provide a declarative approach for modeling and supporting the generation of agent-based systems. To be effective, the proposed DSMLs need to meet the various stakeholder concerns and the related quality criteria for the corresponding MASs. Unfortunately, very often the evaluation of the DSML is completely missing or has been carried out in idiosyncratic approach. If the DSMLs are not well defined, then implicitly this will have an impact on the quality of the MASs. In this paper, we present an evaluation framework and systematic approach for assessing existing or newly defined DSMLs for MASs. The evaluation is specific for MAS DSMLs and targets both the language and the corresponding tools. To illustrate the evaluation approach, we first present SEA_ML, which is a model-driven MAS DSML for supporting the modeling and generation of agent-based systems. The evaluation of SEA_ML is based on a multi-case study research approach and provides both qualitative evaluation and quantitative analysis. We report on the lessons learned considering the adoption of the evaluation approach as well as the SEA_ML for supporting the generation of agent-based systems.


Domain-specific modeling language (DSML) Evaluation framework Evaluation of DSMLs Multi-agent system (MAS) Software agent SEA_ML 


  1. Abran, A. (2010). Software metrics and software metrology. New Jersey: Wiley and IEEE-CS Press.CrossRefMATHGoogle Scholar
  2. AOS. (2001). Agent Oriented Software Pty., Ltd. JACK Environment. Last Accessed August 2015.
  3. Barisic, A. (2013). Evaluating the quality in use of domain-specific languages in an agile way. In Proceedings of the doctoral symposium at 16th international conference on model driven engineering languages and systems (MODELS), September, 2013, CEUR-WS.Google Scholar
  4. Barisic, A., Amaral, V., Goulao, M., & Barroca, B. (2011a). How to reach a usable DSL? Moving toward a systematic evaluation. Electronic Communications of the EASST, 50, 1–12.Google Scholar
  5. Barisic, A., Amaral, V., Goulao, M., & Barroca, B. (2011b). Quality in use of domain-specific languages: A case study. In Proceedings of the 3rd ACM SIGPLAN workshop on evaluation and usability of programming languages and tools (pp. 65–72). PLATEAU’11, New York, USA.Google Scholar
  6. Barisic, A., Amaral, V., Goulao, M., & Barroca, B. (2012a). Evaluating the usability of domain-specific languages. In M. Mernik (Ed.), Formal and practical aspects of domain-specific languages: Recent developments (pp. 386–407). Hershey: IGI Global.Google Scholar
  7. Barisic, A., Amaral, V., Goulao, M., & Monteiro, M. P. (2012b). Patterns for evaluating usability of domain-specific languages. In Proceedings of the 19th conference on pattern languages of programs (PLoP), SPLASH.Google Scholar
  8. Basili, V., Shull, F., & Lanubile, F. (1999). Building knowledge through families of experiments. IEEE Transactions on Software Engineering, 25(4), 456–473.CrossRefGoogle Scholar
  9. Bayona-Oré, S., Calvo-Manzano, J. A., Cuevas, G., & San-Feliu, T. (2014). Critical success factors taxonomy for software process deployment. Software Quality Journal, 22(1), 21–48.CrossRefGoogle Scholar
  10. Bellifemine, F., Poggi, A., & Rimassa, G. (2001). Developing multi-agent systems with a FIPA-compliant agent framework. Software: Practice and Experience, 31(2), 103–128.MATHGoogle Scholar
  11. Berners-Lee, T., Hendler, J., & Lassila, O. (2001). The semantic web. Scientific American, 284(5), 34–43.CrossRefGoogle Scholar
  12. Bernon, C., Cossentino, M., Gleizes, M.-P., Turci, P., & Zambonelli, F. (2005). A study of some multi-agent meta-models. Lecture Notes in Computer Science, 3382, 62–77.CrossRefGoogle Scholar
  13. Beydoun, G., Low, G. C., Henderson-Sellers, B., Mouratidis, H., Gomez-Sanz, J. J., Pavon, J., & Gonzalez-Perez, C. (2009). FAML: A generic metamodel for MAS development. IEEE Transactions on Software Engineering, 35(6), 841–863.CrossRefGoogle Scholar
  14. Bobkowska, A. (2005). A methodology of visual modeling language evaluation. In SOFSEM 2005: Theory and practice of computer systems. Lecture Notes in Computer Science (Vol. 3381).Google Scholar
  15. Bresciani, P., Perini, A., Giorgini, P., Giunchiglia, F., & Mylopoulos, J. (2004). Tropos: An agent-oriented software development methodology. Autonomous Agents and Multi-Agent Systems, 8(3), 203–236.CrossRefMATHGoogle Scholar
  16. Carver, J., Jaccheri, L., Morasca, S., & Shull, F. (2010). A checklist for integrating student empirical studies with research and teaching goals. Empirical Software Engineering, 15(1), 35–59.CrossRefGoogle Scholar
  17. Challenger, M., Demirkol, S., Getir, S., Mernik, M., Kardas, G., & Kosar, T. (2014). On the use of a domain-specific modeling language in the development of multi-agent systems. Engineering Applications of Artificial Intelligence, 28, 111–141.CrossRefGoogle Scholar
  18. Challenger, M., Getir, S., Demirkol, S., & Kardas, G. (2011). A domain specific metamodel for semantic web enabled multi-agent systems. Lecture Notes in Business Information Processing, 83, 177–186.CrossRefGoogle Scholar
  19. Ciobanu, G., & Juravle, C. (2012). Flexible software architecture and language for mobile agents. Concurrency and Computation: Practice and Experience, 24(6), 559–571.CrossRefGoogle Scholar
  20. Clark, T., Evans, A., Sammut, P., & Willans, J. (2004). Language driven development and MDA. MDA Journal, 2–13. Last Accessed August 2015.
  21. CMU/SEI. (2010). Software Engineering Institute, CMMI for Acquisition, Version 1.3, Technical Report, CMU/SEI-2010-TR-032, Carnegie Mellon (November 2012).Google Scholar
  22. Demirkol, S., Challenger, M., Getir, S., Kosar, T., Kardas, G., & Mernik, M. (2013). A DSL for the development of software agents working within a semantic web environment. Computer Science and Information Systems, Special Issue on Advances in Model Driven Engineering, Languages and Agents, 10(4), 1525–1556.Google Scholar
  23. Demirkol, S., Getir, S., Challenger, M., & Kardas, G. (2011). Development of an agent based E-barter System. In: International symposium on innovations in intelligent systems and applications (INISTA) (pp. 193–198). IEEE Computer Society, Istanbul-Turkey.Google Scholar
  24. Demirli, E., & Tekinerdogan, B. (2011). Software language engineering of architectural viewpoints. In Proceeding of the 5th European Conference on Software Architecture (ECSA 2011), Lecture Notes on Computer Science (Vol. 6903, pp. 336–343).Google Scholar
  25. Eclipse. (2004). The Eclipse Foundation, Eclipse Integrated Development Environment (IDE). Last Accessed August 2015.
  26. Fister, I, Jr, Fister, I., Mernik, M., & Brest, J. (2011). Design and implementation of domain-specific language easytime. Computer Languages, Systems and Structures, 37(4), 151–167.CrossRefGoogle Scholar
  27. Fowler, M. (2011). Domain-specific languages. Boston: Addison-Wesley.Google Scholar
  28. Fuentes-Fernandez, R., Garcia-Magarino, I., Gomez-Rodriguez, A. M., & Gonzalez-Moreno, J. C. (2010). A technique for defining agent-oriented engineering processes with tool support. Engineering Applications of Artificial Intelligence, 23(3), 432–444.CrossRefGoogle Scholar
  29. Garcia, A. F., Sant’Anna, C., Figueiredo, E., Kulesza, U., Pereira de Lucena, C. J., & von Staa, A. (2005). Modularizing design patterns with aspects: A quantitative study. In Proceedings of the 4th international conference on Aspect-oriented software development (AOSD 2005) (pp. 3–14).Google Scholar
  30. Gascuena, J. M., Navarro, E., & Fernandez-Caballero, A. (2012). Model-driven engineering techniques for the development of multi-agent systems. Engineering Applications of Artificial Intelligence, 25(1), 159–173.CrossRefGoogle Scholar
  31. Getir, S., Challenger, M., Demirkol, S., & Kardas, G. (2012). The semantics of the interaction between agents and web services on the semantic web. In Proceedings of the 7th IEEE international workshop on engineering semantic agent systems (ESAS 2012), held in conjunction with the 36th IEEE signature conference on computers, software, and applications (COMPSAC 2012) (pp. 619–624).Google Scholar
  32. Getir, S., Challenger, M., & Kardas, G. (2014). The formal semantics of a domain-specific modeling language for semantic web enabled multi-agent systems. International Journal of Cooperative Information Systems, 23(3), 1450005.CrossRefGoogle Scholar
  33. Gray, J., Tolvanen, J.-P., Kelly, S., Gokhale, A., Neema, S., & Sprinkle, J. (2007). Domain-specific modeling. In P. A. Fishwick (Ed.), Handbook of dynamic system modeling (pp. 1–7). Boca Raton: CRC Press.Google Scholar
  34. Hahn, C. (2008). A domain specific language for multiagent systems. In Proceedings of the 7th autonomous agents and multiagent systems conference (AAMAS’08), Estoril, Portugal (pp. 233–240).Google Scholar
  35. Hahn, C., Madrigal-Mora, C., & Fischer, K. (2009). A platform-independent metamodel for multiagent systems. Autonomous Agents and Multi-Agent Systems, 18(2), 239–266.CrossRefGoogle Scholar
  36. Hahn, C., Nesbigall, S., Warwas, S., Zinnikus, I., Fischer, K., & Klusch, M. (2008). Integration of multiagent systems and semantic web services on a platform independent level. In Proceedings of IEEE/WIC/ACM international conference on web intelligence and intelligent agent technology (WI-IAT 2008), Sydney, Australia (pp. 200–206).Google Scholar
  37. Hannemann, J., & Kiczales, G. (2002). Design patterns implementation in Java and AspectJ. In Proceeding of object oriented programming systems languages and applications (OOPSLA’02) (pp. 161–173).Google Scholar
  38. ISO/IEC. (1998). International Organization for Standardization (Ed.), ISO/IEC TR 15504-2. Information Technology-Software Process Assessment—Part 2: A reference model for processes and process capability. Case Postale 56, CH-1211 Geneva, Switzerland.Google Scholar
  39. ISO/IEC. (2011). ISO/IEC 25010. Systems and software engineering systems and software quality requirements and evaluation (SQuaRE), System and software quality models, International Standards Organization/International Electrotechnical Commission.Google Scholar
  40. Jackson, D. (2002). Alloy: A lightweight object modeling notation. ACM Transactions on Software Engineering and Methodology, 11(2), 256–290.CrossRefGoogle Scholar
  41. Jouault, F., Allilaire, F., Bezivin, J., & Kurtev, I. (2008). ATL: A model transformation tool. Science of Computer Programming, 72(1–2), 31–39.MathSciNetCrossRefMATHGoogle Scholar
  42. Kahlaoui, A., Abran, A., & Lefebvre, E. (2008). DSML success factors and their assessment criteria. Metrics News, 13(1), 43–51.Google Scholar
  43. Kahraman, G., & Bilgen, S. (2013). A framework for qualitative assessment of domain-specific languages. Software & Systems Modeling. doi:10.1007/s10270-013-0387-8.Google Scholar
  44. Kardas, G., Challenger, M., Yildirim, S., & Yamuc, A. (2012). Design and implementation of a multi-agent stock trading system. Software Practice and Experience, 42, 1247–1273.CrossRefGoogle Scholar
  45. Kardas, G., Goknil, A., Dikenelli, O., & Topaloglu, N. Y. (2009). Model driven development of semantic web enabled multi-agent systems. International Journal of Cooperative Information Systems, 18(2), 261–308.CrossRefGoogle Scholar
  46. Kelly, S., & Tolvanen, J.-P. (2008). Domain-specific modeling- enabling full code generation. Los Alamitos: IEEE Computer Society Publications.CrossRefGoogle Scholar
  47. Khedker, U. P. (1997). What makes a good programming language? Technical Report TR-97-upk-1, Department of Computer Science, University of Pune.Google Scholar
  48. Kirstan, S., & Zimmermann, J. (2010). Evaluating costs and benefits of model-based development of embedded software systems in the car industry—Results of a qualitative Case Study. In Proceedings of ECMFA 2010 workshop C2 M:EEMDD-from Code Centric to Model Centric: Evaluating the effectiveness of MDD (pp 18–29).Google Scholar
  49. Kitchenham, B. A., Linkman, S., & Law, D. (1997). DESMET: A methodology for evaluating software engineering methods and tools. Computing and Control Engineering Journal, 8(3), 120–126.CrossRefGoogle Scholar
  50. Kosar, T., Mernik, M., & Carver, J. C. (2012). Program comprehension of domain-specific and general-purpose languages: Comparison using a family of experiments. Empircal Software Engineering, 17, 276–304.CrossRefGoogle Scholar
  51. Kosar, T., Oliveira, N., Mernik, M., Varanda Pereira, M. J., Crepinsek, M., Carneiro da Cruz, D., & Henriques, P. R. (2010). Comparing general-purpose and domain-specific languages: An empirical study. Computer Science and Information Systems, 7(2), 247–264.CrossRefGoogle Scholar
  52. Marin, B., Pastor, O., & Abran, A. (2010). Towards an accurate functional size measurement procedure for conceptual models in an MDA environment. Journal of Data and Knowledge Engineering, 69(5), 472–490.CrossRefGoogle Scholar
  53. Martin, D., Burstein, M., Hobbs, J., Lassila, O., McDermott, D., McIlraith, S., Narayanan, S., Paolucci, M., Parsia, B., Payne, T., Sirin, E., Srinivasan, N., & Sycara, K. (2004). OWL-S: Semantic markup for web services. W3C Member Submission. Last Accessed August 2015.
  54. Mernik, M., Heering, J., & Sloane, A. (2005). When and how to develop domain-specific languages. ACM Computing Surveys, 37(4), 316–344.CrossRefGoogle Scholar
  55. Mernik, M., & Zumer, V. (2005). Incremental programming language development. Computer Languages, Systems and Structures, 31(1), 1–16.CrossRefMATHGoogle Scholar
  56. MOFScript. (2005). The Eclipse Foundation, MOFScript model-to-text transformation language and tool. Last Accessed August 2015.
  57. Mohagheghi, P., & Dehlen, V. (2008). Where is the proof? A review of experiences from applying MDE in industry. In: I. Schieferdecker, & A. Hartman (Eds.) ECMDA-FA 2008. Lecture notes in computer science (Vol. 5095, pp. 432–443).Google Scholar
  58. Mohagheghi, P., Gilani, W., Stefanescu, A., & Fernandez, M. A. (2013). An empirical study of the state of the practice and acceptance of model-driven engineering in four industrial cases. Empirical Software Engineering, 18, 89–116.CrossRefGoogle Scholar
  59. Monteiro, M. P., & Fernandes, J. M. (2005). Towards a catalog of aspect oriented refactorings. In Proceedings of the 4th international conference on aspect oriented software development (AOSD’05) (pp. 111–122).Google Scholar
  60. OCL. (2012). Object Management Group, Object Constraint Language (OCL). Last Accessed August 2015.
  61. Odell, J., Nodine, M., & Levy, R. (2005). A metamodel for agents, roles, and groups. Agent-oriented software engineering. Lecture Notes in Computer Science, 3382, 78–92.CrossRefGoogle Scholar
  62. Oldevik, J., Neple, T., Gronmo, R., Aagedal, J., & Berre, A. J. (2005). Toward standardized model to text transformations. Lecture Notes in Computer Science, 3748, 239–253.CrossRefGoogle Scholar
  63. Omicini, A., Ricci, A., & Viroli, M. (2008). Artifacts in the A&A meta-model for multi-agent systems. Autonomous Agents and Multi-Agent Systems, 17(3), 432–456.CrossRefGoogle Scholar
  64. Padgham, L., & Winikoff, M. (2004). Developing intelligent agent systems: A practical guide. New York: Wiley.CrossRefMATHGoogle Scholar
  65. Paige, R. F., Ostroff, J. S., & Brooke, P. J. (2000). Principles for modeling language design. Information and Software Technology, 42(10), 665–675.CrossRefGoogle Scholar
  66. Pavon, J., Gomez-Sanz, J. J., & Fuentes, R. (2006). Model driven development of multi-agent systems. Lecture Notes in Computer Science, 4066, 284–298.CrossRefGoogle Scholar
  67. Pešović, D., Vidaković, M., Ivanović, M., Budimac, Z., & Vidaković, J. (2011). Usage of agents in document management. Computer Science and Information Systems, 8, 193–210.CrossRefGoogle Scholar
  68. Pokahr, A., Braubach, L., & Lamersdorf, W. (2005). JADEX: A BDI reasoning engine. In R. H. Bordini, et al. (Eds.), Multi-agent programming languages, platforms and applications (pp. 149–174). Berlin: Springer.CrossRefGoogle Scholar
  69. Rao, A., & Georgeff, M. (1995). BDI agents: From theory to practice. In Proceedings of the 1st international conference on multi-agent systems (ICMAS-95), San Francisco (pp. 312–319).Google Scholar
  70. Rougemaille, S., Migeon, F., Maurel, C., & Gleizes, M.-P. (2007). Model driven engineering for designing adaptive multi-agent systems. Lecture Notes in Artificial Intelligence, 4995, 318–332.Google Scholar
  71. Runeson, P., & Höst, M. (2009). Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering, 14, 131–164.CrossRefGoogle Scholar
  72. Russell, S. J., & Norvig, P. (2003). Artificial intelligence: A modern approach (2nd ed.). USA: Pearson Education.MATHGoogle Scholar
  73. Saritas, H. B., & Kardas, G. (2014). A model driven architecture for the development of smart card software. Computer Languages, Systems and Structures, 40(2), 53–72.CrossRefGoogle Scholar
  74. Schmid, K., & John, I. (2002). Developing, validating and evolving an approach to product line benefit and risk assessment. In Proceedings of the 28th Euro-micro conference (EUROMICRO’02) (pp. 272–283).Google Scholar
  75. Schmidt, D. C. (2006). Guest editor’s introduction: Model-driven engineering. IEEE Computer, 39(2), 25–31.CrossRefGoogle Scholar
  76. Shadbolt, N., Hall, W., & Berners-Lee, T. (2006). The semantic web revisited. IEEE Intelligent Systems, 21(3), 96–101.CrossRefGoogle Scholar
  77. Shih, D. H., Huang, S. Y., & Yen, D. C. (2005). A new reverse auction agent system for M-commerce using mobile agents. Computer Standards and Interfaces, 27, 383–395.CrossRefGoogle Scholar
  78. Smith, R. (1980). The Contract Net protocol: High Level Communication and Control in a Distributed Problem Solver. IEEE Transactions on Computers, 29(12), 1104–1113.CrossRefGoogle Scholar
  79. Sprinkle, J., Mernik, M., Tolvanen, J.-P., & Spinellis, D. (2009). Guest editors’ introduction: What kinds of nails need a domain-specific hammer? IEEE Software, 26(4), 15–18.CrossRefGoogle Scholar
  80. Sycara, K. (1998). Multiagent systems. AI Magazine, 19(4), 79–92.Google Scholar
  81. Sycara, K., Paolucci, M., Ankolekar, A., & Srinivasan, N. (2003). Automated discovery, interaction and composition of semantic web services. Journal of Web Semantics, 1(1), 27–46.CrossRefGoogle Scholar
  82. Tekinerdogan, B., Bozbey, S., Mester, Y., Turançiftci, E., & Alkışlar, L. (2011). An aspect-oriented tool framework for developing process-sensitive embedded user assistance systems. In Transactions on aspect-oriented software development VIII. Lecture notes computer science (Vol. 6580, pp. 196–220).Google Scholar
  83. Tekinerdogan, B., & Demirli, E. (2013). Evaluation framework for software architecture viewpoint languages. In Proceedings of 9th international ACM Sigsoft conference on the quality of software architectures (QoSA 2013) (pp. 89–98).Google Scholar
  84. van Deursen, A., Klint, P., & Visser, J. (2000). Domain-specific languages: An annotated bibliography. ACM Sigplan Notices, 35(6), 26–36.CrossRefGoogle Scholar
  85. Varanda-Pereira, M. J., Mernik, M., Da-Cruz, D., & Henriques, P. R. (2008). Program comprehension for domain-specific languages. Computer Science and Information Systems, 5(2), 1–17.CrossRefGoogle Scholar
  86. Warmer, J., & Kleppe, A. G. (1998). The object constraint language: Precise modeling with UML. Boston: Addison-Wesley.Google Scholar
  87. Wile, D. (2004). Lessons learned from real DSL experiments. Science of Computer Programming, 51, 265–290.MathSciNetCrossRefGoogle Scholar
  88. Wooldridge, M., & Jennings, N. R. (1995). Intelligent agents: Theory and practice. The Knowledge Engineering Review, 10(2), 115–152.CrossRefGoogle Scholar
  89. Zambonelli, F., Jennings, N. R., & Wooldrige, M. (2003). Developing multiagent systems: The Gaia methodology. ACM Transactions on Software Engineering and Methodologies, 12(3), 317–370.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Moharram Challenger
    • 1
  • Geylani Kardas
    • 1
  • Bedir Tekinerdogan
    • 2
  1. 1.International Computer InstituteEge UniversityIzmirTurkey
  2. 2.Information Technology GroupWageningen UniversityWageningenThe Netherlands

Personalised recommendations