Abstract
Multi-agent systems (MASs) include multiple interacting agents within an environment to provide a solution for complex systems that cannot be easily solved with individual agents or monolithic systems. However, the development of MASs is not trivial due to the various agent properties such as autonomy, responsiveness, and proactiveness, and the need for realization of the many different agent interactions. To support the development of MASs various domain-specific modeling languages (DSMLs) have been introduced that provide a declarative approach for modeling and supporting the generation of agent-based systems. To be effective, the proposed DSMLs need to meet the various stakeholder concerns and the related quality criteria for the corresponding MASs. Unfortunately, very often the evaluation of the DSML is completely missing or has been carried out in idiosyncratic approach. If the DSMLs are not well defined, then implicitly this will have an impact on the quality of the MASs. In this paper, we present an evaluation framework and systematic approach for assessing existing or newly defined DSMLs for MASs. The evaluation is specific for MAS DSMLs and targets both the language and the corresponding tools. To illustrate the evaluation approach, we first present SEA_ML, which is a model-driven MAS DSML for supporting the modeling and generation of agent-based systems. The evaluation of SEA_ML is based on a multi-case study research approach and provides both qualitative evaluation and quantitative analysis. We report on the lessons learned considering the adoption of the evaluation approach as well as the SEA_ML for supporting the generation of agent-based systems.
Similar content being viewed by others
Notes
We keep the descriptions of the case studies short in this paper in order to keep focus on the evaluation. However, detailed descriptions of each case study, their models according to the SEA_ML specifications with including all modeling and code generation tools of SEA_ML can be found online inside a bundle at: http://mas.ube.ege.edu.tr/downloads/sea_ml_bundle.zip.
References
Abran, A. (2010). Software metrics and software metrology. New Jersey: Wiley and IEEE-CS Press.
AOS. (2001). Agent Oriented Software Pty., Ltd. JACK Environment. http://www.aosgrp.com/products/jack/. Last Accessed August 2015.
Barisic, A. (2013). Evaluating the quality in use of domain-specific languages in an agile way. In Proceedings of the doctoral symposium at 16th international conference on model driven engineering languages and systems (MODELS), September, 2013, CEUR-WS.
Barisic, A., Amaral, V., Goulao, M., & Barroca, B. (2011a). How to reach a usable DSL? Moving toward a systematic evaluation. Electronic Communications of the EASST, 50, 1–12.
Barisic, A., Amaral, V., Goulao, M., & Barroca, B. (2011b). Quality in use of domain-specific languages: A case study. In Proceedings of the 3rd ACM SIGPLAN workshop on evaluation and usability of programming languages and tools (pp. 65–72). PLATEAU’11, New York, USA.
Barisic, A., Amaral, V., Goulao, M., & Barroca, B. (2012a). Evaluating the usability of domain-specific languages. In M. Mernik (Ed.), Formal and practical aspects of domain-specific languages: Recent developments (pp. 386–407). Hershey: IGI Global.
Barisic, A., Amaral, V., Goulao, M., & Monteiro, M. P. (2012b). Patterns for evaluating usability of domain-specific languages. In Proceedings of the 19th conference on pattern languages of programs (PLoP), SPLASH.
Basili, V., Shull, F., & Lanubile, F. (1999). Building knowledge through families of experiments. IEEE Transactions on Software Engineering, 25(4), 456–473.
Bayona-Oré, S., Calvo-Manzano, J. A., Cuevas, G., & San-Feliu, T. (2014). Critical success factors taxonomy for software process deployment. Software Quality Journal, 22(1), 21–48.
Bellifemine, F., Poggi, A., & Rimassa, G. (2001). Developing multi-agent systems with a FIPA-compliant agent framework. Software: Practice and Experience, 31(2), 103–128.
Berners-Lee, T., Hendler, J., & Lassila, O. (2001). The semantic web. Scientific American, 284(5), 34–43.
Bernon, C., Cossentino, M., Gleizes, M.-P., Turci, P., & Zambonelli, F. (2005). A study of some multi-agent meta-models. Lecture Notes in Computer Science, 3382, 62–77.
Beydoun, G., Low, G. C., Henderson-Sellers, B., Mouratidis, H., Gomez-Sanz, J. J., Pavon, J., & Gonzalez-Perez, C. (2009). FAML: A generic metamodel for MAS development. IEEE Transactions on Software Engineering, 35(6), 841–863.
Bobkowska, A. (2005). A methodology of visual modeling language evaluation. In SOFSEM 2005: Theory and practice of computer systems. Lecture Notes in Computer Science (Vol. 3381).
Bresciani, P., Perini, A., Giorgini, P., Giunchiglia, F., & Mylopoulos, J. (2004). Tropos: An agent-oriented software development methodology. Autonomous Agents and Multi-Agent Systems, 8(3), 203–236.
Carver, J., Jaccheri, L., Morasca, S., & Shull, F. (2010). A checklist for integrating student empirical studies with research and teaching goals. Empirical Software Engineering, 15(1), 35–59.
Challenger, M., Demirkol, S., Getir, S., Mernik, M., Kardas, G., & Kosar, T. (2014). On the use of a domain-specific modeling language in the development of multi-agent systems. Engineering Applications of Artificial Intelligence, 28, 111–141.
Challenger, M., Getir, S., Demirkol, S., & Kardas, G. (2011). A domain specific metamodel for semantic web enabled multi-agent systems. Lecture Notes in Business Information Processing, 83, 177–186.
Ciobanu, G., & Juravle, C. (2012). Flexible software architecture and language for mobile agents. Concurrency and Computation: Practice and Experience, 24(6), 559–571.
Clark, T., Evans, A., Sammut, P., & Willans, J. (2004). Language driven development and MDA. MDA Journal, 2–13. http://www.bptrends.com/bpt/wp-content/publicationfiles/10-04%20COL%20MDA%20-%20Frankel%20-%20Xactium.pdf. Last Accessed August 2015.
CMU/SEI. (2010). Software Engineering Institute, CMMI for Acquisition, Version 1.3, Technical Report, CMU/SEI-2010-TR-032, Carnegie Mellon (November 2012).
Demirkol, S., Challenger, M., Getir, S., Kosar, T., Kardas, G., & Mernik, M. (2013). A DSL for the development of software agents working within a semantic web environment. Computer Science and Information Systems, Special Issue on Advances in Model Driven Engineering, Languages and Agents, 10(4), 1525–1556.
Demirkol, S., Getir, S., Challenger, M., & Kardas, G. (2011). Development of an agent based E-barter System. In: International symposium on innovations in intelligent systems and applications (INISTA) (pp. 193–198). IEEE Computer Society, Istanbul-Turkey.
Demirli, E., & Tekinerdogan, B. (2011). Software language engineering of architectural viewpoints. In Proceeding of the 5th European Conference on Software Architecture (ECSA 2011), Lecture Notes on Computer Science (Vol. 6903, pp. 336–343).
Eclipse. (2004). The Eclipse Foundation, Eclipse Integrated Development Environment (IDE). http://www.eclipse.org/. Last Accessed August 2015.
Fister, I, Jr, Fister, I., Mernik, M., & Brest, J. (2011). Design and implementation of domain-specific language easytime. Computer Languages, Systems and Structures, 37(4), 151–167.
Fowler, M. (2011). Domain-specific languages. Boston: Addison-Wesley.
Fuentes-Fernandez, R., Garcia-Magarino, I., Gomez-Rodriguez, A. M., & Gonzalez-Moreno, J. C. (2010). A technique for defining agent-oriented engineering processes with tool support. Engineering Applications of Artificial Intelligence, 23(3), 432–444.
Garcia, A. F., Sant’Anna, C., Figueiredo, E., Kulesza, U., Pereira de Lucena, C. J., & von Staa, A. (2005). Modularizing design patterns with aspects: A quantitative study. In Proceedings of the 4th international conference on Aspect-oriented software development (AOSD 2005) (pp. 3–14).
Gascuena, J. M., Navarro, E., & Fernandez-Caballero, A. (2012). Model-driven engineering techniques for the development of multi-agent systems. Engineering Applications of Artificial Intelligence, 25(1), 159–173.
Getir, S., Challenger, M., Demirkol, S., & Kardas, G. (2012). The semantics of the interaction between agents and web services on the semantic web. In Proceedings of the 7th IEEE international workshop on engineering semantic agent systems (ESAS 2012), held in conjunction with the 36th IEEE signature conference on computers, software, and applications (COMPSAC 2012) (pp. 619–624).
Getir, S., Challenger, M., & Kardas, G. (2014). The formal semantics of a domain-specific modeling language for semantic web enabled multi-agent systems. International Journal of Cooperative Information Systems, 23(3), 1450005.
Gray, J., Tolvanen, J.-P., Kelly, S., Gokhale, A., Neema, S., & Sprinkle, J. (2007). Domain-specific modeling. In P. A. Fishwick (Ed.), Handbook of dynamic system modeling (pp. 1–7). Boca Raton: CRC Press.
Hahn, C. (2008). A domain specific language for multiagent systems. In Proceedings of the 7th autonomous agents and multiagent systems conference (AAMAS’08), Estoril, Portugal (pp. 233–240).
Hahn, C., Madrigal-Mora, C., & Fischer, K. (2009). A platform-independent metamodel for multiagent systems. Autonomous Agents and Multi-Agent Systems, 18(2), 239–266.
Hahn, C., Nesbigall, S., Warwas, S., Zinnikus, I., Fischer, K., & Klusch, M. (2008). Integration of multiagent systems and semantic web services on a platform independent level. In Proceedings of IEEE/WIC/ACM international conference on web intelligence and intelligent agent technology (WI-IAT 2008), Sydney, Australia (pp. 200–206).
Hannemann, J., & Kiczales, G. (2002). Design patterns implementation in Java and AspectJ. In Proceeding of object oriented programming systems languages and applications (OOPSLA’02) (pp. 161–173).
ISO/IEC. (1998). International Organization for Standardization (Ed.), ISO/IEC TR 15504-2. Information Technology-Software Process Assessment—Part 2: A reference model for processes and process capability. Case Postale 56, CH-1211 Geneva, Switzerland.
ISO/IEC. (2011). ISO/IEC 25010. Systems and software engineering systems and software quality requirements and evaluation (SQuaRE), System and software quality models, International Standards Organization/International Electrotechnical Commission.
Jackson, D. (2002). Alloy: A lightweight object modeling notation. ACM Transactions on Software Engineering and Methodology, 11(2), 256–290.
Jouault, F., Allilaire, F., Bezivin, J., & Kurtev, I. (2008). ATL: A model transformation tool. Science of Computer Programming, 72(1–2), 31–39.
Kahlaoui, A., Abran, A., & Lefebvre, E. (2008). DSML success factors and their assessment criteria. Metrics News, 13(1), 43–51.
Kahraman, G., & Bilgen, S. (2013). A framework for qualitative assessment of domain-specific languages. Software & Systems Modeling. doi:10.1007/s10270-013-0387-8.
Kardas, G., Challenger, M., Yildirim, S., & Yamuc, A. (2012). Design and implementation of a multi-agent stock trading system. Software Practice and Experience, 42, 1247–1273.
Kardas, G., Goknil, A., Dikenelli, O., & Topaloglu, N. Y. (2009). Model driven development of semantic web enabled multi-agent systems. International Journal of Cooperative Information Systems, 18(2), 261–308.
Kelly, S., & Tolvanen, J.-P. (2008). Domain-specific modeling- enabling full code generation. Los Alamitos: IEEE Computer Society Publications.
Khedker, U. P. (1997). What makes a good programming language? Technical Report TR-97-upk-1, Department of Computer Science, University of Pune.
Kirstan, S., & Zimmermann, J. (2010). Evaluating costs and benefits of model-based development of embedded software systems in the car industry—Results of a qualitative Case Study. In Proceedings of ECMFA 2010 workshop C2 M:EEMDD-from Code Centric to Model Centric: Evaluating the effectiveness of MDD (pp 18–29).
Kitchenham, B. A., Linkman, S., & Law, D. (1997). DESMET: A methodology for evaluating software engineering methods and tools. Computing and Control Engineering Journal, 8(3), 120–126.
Kosar, T., Mernik, M., & Carver, J. C. (2012). Program comprehension of domain-specific and general-purpose languages: Comparison using a family of experiments. Empircal Software Engineering, 17, 276–304.
Kosar, T., Oliveira, N., Mernik, M., Varanda Pereira, M. J., Crepinsek, M., Carneiro da Cruz, D., & Henriques, P. R. (2010). Comparing general-purpose and domain-specific languages: An empirical study. Computer Science and Information Systems, 7(2), 247–264.
Marin, B., Pastor, O., & Abran, A. (2010). Towards an accurate functional size measurement procedure for conceptual models in an MDA environment. Journal of Data and Knowledge Engineering, 69(5), 472–490.
Martin, D., Burstein, M., Hobbs, J., Lassila, O., McDermott, D., McIlraith, S., Narayanan, S., Paolucci, M., Parsia, B., Payne, T., Sirin, E., Srinivasan, N., & Sycara, K. (2004). OWL-S: Semantic markup for web services. W3C Member Submission. http://www.w3.org/Submission/OWL-S/. Last Accessed August 2015.
Mernik, M., Heering, J., & Sloane, A. (2005). When and how to develop domain-specific languages. ACM Computing Surveys, 37(4), 316–344.
Mernik, M., & Zumer, V. (2005). Incremental programming language development. Computer Languages, Systems and Structures, 31(1), 1–16.
MOFScript. (2005). The Eclipse Foundation, MOFScript model-to-text transformation language and tool. http://www.eclipse.org/gmt/mofscript/. Last Accessed August 2015.
Mohagheghi, P., & Dehlen, V. (2008). Where is the proof? A review of experiences from applying MDE in industry. In: I. Schieferdecker, & A. Hartman (Eds.) ECMDA-FA 2008. Lecture notes in computer science (Vol. 5095, pp. 432–443).
Mohagheghi, P., Gilani, W., Stefanescu, A., & Fernandez, M. A. (2013). An empirical study of the state of the practice and acceptance of model-driven engineering in four industrial cases. Empirical Software Engineering, 18, 89–116.
Monteiro, M. P., & Fernandes, J. M. (2005). Towards a catalog of aspect oriented refactorings. In Proceedings of the 4th international conference on aspect oriented software development (AOSD’05) (pp. 111–122).
OCL. (2012). Object Management Group, Object Constraint Language (OCL). http://www.omg.org/spec/OCL/2.3.1/. Last Accessed August 2015.
Odell, J., Nodine, M., & Levy, R. (2005). A metamodel for agents, roles, and groups. Agent-oriented software engineering. Lecture Notes in Computer Science, 3382, 78–92.
Oldevik, J., Neple, T., Gronmo, R., Aagedal, J., & Berre, A. J. (2005). Toward standardized model to text transformations. Lecture Notes in Computer Science, 3748, 239–253.
Omicini, A., Ricci, A., & Viroli, M. (2008). Artifacts in the A&A meta-model for multi-agent systems. Autonomous Agents and Multi-Agent Systems, 17(3), 432–456.
Padgham, L., & Winikoff, M. (2004). Developing intelligent agent systems: A practical guide. New York: Wiley.
Paige, R. F., Ostroff, J. S., & Brooke, P. J. (2000). Principles for modeling language design. Information and Software Technology, 42(10), 665–675.
Pavon, J., Gomez-Sanz, J. J., & Fuentes, R. (2006). Model driven development of multi-agent systems. Lecture Notes in Computer Science, 4066, 284–298.
Pešović, D., Vidaković, M., Ivanović, M., Budimac, Z., & Vidaković, J. (2011). Usage of agents in document management. Computer Science and Information Systems, 8, 193–210.
Pokahr, A., Braubach, L., & Lamersdorf, W. (2005). JADEX: A BDI reasoning engine. In R. H. Bordini, et al. (Eds.), Multi-agent programming languages, platforms and applications (pp. 149–174). Berlin: Springer.
Rao, A., & Georgeff, M. (1995). BDI agents: From theory to practice. In Proceedings of the 1st international conference on multi-agent systems (ICMAS-95), San Francisco (pp. 312–319).
Rougemaille, S., Migeon, F., Maurel, C., & Gleizes, M.-P. (2007). Model driven engineering for designing adaptive multi-agent systems. Lecture Notes in Artificial Intelligence, 4995, 318–332.
Runeson, P., & Höst, M. (2009). Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering, 14, 131–164.
Russell, S. J., & Norvig, P. (2003). Artificial intelligence: A modern approach (2nd ed.). USA: Pearson Education.
Saritas, H. B., & Kardas, G. (2014). A model driven architecture for the development of smart card software. Computer Languages, Systems and Structures, 40(2), 53–72.
Schmid, K., & John, I. (2002). Developing, validating and evolving an approach to product line benefit and risk assessment. In Proceedings of the 28th Euro-micro conference (EUROMICRO’02) (pp. 272–283).
Schmidt, D. C. (2006). Guest editor’s introduction: Model-driven engineering. IEEE Computer, 39(2), 25–31.
Shadbolt, N., Hall, W., & Berners-Lee, T. (2006). The semantic web revisited. IEEE Intelligent Systems, 21(3), 96–101.
Shih, D. H., Huang, S. Y., & Yen, D. C. (2005). A new reverse auction agent system for M-commerce using mobile agents. Computer Standards and Interfaces, 27, 383–395.
Smith, R. (1980). The Contract Net protocol: High Level Communication and Control in a Distributed Problem Solver. IEEE Transactions on Computers, 29(12), 1104–1113.
Sprinkle, J., Mernik, M., Tolvanen, J.-P., & Spinellis, D. (2009). Guest editors’ introduction: What kinds of nails need a domain-specific hammer? IEEE Software, 26(4), 15–18.
Sycara, K. (1998). Multiagent systems. AI Magazine, 19(4), 79–92.
Sycara, K., Paolucci, M., Ankolekar, A., & Srinivasan, N. (2003). Automated discovery, interaction and composition of semantic web services. Journal of Web Semantics, 1(1), 27–46.
Tekinerdogan, B., Bozbey, S., Mester, Y., Turançiftci, E., & Alkışlar, L. (2011). An aspect-oriented tool framework for developing process-sensitive embedded user assistance systems. In Transactions on aspect-oriented software development VIII. Lecture notes computer science (Vol. 6580, pp. 196–220).
Tekinerdogan, B., & Demirli, E. (2013). Evaluation framework for software architecture viewpoint languages. In Proceedings of 9th international ACM Sigsoft conference on the quality of software architectures (QoSA 2013) (pp. 89–98).
van Deursen, A., Klint, P., & Visser, J. (2000). Domain-specific languages: An annotated bibliography. ACM Sigplan Notices, 35(6), 26–36.
Varanda-Pereira, M. J., Mernik, M., Da-Cruz, D., & Henriques, P. R. (2008). Program comprehension for domain-specific languages. Computer Science and Information Systems, 5(2), 1–17.
Warmer, J., & Kleppe, A. G. (1998). The object constraint language: Precise modeling with UML. Boston: Addison-Wesley.
Wile, D. (2004). Lessons learned from real DSL experiments. Science of Computer Programming, 51, 265–290.
Wooldridge, M., & Jennings, N. R. (1995). Intelligent agents: Theory and practice. The Knowledge Engineering Review, 10(2), 115–152.
Zambonelli, F., Jennings, N. R., & Wooldrige, M. (2003). Developing multiagent systems: The Gaia methodology. ACM Transactions on Software Engineering and Methodologies, 12(3), 317–370.
Acknowledgments
This study is partially funded by the Scientific and Technological Research Council of Turkey (TUBITAK) Electric, Electronic and Informatics Research Group (EEEAG) under Grant 109E125. The authors wish to thank the former members of the SEA_ML project, Dr. Marjan Mernik, Dr. Tomaz Kosar, Sinem Getir and Sebla Demirkol for their valuable contributions on design and implementation of SEA_ML. Finally, the authors would like to thank the participants, various Ph.D. and M.Sc. students, who actively got involved in the language evaluation and helped to the realization of the empirical study presented in this paper.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Challenger, M., Kardas, G. & Tekinerdogan, B. A systematic approach to evaluating domain-specific modeling language environments for multi-agent systems. Software Qual J 24, 755–795 (2016). https://doi.org/10.1007/s11219-015-9291-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-015-9291-5