Software & Systems Modeling

, Volume 11, Issue 1, pp 11–28 | Cite as

Supporting the internet-based evaluation of research software with cloud infrastructure

  • Pieter Van GorpEmail author
  • Paul Grefen
Open Access
Theme Section


Due to license restrictions and installation issues, it is often not feasible to experiment with software without making substantial investments. Especially in the case of legacy tools, it turns out that even free software is often too costly (i.e., time-consuming) to be installed for evaluating the quality of a research contribution. After organizing a series of events related to software modeling, we have constructed (and started to use) SHARE, a system for sharing practically any type of software artifact to reviewers and to other participants who have very limited time available. The system relies on cloud-computing technologies to provide online access to interactive environments containing all the tools, documentation, input and output models to reproduce alleged research results. The system also enables one to clone such an environment and add additional models or tools in order to extend a contribution or pinpoint a problem. In retrospect, we observe that the approach is not limited to software modeling and SHARE is in fact gaining acceptance in other fields already.


Reproducible research Model transformation Tool contest Peer review Cloud computing 



The authors wish to thank Stefan Blom and Axel Belinfante for their contributions to the SHARE source code. Additionally, we wish to thank Leon Osinski for organizing an excellent workshop related to reproducible research [37]. Finally, we wish to thank Marcel Hartgerink (from Wibu Systems) and Arnoud Engelfriet (from ICTRecht) for fruitful discussions related to the technical and legal aspects of software licenses.

Open Access

This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.


  1. 1.
    Agrawal A.: Graph rewriting and transformation (GReAT): a solution for the model integrated computing (MIC) bottleneck. ASE 0, 364 (2003)Google Scholar
  2. 2.
    Amazon.: Amazon Elastic Compute Cloud (Amazon EC2). Amazon. (2009)
  3. 3.
    Ambler, S.W.: A UML profile for data modeling. (2009)
  4. 4.
    Angelov S., Grefen P.: The 4W framework for B2B e-contracting. Int. J. Netw. Virtual Organ 2(1), 78–97 (2003)CrossRefGoogle Scholar
  5. 5.
    Barrett, D.J., Silverman, R.E., Byrnes, R.G.: SSH: The Secure Shell: The Definitive Guide, 2nd edn. O’Reilly and Associates (2005)Google Scholar
  6. 6.
    Bézivin, J., Dupé, G., Jouault, F., Pitette, G., Rougui, J.E.: First experiments with the ATL model transformation language: transforming XSLT into XQuery. In: OOPSLA—Generative Techniques in the Context of Model Driven Architecture. Anaheim, California (2003)Google Scholar
  7. 7.
    Bézivin, J., Jouault, F., Brunelière, H., Garces, K., Combemale, B., Sottet, J.-S., Kleiner, M., Doux, G., Tisi, M.: Zoos. Accessed Mar 2010
  8. 8.
    Biermann, E., Ehrig, K., Köhler, C., Kuhns, G., Taentzer, G., Weiss, E.: Emf model refactoring based on graph transformation concepts. ECEASST—Electronic Communications of the EASST, vol. 3 (2006)Google Scholar
  9. 9.
    Buckheit, J., Donoho, D.L.: Wavelets and Statistics, vol. 103. In: Wavelab and Reproducible Research, pp. 55–81. Springer-Verlag, New York (1995)Google Scholar
  10. 10.
    Buyya R., Yeo C.S., Venugopal S., Broberg J., Brandic I.: Cloud computing and emerging it platforms: vision, hype, and reality for delivering computing as the 5th utility. Future Gener. Comput. Syst. 25(6), 599–616 (2009)CrossRefGoogle Scholar
  11. 11.
    Chen J.-Y., Lin C.-H.: Scenario-based service specification and testing. J. Softw. Eng. Stud. 2, 69–80 (2007)Google Scholar
  12. 12.
    Claerbout, J.: Electronic documents give reproducible research a new meaning. In: Proc. Ann. Int. Mtg Soc. Expl. Geophys., pp. 601–604 (1992)Google Scholar
  13. 13.
    Cuban, L.: Oversold and Underused: Computers in the Classroom. Harvard University Press (2003)Google Scholar
  14. 14.
    Decker, G., Overdick, H., Weske, M.: Oryx—sharing conceptual models on the web. In: ER’08: Proceedings of the 27th International Conference on Conceptual Modeling, pp. 536–537. Springer-Verlag, Berlin (2008)Google Scholar
  15. 15.
    Deelman E., Gannon D., Shields M., Taylor I.: Workflows and e-science: an overview of workflow system features and capabilities. Future Gener. Comput. Syst. 25(5), 528–540 (2009)CrossRefGoogle Scholar
  16. 16.
    Descartes, R.: Discourse on Method. Jan Maire, Leiden (1637/1991). Available in The Philosophical Writings of Descartes (trans: Cottingham, J., Stoothoff, R., Murdoch, D., Kenny, A.). Cambridge University PressGoogle Scholar
  17. 17.
    Dittner R., Rule D.: The Best Damn Server Virtualization Book Period. Elsevier, Amsterdam (2007)Google Scholar
  18. 18.
    Donoho D., Maleki A., Rahman I., Shahram M., Stodden V.: Reproducible research in computational harmonic analysis. Comput. Sci. Eng. 11(1), 8–18 (2009)CrossRefGoogle Scholar
  19. 19.
    Fomel, S.: Madagascar—reproducible documents. Accessed Mar 2010
  20. 20.
    Fomel S., Claerbout J.: Guest editors’ introduction: reproducible research. Comput. Sci. Eng. 11(1), 5–7 (2009)CrossRefGoogle Scholar
  21. 21.
    Geiger, L., Zündorf, A.: eDOBS—graphical debugging for eclipse. In: 3rd International Workshop on Graph-Based Tools (GraBaTs) ICGT Workshop, Natal, Brasil (Sept 2006)Google Scholar
  22. 22.
    Halpin T., Bloesch A.: Data modeling in UML and ORM: a comparison. J. Database Manag. 10(4), 4–13 (1999)Google Scholar
  23. 23.
    Hong, L., Chi, E.H., Budiu, R., Pirolli, P., Nelson, L.: a low cost tagging system for foraging of web content. In: AVI ’08: Proceedings of the Working Conference on Advanced Visual Interfaces, pp. 65–72, New York, NY, USA. ACM (2008)Google Scholar
  24. 24.
    Jensen K., Kristensen L.: Coloured Petri Nets. Springer-Verlag, Berlin (2009)zbMATHCrossRefGoogle Scholar
  25. 25.
    Jensen K., Kristensen L.M., Wells L.: Coloured Petri Nets and CPN Tools for modelling and validation of concurrent systems. Int. J. Softw. Tools Technol. Transf. 9(3), 213–254 (2007)CrossRefGoogle Scholar
  26. 26.
    Keahey K., Tsugawa M., Matsunaga A., Fortes J.: Sky computing. IEEE Internet Comput. 13(5), 43–51 (2009)CrossRefGoogle Scholar
  27. 27.
    Kumar, S.: Wink. Accessed Mar 2010
  28. 28.
    Kurp P.: Green computing. Commun. ACM 51(10), 11–13 (2008)CrossRefGoogle Scholar
  29. 29.
    Lerdorf, R.J., Tatroe, K., Kaehms, B., McGredy, R.: Programming PHP. O’Reilly and Associates (2002)Google Scholar
  30. 30.
    Margaria T.: Web services-based tool-integration in the ETI platform. Softw. Syst. Model. 4(2), 141–156 (2005)CrossRefGoogle Scholar
  31. 31.
    Mylopoulos J., Chung L., Yu E.: From object-oriented to goal-oriented requirements analysis. Commun. ACM 42(1), 31–37 (1999)CrossRefGoogle Scholar
  32. 32.
    Nagel, R.: How to setup a jETI service provider (jETI server). Accessed June 2006
  33. 33.
    Newham, C., Rosenblatt, B.: Learning the Bash Shell, 2nd edn. O’Reilly and Associates (1998)Google Scholar
  34. 34.
    Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., Zagorodnov, D.: The eucalyptus open-source cloud-computing system. In: Proceedings of 9th IEEE International Symposium on Cluster Computing and the Grid (2009)Google Scholar
  35. 35.
    D.U. of Technology, E. U. of Technology, and U. of Twente.: 3TU. Data Centre. Accessed Mar 2010
  36. 36.
    Open Source Initiative.: Open source licenses by category. Accessed Mar 2010
  37. 37.
    Osinski, L.: Research data! Who cares? Accessed Sept 2009
  38. 38.
    Piwowar H.A., Day R.S., Fridsma D.B.: Sharing detailed research data is associated with increased citation rate. PLoS ONE 2(3), e308+ (2007)CrossRefGoogle Scholar
  39. 39.
    Rensink, A., Taentzer, G.: AGTiVE 2007 graph transformation tool contest. In: Schürr, et al. (eds.) Lecture Notes in Computer Science, vol. 5088. Springer, New York (2008)Google Scholar
  40. 40.
    Schürr, A., Nagl, M., Zündorf, A. (eds.): Applications of Graph Transformations with Industrial Relevance, Third International Symposium, AGTiVE 2007, Revised Selected and Invited Papers. Lecture Notes in Computer Science, vol. 5088. Springer, New York (2008)Google Scholar
  41. 41.
    Stardock Corporation. Fences. Accessed Mar 2010
  42. 42.
    Steffen B., Margaria T., Nagel R., Jörges S., Kubczak C.: Model-driven development with the jABC. In: Bin, E., Ziv, A., Ur, S. (eds) Haifa Verification Conference. Lecture Notes in Computer Science, vol. 4383, pp. 92–108. Springer, New York (2006)Google Scholar
  43. 43.
    Stodden V.: The legal framework for reproducible scientific research: licensing and copyright. Comput. Sci. Eng. 11(1), 35–40 (2009)CrossRefGoogle Scholar
  44. 44.
    R. D. S. T. S. Team: Top 10 RDP Protocol Misconceptions—part 2. Accessed Mar 2009
  45. 45.
    van den Brand, M.: Guest editor’s introduction: experimental software and toolkits (EST). Sci. Comput. Program. 69(1–3):1–2. Special issue on Exp. Softw. Toolkits (2007)Google Scholar
  46. 46.
    Van Gorp, P.: SHARE image with CPNtools and behavioral models of SHARE. Accessed Feb 2009
  47. 47.
    Van Gorp, P.: VRDP connection count bug? (wrong value of mcVRDPC lients). Accessed July 2009
  48. 48.
    Van Gorp, P.: SHARE group with basic operating system images\%20Systems. Accessed Mar 2010
  49. 49.
    Van Gorp, P., Blom, S., Belinfante, A.: SHARE–Sharing Hosted Autonomous Research Environments. (2009)
  50. 50.
    Van Gorp, P., Blom, S., Belinfante, A.: SHARE documentation. Accessed Mar 2010
  51. 51.
    Van Gorp P., Keller A., Janssens D.: Transformation language integration based on profiles and higher order transformations. In: Gasevic, D., Lämmel, R., Wyk, E.V. (eds) SLE. Lecture Notes in Computer Science, vol. 5452, pp. 208–226. Springer-Verlag, New York (2008)Google Scholar
  52. 52.
    Van Gorp, P., Rensink, A.: Call for papers: STTT special section on Graph-Based tool comparison. Accessed Nov 2008
  53. 53.
    Vandewalle P., Kovacevic J., Vetterli M.: Reproducible research in signal processing—what, why, and how. IEEE Signal Process. Mag. 26(3), 37–47 (2009)CrossRefGoogle Scholar
  54. 54.
    Vaquero L.M., Rodero-Merino L., Caceres J., Lindner M.: A break in the clouds: towards a cloud definition. SIGCOMM Comput. Commun. Rev. 39(1), 50–55 (2009)CrossRefGoogle Scholar
  55. 55.
    Varro, D., Asztalos, M., Bisztray, D., Boronat, A., Geiss, R., Gogolla, M., Gorp, P.V., Kniemeyer, O., Narayanan, A., Rencis, E., Weinell, E.: Graph transformation tools contest on the transformation of UML models to CSP. In: Schürr, et al. (eds.) Lecture Notes in Computer Science, vol. 5088. Springer, New York (2008)Google Scholar
  56. 56.
    VMware. VMware ESXi. Accessed Mar 2010
  57. 57.
    Wang L., von Laszewski G., Tao J., Kunze M.: Grid virtualization engine: design, implementation and evaluation. IEEE Syst. J. 3(4), 477–488 (2009)CrossRefGoogle Scholar
  58. 58.
    Wikipedia. Hwang woo-suk-wikipedia, the free encyclopedia. (2006)
  59. 59.
    Winter, R., Fischer, R.: Essential layers, artifacts, and dependencies of enterprise architecture. In: EDOCW ’06: Proceedings of the 10th IEEE on International Enterprise Distributed Object Computing Conference Workshops, p. 30, Washington, DC, USA. IEEE Computer Society (2006)Google Scholar
  60. 60.
    Zhu H.: A framework for service-oriented testing of web services. Comput. Softw. Appl. Conf. Annu. Int. 2, 145–150 (2006)Google Scholar

Copyright information

© The Author(s) 2010

Open AccessThis is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (, which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Authors and Affiliations

  1. 1.School of Industrial EngineeringEindhoven University of TechnologyEindhovenThe Netherlands

Personalised recommendations