Abstract
Due to license restrictions and installation issues, it is often not feasible to experiment with software without making substantial investments. Especially in the case of legacy tools, it turns out that even free software is often too costly (i.e., time-consuming) to be installed for evaluating the quality of a research contribution. After organizing a series of events related to software modeling, we have constructed (and started to use) SHARE, a system for sharing practically any type of software artifact to reviewers and to other participants who have very limited time available. The system relies on cloud-computing technologies to provide online access to interactive environments containing all the tools, documentation, input and output models to reproduce alleged research results. The system also enables one to clone such an environment and add additional models or tools in order to extend a contribution or pinpoint a problem. In retrospect, we observe that the approach is not limited to software modeling and SHARE is in fact gaining acceptance in other fields already.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Agrawal A.: Graph rewriting and transformation (GReAT): a solution for the model integrated computing (MIC) bottleneck. ASE 0, 364 (2003)
Amazon.: Amazon Elastic Compute Cloud (Amazon EC2). Amazon. http://aws.amazon.com/ec2/#pricing (2009)
Ambler, S.W.: A UML profile for data modeling. http://www.agiledata.org/essays/umlDataModelingProfile.html (2009)
Angelov S., Grefen P.: The 4W framework for B2B e-contracting. Int. J. Netw. Virtual Organ 2(1), 78–97 (2003)
Barrett, D.J., Silverman, R.E., Byrnes, R.G.: SSH: The Secure Shell: The Definitive Guide, 2nd edn. O’Reilly and Associates (2005)
Bézivin, J., Dupé, G., Jouault, F., Pitette, G., Rougui, J.E.: First experiments with the ATL model transformation language: transforming XSLT into XQuery. In: OOPSLA—Generative Techniques in the Context of Model Driven Architecture. Anaheim, California (2003)
Bézivin, J., Jouault, F., Brunelière, H., Garces, K., Combemale, B., Sottet, J.-S., Kleiner, M., Doux, G., Tisi, M.: Zoos. http://www.emn.fr/z-info/atlanmod/index.php/Zoos. Accessed Mar 2010
Biermann, E., Ehrig, K., Köhler, C., Kuhns, G., Taentzer, G., Weiss, E.: Emf model refactoring based on graph transformation concepts. ECEASST—Electronic Communications of the EASST, vol. 3 (2006)
Buckheit, J., Donoho, D.L.: Wavelets and Statistics, vol. 103. In: Wavelab and Reproducible Research, pp. 55–81. Springer-Verlag, New York (1995)
Buyya R., Yeo C.S., Venugopal S., Broberg J., Brandic I.: Cloud computing and emerging it platforms: vision, hype, and reality for delivering computing as the 5th utility. Future Gener. Comput. Syst. 25(6), 599–616 (2009)
Chen J.-Y., Lin C.-H.: Scenario-based service specification and testing. J. Softw. Eng. Stud. 2, 69–80 (2007)
Claerbout, J.: Electronic documents give reproducible research a new meaning. In: Proc. Ann. Int. Mtg Soc. Expl. Geophys., pp. 601–604 (1992)
Cuban, L.: Oversold and Underused: Computers in the Classroom. Harvard University Press (2003)
Decker, G., Overdick, H., Weske, M.: Oryx—sharing conceptual models on the web. In: ER’08: Proceedings of the 27th International Conference on Conceptual Modeling, pp. 536–537. Springer-Verlag, Berlin (2008)
Deelman E., Gannon D., Shields M., Taylor I.: Workflows and e-science: an overview of workflow system features and capabilities. Future Gener. Comput. Syst. 25(5), 528–540 (2009)
Descartes, R.: Discourse on Method. Jan Maire, Leiden (1637/1991). Available in The Philosophical Writings of Descartes (trans: Cottingham, J., Stoothoff, R., Murdoch, D., Kenny, A.). Cambridge University Press
Dittner R., Rule D.: The Best Damn Server Virtualization Book Period. Elsevier, Amsterdam (2007)
Donoho D., Maleki A., Rahman I., Shahram M., Stodden V.: Reproducible research in computational harmonic analysis. Comput. Sci. Eng. 11(1), 8–18 (2009)
Fomel, S.: Madagascar—reproducible documents. http://www.reproducibility.org/wiki/Reproducible_Documents. Accessed Mar 2010
Fomel S., Claerbout J.: Guest editors’ introduction: reproducible research. Comput. Sci. Eng. 11(1), 5–7 (2009)
Geiger, L., Zündorf, A.: eDOBS—graphical debugging for eclipse. In: 3rd International Workshop on Graph-Based Tools (GraBaTs) ICGT Workshop, Natal, Brasil (Sept 2006)
Halpin T., Bloesch A.: Data modeling in UML and ORM: a comparison. J. Database Manag. 10(4), 4–13 (1999)
Hong, L., Chi, E.H., Budiu, R., Pirolli, P., Nelson, L.: Spartag.us: a low cost tagging system for foraging of web content. In: AVI ’08: Proceedings of the Working Conference on Advanced Visual Interfaces, pp. 65–72, New York, NY, USA. ACM (2008)
Jensen K., Kristensen L.: Coloured Petri Nets. Springer-Verlag, Berlin (2009)
Jensen K., Kristensen L.M., Wells L.: Coloured Petri Nets and CPN Tools for modelling and validation of concurrent systems. Int. J. Softw. Tools Technol. Transf. 9(3), 213–254 (2007)
Keahey K., Tsugawa M., Matsunaga A., Fortes J.: Sky computing. IEEE Internet Comput. 13(5), 43–51 (2009)
Kumar, S.: Wink. http://www.debugmode.com/wink/. Accessed Mar 2010
Kurp P.: Green computing. Commun. ACM 51(10), 11–13 (2008)
Lerdorf, R.J., Tatroe, K., Kaehms, B., McGredy, R.: Programming PHP. O’Reilly and Associates (2002)
Margaria T.: Web services-based tool-integration in the ETI platform. Softw. Syst. Model. 4(2), 141–156 (2005)
Mylopoulos J., Chung L., Yu E.: From object-oriented to goal-oriented requirements analysis. Commun. ACM 42(1), 31–37 (1999)
Nagel, R.: How to setup a jETI service provider (jETI server). http://jabc.cs.tu-dortmund.de/manual/index.php/JETI#Additional_Documentation. Accessed June 2006
Newham, C., Rosenblatt, B.: Learning the Bash Shell, 2nd edn. O’Reilly and Associates (1998)
Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., Zagorodnov, D.: The eucalyptus open-source cloud-computing system. In: Proceedings of 9th IEEE International Symposium on Cluster Computing and the Grid (2009)
D.U. of Technology, E. U. of Technology, and U. of Twente.: 3TU. Data Centre. http://datacentrum.3tu.nl/. Accessed Mar 2010
Open Source Initiative.: Open source licenses by category. http://www.opensource.org/licenses/category. Accessed Mar 2010
Osinski, L.: Research data! Who cares? http://w3.tue.nl/nl/diensten/bib/over/minisymposium/. Accessed Sept 2009
Piwowar H.A., Day R.S., Fridsma D.B.: Sharing detailed research data is associated with increased citation rate. PLoS ONE 2(3), e308+ (2007)
Rensink, A., Taentzer, G.: AGTiVE 2007 graph transformation tool contest. In: Schürr, et al. (eds.) Lecture Notes in Computer Science, vol. 5088. Springer, New York (2008)
Schürr, A., Nagl, M., Zündorf, A. (eds.): Applications of Graph Transformations with Industrial Relevance, Third International Symposium, AGTiVE 2007, Revised Selected and Invited Papers. Lecture Notes in Computer Science, vol. 5088. Springer, New York (2008)
Stardock Corporation. Fences. http://en.wikipedia.org/wiki/Fences_(software). Accessed Mar 2010
Steffen B., Margaria T., Nagel R., Jörges S., Kubczak C.: Model-driven development with the jABC. In: Bin, E., Ziv, A., Ur, S. (eds) Haifa Verification Conference. Lecture Notes in Computer Science, vol. 4383, pp. 92–108. Springer, New York (2006)
Stodden V.: The legal framework for reproducible scientific research: licensing and copyright. Comput. Sci. Eng. 11(1), 35–40 (2009)
R. D. S. T. S. Team: Top 10 RDP Protocol Misconceptions—part 2. http://blogs.msdn.com/rds/archive/2009/03/12/top-10-rdp-protocol-misconceptions-part-2.aspx. Accessed Mar 2009
van den Brand, M.: Guest editor’s introduction: experimental software and toolkits (EST). Sci. Comput. Program. 69(1–3):1–2. Special issue on Exp. Softw. Toolkits (2007)
Van Gorp, P.: SHARE image with CPNtools and behavioral models of SHARE. http://is.ieis.tue.nl/staff/pvgorp/share/?page=ConfigureNewSession&vdi=ModelsOfSHARE_v1.vdi. Accessed Feb 2009
Van Gorp, P.: VRDP connection count bug? (wrong value of mcVRDPC lients). http://www.virtualbox.org/ticket/4655. Accessed July 2009
Van Gorp, P.: SHARE group with basic operating system images http://is.ieis.tue.nl/staff/pvgorp/share/?page=Signup&bundlename=Operating\%20Systems. Accessed Mar 2010
Van Gorp, P., Blom, S., Belinfante, A.: SHARE–Sharing Hosted Autonomous Research Environments. http://is.ieis.tue.nl/staff/pvgorp/share/ (2009)
Van Gorp, P., Blom, S., Belinfante, A.: SHARE documentation. http://fmt.cs.utwente.nl/redmine/wiki/5/SHARE. Accessed Mar 2010
Van Gorp P., Keller A., Janssens D.: Transformation language integration based on profiles and higher order transformations. In: Gasevic, D., Lämmel, R., Wyk, E.V. (eds) SLE. Lecture Notes in Computer Science, vol. 5452, pp. 208–226. Springer-Verlag, New York (2008)
Van Gorp, P., Rensink, A.: Call for papers: STTT special section on Graph-Based tool comparison. http://www.fots.ua.ac.be/events/grabats2008/sttt-section-cfp.pdf. Accessed Nov 2008
Vandewalle P., Kovacevic J., Vetterli M.: Reproducible research in signal processing—what, why, and how. IEEE Signal Process. Mag. 26(3), 37–47 (2009)
Vaquero L.M., Rodero-Merino L., Caceres J., Lindner M.: A break in the clouds: towards a cloud definition. SIGCOMM Comput. Commun. Rev. 39(1), 50–55 (2009)
Varro, D., Asztalos, M., Bisztray, D., Boronat, A., Geiss, R., Gogolla, M., Gorp, P.V., Kniemeyer, O., Narayanan, A., Rencis, E., Weinell, E.: Graph transformation tools contest on the transformation of UML models to CSP. In: Schürr, et al. (eds.) Lecture Notes in Computer Science, vol. 5088. Springer, New York (2008)
VMware. VMware ESXi. http://www.vmware.com/. Accessed Mar 2010
Wang L., von Laszewski G., Tao J., Kunze M.: Grid virtualization engine: design, implementation and evaluation. IEEE Syst. J. 3(4), 477–488 (2009)
Wikipedia. Hwang woo-suk-wikipedia, the free encyclopedia. http://en.wikipedia.org/wiki/Hwang_Woo-Suk (2006)
Winter, R., Fischer, R.: Essential layers, artifacts, and dependencies of enterprise architecture. In: EDOCW ’06: Proceedings of the 10th IEEE on International Enterprise Distributed Object Computing Conference Workshops, p. 30, Washington, DC, USA. IEEE Computer Society (2006)
Zhu H.: A framework for service-oriented testing of web services. Comput. Softw. Appl. Conf. Annu. Int. 2, 145–150 (2006)
Acknowledgments
The authors wish to thank Stefan Blom and Axel Belinfante for their contributions to the SHARE source code. Additionally, we wish to thank Leon Osinski for organizing an excellent workshop related to reproducible research [37]. Finally, we wish to thank Marcel Hartgerink (from Wibu Systems) and Arnoud Engelfriet (from ICTRecht) for fruitful discussions related to the technical and legal aspects of software licenses.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Tony Clark and Jorn Bettin.
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Van Gorp, P., Grefen, P. Supporting the internet-based evaluation of research software with cloud infrastructure. Softw Syst Model 11, 11–28 (2012). https://doi.org/10.1007/s10270-010-0163-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10270-010-0163-y