Abstract
Continuous integration practice mandates to continuously introduce incremental changes into code, but doing so may introduce new faults too. These faults could be detected automatically through regression testing, but this practice becomes prohibitive as the cost of executing the tests grows. This problem is preponderant in end-to-end testing where the whole system is requested for test execution. However, some of these test cases could be executed with fewer resources (e.g., memory, web services, computation, Cloud instances, among others), by deploying only the subsystems needed by each test. This paper is focused on the optimization of the resources employed in end-to-end testing by means of a resource-aware test orchestration technique in the context of continuous integration practices in the Cloud. The RETORCH approach proposes a novel way to identify the resources required by end-to-end test cases and to use this information to group together those tests requiring equivalent resources. Besides, the approach proposes to deploy the grouped tests in isolated and elastic environments, so that their execution can be scheduled in parallel on several machines. RETORCH is exemplified with a real-world application, and its performance evaluation shows promising savings in terms of resource usage and time.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Augusto, C., Morán, J., Bertolino, A., de la Riva, C., & Tuya, J. (2019). RETORCH: Resource-aware end-to-end test orchestration. In M. Piattini, P. Rupino da Cunha, I. García-Rodríguez de Guzmán, & R. Pérez-Castillo (Eds.), 12th International Conference on the Quality of Information and Communications Technology (QUATIC 2019) (p. 14). https://doi.org/10.1007/978-3-030-29238-6_22.
Bell, J., Kaiser, G., Melski, E., & Dattatreya, M. (2015). Efficient dependency detection for safe Java test acceleration. 2015 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE 2015 - Proceedings, 770–781. https://doi.org/10.1145/2786805.2786823.
Bertolino, A. (2007). Software testing research: Achievements, challenges, dreams. FoSE 2007: Future of Software Engineering, 85–103. https://doi.org/10.1109/FOSE.2007.25.
Bertolino, A., Calabró, A., De Angelis, G., Gallego, M., García, B., & Gortázar, F. (2018). When the testing gets tough, the tough get ElasTest. Proceedings - International Conference on Software Engineering (pp. 17–20). https://doi.org/10.1145/3183440.3183497.
Bertolino, A., de Angelis, G., Gallego, M., García, B., Gortázar, F., Lonetti, F., & Marchetti, E. (2019). A systematic review on cloud testing. ACM Computing Surveys, 52(5), 1–42. https://doi.org/10.1145/3331447.
Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. (2016). Borg, Omega, and Kubernetes. Communications of the ACM, 59(5), 50–57. https://doi.org/10.1145/2890784.
Casalicchio, E. (2017). Autonomic orchestration of containers: Problem definition and research challenges. In ValueTools 2016 - 10th EAI international conference on performance evaluation methodologies and tools (pp. 287–290). https://doi.org/10.4108/eai.25-10-2016.2266649.
Chakraborty, S. S., & Shah, V. (2011). Towards an approach and framework for test-execution plan derivation. In 2011 26th IEEE/ACM international conference on automated software engineering, ASE 2011, proceedings (pp. 488–491). https://doi.org/10.1109/ASE.2011.6100106.
De Brito, M. S., Hoque, S., Magedanz, T., Steinke, R., Willner, A., Nehls, D., et al. (2017). A service orchestration architecture for fog-enabled infrastructures. In 2017 2nd international conference on fog and mobile edge computing, FMEC 2017 (pp. 127–132). https://doi.org/10.1109/FMEC.2017.7946419.
Docker Inc. (2017). Overview of Docker compose | Docker documentation. Retrieved October 14, 2019, from Docker Inc. website: https://docs.docker.com/compose/.
Docker Inc. (2019). Swarm mode overview | Docker documentation. Retrieved October 15, 2019, from https://docs.docker.com/engine/swarm/.
Draft, W. (2014). TOSCA Simple Profile in YAML Version 1.0. (March), 1–83. Retrieved from http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.1/TOSCA-Simple-Profile-YAML-v1.1.html.
ElasTest Developers Team. (2017). ElasTest: Full-teaching. Retrieved October 28, 2019, from https://github.com/elastest/full-teaching.
ElasTest Developers Team. (2018). ElasTest: FullTeaching-experiment. Retrieved October 28, 2019, from https://github.com/elastest/full-teaching-experiment.
Engström, E., Skoglund, M., & Runeson, P. (2008). Empirical evaluations of regression test selection techniques: A systematic review. ESEM’08: Proceedings of the 2008 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, 22–31. https://doi.org/10.1145/1414004.1414011.
Esfahani, H., Fietz, J., Ke, Q., Kolomiets, A., Lan, E., Mavrinac, E., et al. (2016). CloudBuild: Microsoft’s distributed and caching build service. In Proceedings - international conference on software engineering (pp. 11–20). https://doi.org/10.1145/2889160.2889222.
Fitzgerald, B., & Stol, K. J. (2017). Continuous software engineering: A roadmap and agenda. Journal of Systems and Software, 123, 176–189. https://doi.org/10.1016/j.jss.2015.06.063.
Gambi, A., Gorla, A., & Zeller, A. (2017). O!Snap: Cost-efficient testing in the cloud. Proceedings - 10th IEEE international conference on software testing, verification and validation, ICST 2017, 454–459. https://doi.org/10.1109/ICST.2017.51.
Gambi, A., Bell, J., & Zeller, A. (2018). Practical test dependency detection. Proceedings - 2018 IEEE 11th international conference on software testing, verification and validation, ICST 2018, 1–11. https://doi.org/10.1109/ICST.2018.00011.
Garcia, B., Lonetti, F., Gallego, M., Miranda, B., Jimenez, E., De Angelis, G., … Marchetti, E. (2018). A proposal to orchestrate test cases. Proceedings - 2018 international conference on the quality of information and communications technology, QUATIC 2018, 38–46. https://doi.org/10.1109/QUATIC.2018.00016.
Giotis, K., Kryftis, Y., & Maglaris, V. (2015). Policy-based orchestration of NFV services in software-defined networks. 1st IEEE Conference on Network Softwarization: Software-Defined Infrastructures for Networks, Clouds, IoT and Services, NETSOFT 2015, 1–5. https://doi.org/10.1109/NETSOFT.2015.7116145.
Guha, S., Rastogi, R., & Shim, K. (2001). CURE: An efficient clustering algorithm for large databases. Information Systems, 26(1), 35–58. https://doi.org/10.1016/S0306-4379(01)00008-4.
Gyori, A., Shi, A., Hariri, F., & Marinov, D. (2015). Reliable testing: Detecting state-polluting tests to prevent test dependency. 2015 International Symposium on Software Testing and Analysis, ISSTA 2015 - Proceedings, 223–233. https://doi.org/10.1145/2771783.2771793.
Harman, M. (2011). Making the case for MORTO: Multi objective regression test optimization. Proceedings - 4th IEEE International Conference on Software Testing, Verification, and Validation Workshops, ICSTW 2011, 111–114. https://doi.org/10.1109/ICSTW.2011.60.
Herzig, K., Greiler, M., Czerwonka, J., & Murphy, B. (2015). The art of testing less without sacrificing quality. Proceedings - International Conference on Software Engineering, 1, 483–493. https://doi.org/10.1109/ICSE.2015.66.
Lachmann, R., Nieke, M., Seidl, C., Schaefer, I., & Schulze, S. (2017). System-level test case prioritization using machine learning. Proceedings - 2016 15th IEEE International Conference on Machine Learning and Applications, ICMLA 2016, 361–368. https://doi.org/10.1109/ICMLA.2016.163.
Liu, C. H., Chen, S. L., & Chen, W. K. (2017). Cost-benefit evaluation on parallel execution for improving test efficiency over cloud. Proceedings of the 2017 IEEE International Conference on Applied System Innovation: Applied System Innovation for Modern Technology, ICASI 2017, 199–202. https://doi.org/10.1109/ICASI.2017.7988384.
Memon, A., Gao, Z., Nguyen, B., Dhanda, S., Nickell, E., Siemborski, R., & Micco, J. (2017). Taming google-scale continuous testing. Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track, ICSE-SEIP 2017, 233–242. https://doi.org/10.1109/ICSE-SEIP.2017.16.
Meyer, M. (2014). Continuous integration and its tools. IEEE Software, 31(3), 14–16. https://doi.org/10.1109/MS.2014.58.
Microsoft. (n.d.). Orchestrator overview | Microsoft Docs. Retrieved October 15, 2019, from https://docs.microsoft.com/en-us/system-center/orchestrator/learn-about-orchestrator?view=sc-orch-2019.
Oracle. (2019). MySQL. Retrieved November 3, 2019, from https://www.mysql.com/.
Pérez, P. F. (2017). Fullteaching: A web application to make teaching online easy. Retrieved from https://github.com/pabloFuente/full-teaching.
Rothermel, G., Harrold, M. J., Von Ronne, J., & Hong, C. (2002). Empirical studies of test-suite reduction. Software Testing Verification and Reliability, 12(4), 219–249. https://doi.org/10.1002/stvr.256.
Shull, F., Basili, V., Boehm, B., Brown, A. W., Costa, P., Lindvall, M., … Vinter, O. (2002). What we have learned about fighting defects. Proceedings - International Software Metrics Symposium, 2002-Janua, 249–258. https://doi.org/10.1109/METRIC.2002.1011343.
Singh, S., & Chana, I. (2015). QoS-aware autonomic resource management in cloud computing: A systematic review. ACM Computing Surveys, 48(3), 1–46. https://doi.org/10.1145/2843889.
Technologies, K. (2014). Kurento. Retrieved from https://www.kurento.org/.
Toczé, K., & Nadjm-Tehrani, S. (2018, June 4). A taxonomy for management and optimization of multiple resources in edge computing. Wireless Communications and Mobile Computing, 2018, 1–23. https://doi.org/10.1155/2018/7476201.
Uberti, J., & Thatcher, P. (2018). WebRTC Home. Retrieved from https://webrtc.org/.
University, R. J. C. (2017). OpenVidu. Retrieved from https://openvidu.io/.
Velasquez, K., Abreu, D. P., Goncalves, D., Bittencourt, L., Curado, M., Monteiro, E., & Madeira, E. (2017). Service orchestration in fog environments. Proceedings - 2017 IEEE 5th International Conference on Future Internet of Things and Cloud, FiCloud 2017, 2017-Janua, 329–336. https://doi.org/10.1109/FiCloud.2017.49.
Velasquez, K., Abreu, D. P., Assis, M. R. M., Senna, C., Aranha, D. F., Bittencourt, L. F., Laranjeiro, N., Curado, M., Vieira, M., Monteiro, E., & Madeira, E. (2018). Fog orchestration for the internet of everything: State-of-the-art and research challenges. Journal of Internet Services and Applications, 9(1), 14–23. https://doi.org/10.1186/s13174-018-0086-3.
Wong, W. E., Morgan, J. R., London, S., & Mathur, A. P. (1998). Effect of test set minimization on fault detection effectiveness. Software - Practice and Experience, 28(4), 347–369. https://doi.org/10.1002/(SICI)1097-024X(19980410)28:4<347::AID-SPE145>3.0.CO;2-L.
Xie, P., & Yang, D. (2018). Research on scheduling of software cloud testing. 2017 International Conference on Computer Systems, Electronics and Control, ICCSEC 2017, 1311–1314. https://doi.org/10.1109/ICCSEC.2017.8446709.
Yoo, S., & Harman, M. (2012). Regression testing minimization, selection and prioritization: A survey. Software Testing Verification and Reliability, 22, 67–120. https://doi.org/10.1002/stv.430.
Yu, L., Su, Y., & Wang, Q. (2009). Scheduling test execution of WBEM applications. Proceedings - Asia-Pacific Software Engineering Conference, APSEC, 323–330. https://doi.org/10.1109/APSEC.2009.27.
Zhang, Z., Li, C., Tao, Y., Yangy, R., Tang, H., & Xu, J. (2014). Fuxi: A fault-tolerant resource management and job scheduling system at internet scale. Proceedings of the VLDB Endowment, 7(13), 1393–1404. https://doi.org/10.14778/2733004.2733012.
Funding
This work was supported in part by the Spanish Ministry of Economy and Competitiveness under TestEAMoS (TIN2016-76956-C3-1-R) project and ERDF funds and by the European Project ElasTest in the Horizon 2020 research and innovation program (GA No. 731535).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Augusto, C., Morán, J., Bertolino, A. et al. RETORCH: an approach for resource-aware orchestration of end-to-end test cases. Software Qual J 28, 1147–1171 (2020). https://doi.org/10.1007/s11219-020-09505-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-020-09505-2