Abstract
Service providers use cloud to reduce the cost of their services and speed up their time to market. There is a huge gap between the environment where the services are developed and first tested, and the environment where they operate. Live testing is defined as testing a service in its production environment without causing any intolerable disruption to its usage. It is considered as one of the solutions to overcome the impact of the difference between the development environment and the production environment on the reliability of the test results. Test interferences are a major challenge for live testing as they may lead to the system mishandling the production traffic. Existing solutions to alleviate the risk of interferences have limited applicability. In this paper we propose a set of test methods, applicable in different situations, for live testing to improve the reliability of test results without causing any intolerable disturbance to the cloud services. To reduce the complexity of these test methods we define the concept of boundary environments and a set of coverage criteria to aim at during live testing.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
https://status.cloud.google.com/incident/compute/16012. Accessed 18 June 2020
https://status.cloud.google.com/incident/compute/15046. Accessed 18 June 2020
https://status.cloud.google.com/incident/appengine/19001. Accessed 18 June 2020
https://status.cloud.google.com/incident/appengine/16002. Accessed 18 June 2020
https://status.cloud.google.com/incident/cloud-networking/18012. Accessed 18 June 2020
Feitelson, D.G., Frachtenberg, E., Beck, K.L.: Development and deployment at Facebook. In: IEEE Internet Computing (2013)
20th November 2019 incident. https://status.azure.com/en-us/status/history/. Accessed 18 June 2020
Docker. https://www.docker.com. Accessed 18 June 2020
Linux containers. https://www.linuxcontainers.org. Accessed 18 June 2020
Kubernetes. https://www.kubernetes.io. Accessed 18 June 2020
Openstack. https://www.openstack.org. Accessed 18 June 2020
Brenner, D., Atkinson, C., Malaka, R., Merdes, M., Paech, B., Suliman, D.: Reducing verification effort in component-based software engineering through built-in testing. Inf. Syst. Front. 9(2–3), 151–162 (2007)
Sanchez, A.G.: Cost Optimizations in Runtime Testing and Diagnosis. Ph.D. Thesis. Delft University of Technology, September (2011)
Suliman, D., et al.: The MORABIT Approach to Runtime Component Testing. In the proceedings of the 30th Annual International Computer Software and Applications Conference COMPSAC 2006 (2006)
Lahami, M., Krichen, M., Jmaiel, M.: Safe and efficient runtime testing framework applied in dynamic and distributed systems. Sci. Comput. Program. 122, 1–28 (2016)
Lee, K., Sha, L.: A dependable online testing and upgrade architecture for real-time embedded systems. In: The proceedings of the 11th IEEE International Conference on Embedded and Real-Time Computing Systems and Applications RTCSA 2005 (2005)
Tang, C., et al.: Holistic configuration management at Facebook. In: The proceedings of the 25th Symposium on Operating Systems Principles SOSP 2015 (2015)
Blue-green. https://docs.cloudfoundry.org/devguide/deploy-apps/blue-green.html. Accessed 18 June 2020
Roush, E.T.: Cluster rolling upgrade using multiple version support. In: The Proceedings of IEEE International Conference on Cluster Computing ICCC 2001 (2001)
Das, T., Roush, E.T., Nandana, P.: Quantum leap cluster upgrade. In: The Proceedings of the 2nd Bangalore Annual Compute Conference COMPUTE 2009 (2009)
Ouyang, X., Ding, B., Wang, H.: Delayed switch: cloud service upgrade with low availability and capacity loss. In: The proceedings of the 5th International Conference on Software Engineering and Service Science ICSESS 2014 (2014)
de Lemos, R., et al.: Software engineering for self-adaptive systems: a second research roadmap. In: de Lemos, R., Giese, H., Müller, H.A., Shaw, M. (eds.) Software Engineering for Self-Adaptive Systems II. LNCS, vol. 7475, pp. 1–32. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-35813-5_1
ETSI ES 201 873-5 v4.8.1. Methods for Testing and Specification (MTS): The Testing and Test Control Notation version 3; Part 5: TTCN-3 Runtime Interface (TRI)
Jebbar, O., Khendek, F., Toeroe, M.: Architecture for the automation of live testing of cloud systems. In: The Proceedings of the 20th IEEE International Conference on Software Quality, Reliability, and Security IEEE QRS (2020)
Acknowledgement
This work has been partially supported by Natural Sciences and Engineering Research Council of Canada (NSERC) and Ericsson.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 IFIP International Federation for Information Processing
About this paper
Cite this paper
Jebbar, O., Khendek, F., Toeroe, M. (2020). Methods for Live Testing of Cloud Services. In: Casola, V., De Benedictis, A., Rak, M. (eds) Testing Software and Systems. ICTSS 2020. Lecture Notes in Computer Science(), vol 12543. Springer, Cham. https://doi.org/10.1007/978-3-030-64881-7_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-64881-7_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-64880-0
Online ISBN: 978-3-030-64881-7
eBook Packages: Computer ScienceComputer Science (R0)
