Skip to main content
SpringerLink
Log in

Efficient regression testing of distributed real-time reactive systems in the context of model-driven development

  • Special Section Paper
  • Published:
Software and Systems Modeling Aims and scope Submit manuscript

Abstract

Regression testing is indispensable, especially for real-time distributed systems to ensure that existing functionalities are not affected by changes. Despite recent advances, regression testing for distributed systems remains challenging and extremely costly. Existing techniques often require running a failing system several times before detecting a regression. As a result, conventional approaches that use re-execution without considering the inherent non-determinism of distributed systems, and providing no (or low) control over execution are inadequate in many ways. In this paper, we present MRegTest, a replay-based regression testing framework in the context of model-driven development to facilitate deterministic replay of traces for detecting regressions while offering sufficient control for the purpose of testing over the execution of the changed system. The experimental results show that compared to the traditional approaches that annotate traces with timestamps and variable values MRegTest detects almost all regressions while reducing the size of the trace significantly and incurring similar runtime overhead.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. These tests haven not entirely been designed for IoT systems.

  2. Referring to an original model whose state machine has not been changed.

  3. Traces collected from executions of an instrumented base model.

  4. A video that demonstrates the tool: https://youtu.be/1PXjmKgadQI

  5. ‘x’ is a numerical variable and ‘exp’ is an expression that will be evaluated to a numerical value.

References

  1. Elbaum, S., Rothermel, G., Penix, J.: Techniques for improving regression testing in continuous integration development environments, ser. FSE,: new York, NY, USA: Assoc. Comput. Mach. 2014, 235–245 (2014)

    Google Scholar 

  2. Gabrielova, E.: End-to-end regression testing for distributed systems, ser. Middleware ’17. New York, NY, USA: Association for Computing Machinery, p. 9-12 (2017)

  3. Aumayr, D., Marr, S., Béra, C., Boix, E.G., Mössenböck, H.: Efficient and deterministic record & replay for actor languages, ser. ManLang ’18. New York, NY, USA: Association for Computing Machinery, (2018)

  4. Tveito, L., Johnsen, E.B., Schlatte, R.: Global reproducibility through local control for distributed active objects, In FASE, ser. Lecture Notes in Computer Science, vol. 12076. Springer, pp. 140–160 (2020)

  5. Lanese, I., Palacios, A., Vidal, G.: Causal-consistent replay debugging for message passing programs, In IFIP, ser Lecture Notes in Computer Science, Springer 11535, 167–184 (2019)

  6. Kazuhiro Shibanai, K., Watanabe, T.: Distributed functional reactive programming on actor-based runtime, In SIGPLAN. ACM, pp. 13–22 (2018)

  7. Sen, K., Kalasapur, S., Brutch, T., Gibbs, S.: Jalangi: a selective record-replay and dynamic analysis framework for JavaScript, ser. ESEC/FSE. New York, NY, USA: Association for Computing Machinery, p. 488-498 (2013)

  8. Leshed, G., Haber, E.M., Matthews, T., Lau, T.: Coscripter: Automating & sharing how-to knowledge in the enterprise, In SIGCHI, ser. CHI ’08. New York, NY, USA: association for Computing Machinery, p. 1719-1728 (2008)

  9. QF-Test website, https://www.qfs.de/en.html, retrieved (2021)

  10. Holmes, A., Kellogg, M.: Automating functional tests using selenium, In AGILE’06, (2006)

  11. Smartbear, “TestComplete,” https://smartbear.com/product/testcomplete/overview/, retrieved (2021)

  12. Kresse, A., Kruse, P.M.: Development and maintenance efforts testing graphical user interfaces: a comparison, ser. A-TEST. New York, NY, USA: association for Computing Machinery, p. 52-58 (2016)

  13. Sigelman, B.H., Barroso, L.A., Burrows, M., Stephenson, P., Plakal, M., Beaver, D., Jaspan, S., Shanbhag, C.: Dapper, a Large-Scale Distributed Systems Tracing Infrastructure. Google Inc, Tech. Rep. (2010)

    Google Scholar 

  14. Park, S., Lu, S., Zhou, Y.: Ctrigger: Exposing atomicity violation bugs from their hiding places, ser. ASPLOS XIV. New York, NY, USA: association for Computing Machinery, p. 25-36 (2009)

  15. Babaei, M., Dingel, J.: MRegTest repository, https://github.com/MajidGitHubRepos/MRegTest, retrieved (2021)

  16. Babaei, M., Bagherzadeh, M., Dingel, J.: Efficient reordering and replay of execution traces of distributed reactive systems in the context of model-driven development, In Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, ser. MODELS’20. New York, NY, USA: Association for Computing Machinery, p. 285-296 (2020)

  17. Babaei, M.: Regression testing of distributed real-time embedded systems in the context of model-driven development, PhD dissertation, Kingston, Ontario, Canada, (2021)

  18. Ural, H., Whittier, D.: Distributed testing without encountering controllability and observability problems. Inf. Process. Lett. 88(3), 133–141 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  19. Hierons, R.M., Merayo, M.G., Nunez, M.: Controllability through nondeterminism in distributed testing. In: Wotawa, F., Nica, M., Kushik, N. (eds.) Testing Software and Systems, pp. 89–105. Springer International Publishing, Cham (2016)

    Chapter  Google Scholar 

  20. Cacciari, L., Rafiq, O.: Controllability and observability in distributed testing. Inf. Softw. Technol. 41(11), 767–780 (1999)

    Article  Google Scholar 

  21. Terragni, V., Cheung, S., Zhang, C.: Recontest: effective regression testing of concurrent programs,’ In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, 1, pp. 246–256 (2015)

  22. Yu, T., Srisa-an, W., Rothermel, G.: SimRT: An automated framework to support regression testing for data races, ser. ICSE. New York, NY, USA: association for Computing Machinery, p. 48-59 (2014)

  23. Joshi, P., Ganai, M., Balakrishnan, G., Gupta, A., Papakonstantinou, N.:“SETSUDundefined: Perturbation-based testing framework for scalable distributed systems, ser. TRIOS. New York, NY, USA: association for Computing Machinery, (2013)

  24. Lima, B.: Automated scenario-based integration testing of time-constrained distributed systems, In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST), pp. 486–488 (2019)

  25. Bianchi, F.A., Margara, A., Pezzè, M.: A survey of recent trends in testing concurrent software systems. IEEE Trans. Software Eng. 44(8), 747–783 (2018)

    Article  Google Scholar 

  26. Torens, C., Ebrecht, L.: RemoteTest: a framework for testing distributed systems, pp. 441–446 (2010)

  27. Milka, G., Rzadca, K.: Dfuntest: a testing framework for distributed applications. CoRR, (2018)

  28. Foundation, E.: Eclipse Papyrus for real time (Papyrus-RT), https://www.eclipse.org/papyrus-rt, 2016, retrieved June 5, (2019)

  29. Jahed, Karim: Papyrus-RT Distribution, https://github.com/kjahed/papyrusrt-distribution, retrieved: 2019-8-10 (2019)

  30. Babaei, M., Bagherzadeh, M., Dingel, J.: Mreplayer: a trace replayer of distributed UML-RT models,” ser. MODELS’20. New York, NY, USA: association for Computing Machinery, (2020)

  31. The Object Management Group, UML Superstructure Specification. Version 2.5.1, http://www.omg.org/spec/UML/2.5.1/PDF, 2017, retrieved August 23, (2019)

  32. Stivan, G., Peruffo, A., Haller, P.: Akka.js: towards a portable actor runtime environment, ser. AGERE. New York, NY, USA: Association for Computing Machinery, p. 57-64 (2015)

  33. Bykov, S., Geller, A., Kliot, G., Larus, J.R., Pandya, R., Thelin, J.: Orleans: Cloud computing for everyone, ser. SOCC. New York, NY, USA: Association for Computing Machinery, (2011)

  34. McCaffrey, C.: Building the Halo 4 services with Orleans, 2015, presentation at QCon London

  35. Boer, F.D., Serbanescu, V., Hähnle, R., Henrio, L., Rochas, J., Din, C.C., Johnsen, E.B., Sirjani, M., Khamespanah, E., Fernandez-Reyes, K., Yang, A.M.: A survey of active object languages,” ACM Comput. Surv., 50(5), (2017)

  36. Rothermel, G., Harrold, M.J.: Analyzing regression test selection techniques. IEEE Trans. Software Eng. 22(8), 529–551 (1996)

    Article  Google Scholar 

  37. Hammoudi, M.: Regression testing of web applications using record/replay tools, ser. FSE. New York, NY, USA: Association for Computing Machinery, p. 1079-1081 (2016)

  38. Selic, B., Gullekson, G., Ward, P.T.: Real-Time Object-Oriented Modeling. John Wiley and Sons New York, 2, (1994)

  39. Posse, E., Dingel, J.: An executable formal semantics for UML-RT. Softw. Syst. Model. 15(1), 179–217 (2016)

    Article  Google Scholar 

  40. IBM, IBM RSARTE, https://www.ibm.com/developerworks/community/wikis, 2016, retrieved July 19, (2016)

  41. HCL, RealTime Software Tooling (RTist), https://www.hcltech.com/software/rtist, retrieved (2020)

  42. Eclipse Foundation, Eclipse eTrice Real-Time Modeling Tools, https://www.eclipse.org/etrice, (2020)

  43. Selic, B.: Using UML for modeling complex real-time systems, in Languages, compilers, and tools for embedded systems. Springer, pp. 250–260 (1998)

  44. Bagherzadeh, M.: Model-level debugging in the context of the model driven development, PhD dissertation, Kingston, Ontario, Canada, (2019)

  45. von der Beeck, M.: A formal semantics of UML-RT, In Model Driven Engineering Languages and Systems, O. Nierstrasz, J. Whittle, D. Harel, and G. Reggio, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 768–782 (2006)

  46. Mealy, G.H.: A method for synthesizing sequential circuits. Bell Syst. Tech. J. 34(5), 1045–1079 (1955)

    Article  MathSciNet  Google Scholar 

  47. Ahmadi, R.: Testing models of reactive systems, PhD dissertation, Kingston, Ontario, Canada, (2019)

  48. Esmaeilsabzali, S., Day, N., Atlee, J., Niu, J.: Deconstructing the semantics of big-step modelling languages. Requir. Eng. 15(06), 235–265 (2010)

    Article  Google Scholar 

  49. Parr, T.: Antlr (another tool for language recognition), https://www.antlr.org/, 2017, retrieved June 5, (2019)

  50. Milner, R.: A Calculus of Communicating Systems. Springer-Verlag, Berlin, Heidelberg (1982)

    MATH  Google Scholar 

  51. Hoare, C.A.R.: Communicating sequential processes, 21(8), 666-677, (1978) [Online]. Available: https://doi.org/10.1145/359576.359585

  52. Kolovos, D.S., Paige, R.F., Polack, F.A.: The Epsilon transformation language, In International Conference on Theory and Practice of Model Transformations. Springer, pp. 46–60 (2008)

  53. Dunfield, J.: Lecture notes in cisc 327 - software quality assurance, Fall (2018)

  54. Rothermel, G., Harrold, M.: Analyzing regression test selection techniques. IEEE Trans. Softw. Eng. 22(8), 529–551 (1996)

    Article  Google Scholar 

  55. Briand, L., Labiche, Y., He, S.: Automating regression test selection based on UML designs. Inf. Softw. Technol. 51(1), 16–30 (2009)

    Article  Google Scholar 

  56. Balasubramanian, J., Tambe, S., Lu, C., Gokhale, A., Gill, C., Schmidt, D.C.: Adaptive failover for real-time middleware with passive replication, In 15th IEEE Symposium on Real-Time and Embedded Technology and Applications. IEEE, pp. 118–127 (2009)

  57. Kahani, N., Hili, N., Cordy, J.R., Dingel, J.: Evaluation of UML-RT and Papyrus-RT for modelling self-adaptive systems, In Proceedings of the 9th International Workshop on Modelling in Software Engineering. IEEE Press, pp. 12–18 (2017)

  58. Swartout, W., Balzer, R.: On the inevitable intertwining of specification and implementation. Commun. ACM 25(7), 438–440 (1982)

    Article  Google Scholar 

  59. Magee, J., Kramer, J.: State Models and Java Programs. Wiley, (1999)

  60. Bagherzadeh, M., Hili, N., Dingel, J.: Model-level, platform-independent debugging in the context of the model-driven development of real-time systems, In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2017. New York, NY, USA: ACM, pp. 419–430 (2017)

  61. “ATM UML-RT model,” https://github.com/MajidGitHubRepos/atm_umlrt, retrieved (2021)

  62. Ekelund, E.D., Engström, E.: Efficient regression testing based on test history: An industrial evaluation. IEEE Int. Conf. Softw. Maint. Evol. (ICSME) 2015, 449–457 (2015)

    Google Scholar 

  63. Yu, T., Huang, Z., Wang, C.: ConTesa: directed test suite augmentation for concurrent software. IEEE Trans. Software Eng. 46(4), 405–419 (2020)

    Article  Google Scholar 

  64. Chen, Y., Zhang, S., Guo, Q., Li, L., Wu, R., Chen, T.: Deterministic replay: a survey. ACM Comput. Surv. 48(2), 171–1747 (2015)

    Article  Google Scholar 

  65. Hammoudi, M., Rothermel, G., Stocco, A.: WATERFALL: an incremental approach for repairing record-replay tests of web applications, ser. FSE. New York, NY, USA: Association for Computing Machinery, p. 751-762 (2016)

  66. Pal, D., Vain, J.: A systematic approach on modeling refinement and regression testing of real-time distributed systems. IFAC 52(13), 1091–1096 (2019)

    Google Scholar 

  67. Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: A survey. Softw. Test. Verif. Reliab. 22(2), 67–120 (2012)

    Article  Google Scholar 

  68. Biswas, S., Mall, R., Satpathy, M., Sukumaran, S.: Regression test selection techniques: a survey. Informatica (Slovenia) 35(3), 289–321 (2011)

    Google Scholar 

  69. Engström, E., Runeson, P., Skoglund, M.: A systematic review on regression test selection techniques, 52(1), 14-30 (2010)

  70. Orso, A., Shi, N., Harrold, M.J.: Scaling regression testing to large software systems. SIGSOFT Softw. Eng. Notes 29(6), 241–251 (2004)

    Article  Google Scholar 

  71. Zhang, L., Kim, M., Khurshid, S.: Localizing failure-inducing program edits based on spectrum information, In 2011 27th IEEE International Conference on Software Maintenance (ICSM), pp. 23–32 (2011)

  72. Gligoric, M., Eloussi, L., Marinov, D.: Practical regression test selection with dynamic file dependencies, ser. ISSTA,: New York, NY, USA: Association for Computing. Machinery 2015, 211–222 (2015)

  73. Legunsen, O., Hariri, F., Shi, A., Lu, Y., Zhang, L., Marinov, D.: An extensive study of static regression test selection in modern software evolution, ser. FSE. New York, NY, USA: Association for Computing Machinery, p. 583-594 (2016)

  74. Tools, G.E.: Build in the cloud: How the build system works, http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html, 2021, retrieved Jan 30, (2021)

  75. Zhang, L.: Hybrid regression test selection, In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE), pp. 199–209 (2018)

  76. Harrold, M.J., Jones, J. A., Li, T., Liang, D., Orso, A., Pennings, M., Sinha, S., Spoon, S. A., Gujarathi, A.: Regression test selection for java software, In Proceedings of the 16th ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, ser. OOPSLA ’01. New York, NY, USA: Association for Computing Machinery, p. 312-326 (2001)

  77. Orso, A., Shi, N., Harrold, M. J.: Scaling regression testing to large software systems, ser. SIGSOFT’04. New York, NY, USA: Association for Computing Machinery, p. 241-251 (2004)

  78. Legunsen, O., Shi, A., Marinov, D.: STARTS:STAtic regression test selection, pp. 949–954 (2017)

  79. Almasri, N., Tahat, L., Korel, B.: Automatically quantifying the impact of a change in systems (journal-first abstract), ser. ASE. New York, NY, USA: Association for Computing Machinery, p. 952 (2018)

  80. Chen, Y., Probert, R. L., Ural, H.: Model-based regression test suite generation using dependence analysis, ser. A-MOST ’07. New York, NY, USA: Association for Computing Machinery, p. 54-62 (2007)

  81. Biswas, S., Mall, R., Satpathy, M., Sukumaran, S.: A model-based regression test selection approach for embedded applications. SIGSOFT Softw. Eng. Notes 34(4), 1–9 (2009)

    Article  Google Scholar 

  82. Panigrahi, C. R., Mall, R.: Model-based regression test case prioritization, In Information Systems, Technology and Management, S. K. Prasad, H. M. Vin, S. Sahni, M. P. Jaiswal, and B. Thipakorn, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 380–385 (2010)

  83. Honfi, D., Molnár, G., Micskei, Z., Majzik, I.: Model-based regression testing of autonomous robots. In: Csöndes, T., Kovács, G., Réthy, G. (eds.) SDL 2017: Model-Driven Engineering for Future Internet, pp. 119–135. Springer International Publishing, Cham (2017)

    Chapter  Google Scholar 

  84. Biswas, S., Mall, R., Satpathy, M.: A regression test selection technique for embedded software, ACM Trans. Embed. Comput. Syst., 13(3), (2013)

  85. Zech, P., Kalb, P., Felderer, M., Atkinson, C., Breu, R.: Model-based regression testing by OCL. Int. J. Softw. Tools Technol. Transf. 19(1), 115–131 (2017)

    Article  Google Scholar 

  86. Zech, P., Felderer, M., Kalb, P., Breu, R.: A generic platform for model-based regression testing, In Leveraging Applications of Formal Methods, Verification and Validation. Technologies for Mastering Change, T. Margaria and B. Steffen, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 112–126 (2012)

  87. Pal, D., Vain, J.: Model based approach for testing: distributed real-time systems augmented with online monitors. In: Lupeikiene, A., Vasilecas, O., Dzemyda, G. (eds.) Databases and Information Systems, pp. 142–157. Springer International Publishing, Cham (2018)

  88. Korel, B., Tahat, L. H., Vaysburg, B.: Model based regression test reduction using dependence analysis, in International Conference on Software Maintenance, 2002. Proceedings., pp. 214–223 (2002)

  89. Chen, Y., Probert, R. L., Ural, H.: Regression test suite reduction using extended dependence analysis, ser. SOQUA’07. New York, NY, USA: Association for Computing Machinery, p. 62-69 (2007)

  90. Wu, Ye, Offutt, J.: Maintaining evolving component-based software with UML, In Seventh European Conference on Software Maintenance and Re-engineering, 2003. Proceedings., pp. 133–142 (2003)

  91. Briand, L., Labiche, Y., Soccar, G.: Automating impact analysis and regression test selection based on UML designs, In International Conference on Software Maintenance, 2002. Proceedings., pp. 252–261 (2002)

  92. Farooq, Q.-u.-a., Iqbal, M.Z.Z., Malik, Z. I., Nadeem, A.: An approach for selective state machine based regression testing, ser. A-MOST’07. New York, NY, USA: Association for Computing Machinery, p. 44-52 (2007)

  93. Traon, Y., Jeron, T., Jezequel, J., Morel, P.: Efficient object-oriented integration and regression testing. IEEE Trans. Reliab. 49(1), 12–25 (2000)

    Article  Google Scholar 

Download references

Acknowledgements

This work has been supported by Zeligsoft Ltd., Malina Software Corp., Cmind Inc., and the Natural Sciences and Engineering Research Council of Canada (NSERC).

Author information

Authors and Affiliations

  1. School of Computing Queen’s University, Kingston, Canada

    Majid Babaei

  2. Computing Queen’s University, Kingston, Canada

    Juergen Dingel

Authors
  1. Majid Babaei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juergen Dingel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Majid Babaei.

Additional information

Communicated by Shiva Nejati and Daniel Varro.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Babaei, M., Dingel, J. Efficient regression testing of distributed real-time reactive systems in the context of model-driven development. Softw Syst Model 22, 1565–1587 (2023). https://doi.org/10.1007/s10270-023-01086-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10270-023-01086-5

Keywords

Search

Navigation