Model-Based Testing Without Models: The TodoMVC Case Study

Part of the Lecture Notes in Computer Science book series (LNCS, volume 10500)


Web applications define the interface to many of the businesses and services that we interact with and use on a daily basis. The technology stack enabling these applications is constantly changing and applications are accessed from a plethora of different devices. Automated testing of the behavior of applications is a promising strategy for reducing the manual effort that has to be spent on ensuring a consistent user experience across devices. Unfortunately, specifications or models of the desired behavior often do not exist. Model-based testing without models (aka learning-based testing) tries to overcome this hurdle by integrating model learning and model-based testing. In this paper, we sketch the ALEX tool [1, 11] for learning-based testing of web application and demonstrate its operation on benchmarks from the TodoMVC project. Our learning-based conformance analysis reveals that 7 of 27 Todo-apps exhibit behavior that differs from the majority of implementations.


Active automata learning Model-based testing Specification mining Conformance testing 


  1. 1.
    Automata learning experience (2016). Accessed 23 Oct 2016
  2. 2.
    Todomvc (2016). Accessed 23 Oct 2016
  3. 3.
    Todomvc specification (2016). Accessed 21 Oct 2016
  4. 4.
    Aarts, F., Jonsson, B., Uijen, J.: Generating models of infinite-state communication protocols using regular inference with abstraction. In: Petrenko, A., Simão, A., Maldonado, J.C. (eds.) ICTSS 2010. LNCS, vol. 6435, pp. 188–204. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-16573-3_14 CrossRefGoogle Scholar
  5. 5.
    Aarts, F., Kuppens, H., Tretmans, J., Vaandrager, F., Verwer, S.: Improving active mealy machine learning for protocol conformance testing. Mach. Learn. 96(1), 189–224 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Aarts, F., Kuppens, H., Tretmans, J., Vaandrager, F.W., Verwer, S.: Learning and testing the bounded retransmission protocol. In: Proceedings of the Eleventh International Conference on Grammatical Inference, ICGI 2012. University of Maryland, College Park, USA, 5–8 September 2012, pp. 4–18 (2012)Google Scholar
  7. 7.
    Alur, R., Madhusudan, P.: Visibly pushdown languages. In: Proceedings of the Thirty-sixth Annual ACM Symposium on Theory of Computing, STOC 2004, pp. 202–211. ACM, New York, NY, USA (2004)Google Scholar
  8. 8.
    Angluin, D.: Learning regular sets from queries and counter examples. Inf. Comput. 75(2), 87–106 (1987)CrossRefzbMATHGoogle Scholar
  9. 9.
    Argyros, G., Stais, I., Kiayias, A., Keromytis, A.D.: Back in black: towards formal, black box analysis of sanitizers and filters. In: 2016 IEEE Symposium on Security and Privacy (SP), pp. 91–109, May 2016Google Scholar
  10. 10.
    Avasarala, S.: Selenium WebDriver Practical Guide. Packt Publishing, Birmingham (2014). ISBN 9781782168850Google Scholar
  11. 11.
    Bainczyk, A., Schieweck, A., Isberner, M., Margaria, T., Neubauer, J., Steffen, B.: ALEX: mixed-mode learning of web applications at ease. In: Margaria, T., Steffen, B. (eds.) ISoLA 2016. LNCS, vol. 9953, pp. 655–671. Springer, Cham (2016). doi: 10.1007/978-3-319-47169-3_51 CrossRefGoogle Scholar
  12. 12.
    Belinfante, A.: JTorX: a tool for on-line model-driven test derivation and execution. In: Esparza, J., Majumdar, R. (eds.) TACAS 2010. LNCS, vol. 6015, pp. 266–270. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-12002-2_21 CrossRefGoogle Scholar
  13. 13.
    Berg, T., Grinchtein, O., Jonsson, B., Leucker, M., Raffelt, H., Steffen, B.: On the correspondence between conformance testing and regular inference. In: Cerioli, M. (ed.) FASE 2005. LNCS, vol. 3442, pp. 175–189. Springer, Heidelberg (2005). doi: 10.1007/978-3-540-31984-9_14 CrossRefGoogle Scholar
  14. 14.
    Bertolino, A., Calabrò, A., Merten, M., Steffen, B.: Never-stop learning: continuous validation of learned models for evolving systems through monitoring. ERCIM News 2012(88), 28–29 (2012)Google Scholar
  15. 15.
    Brinksma, E., Tretmans, J.: Testing transition systems: an annotated bibliography. In: Cassez, F., Jard, C., Rozoy, B., Ryan, M.D. (eds.) MOVEP 2000. LNCS, vol. 2067, pp. 187–195. Springer, Heidelberg (2001). doi: 10.1007/3-540-45510-8_9 CrossRefGoogle Scholar
  16. 16.
    Cassel, S., Howar, F., Jonsson, B., Steffen, B.: Active learning for extended finite state machines. Formal Aspects Comput. 28(2), 233–263 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Choi, W., Necula, G., Sen, K.: Guided GUI testing of android apps with minimal restart and approximate learning. SIGPLAN Not. 48(10), 623–640 (2013)CrossRefGoogle Scholar
  18. 18.
    Chow, T.S.: Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. SE–4(3), 178–187 (1978)CrossRefzbMATHGoogle Scholar
  19. 19.
    Dinca, I., Ipate, F., Mierla, L., Stefanescu, A.: Learn and test for event-B – A rodin plugin. In: Derrick, J., Fitzgerald, J., Gnesi, S., Khurshid, S., Leuschel, M., Reeves, S., Riccobene, E. (eds.) ABZ 2012. LNCS, vol. 7316, pp. 361–364. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-30885-7_32 CrossRefGoogle Scholar
  20. 20.
    Drews, S., D’Antoni, L.: Learning symbolic automata. In: Legay, A., Margaria, T. (eds.) TACAS 2017. LNCS, vol. 10205, pp. 173–189. Springer, Heidelberg (2017). doi: 10.1007/978-3-662-54577-5_10 CrossRefGoogle Scholar
  21. 21.
    Flanagan, D.: JavaScript: The Definitive Guide: Activate Your Web Pages (Definitive Guides), 6th edn. O’Reilly Media, Sebastopol (2011)Google Scholar
  22. 22.
    Fujiwara, S., von Bochmann, G., Khendek, F., Amalou, M., Ghedamsi, A.: Test selection based on finite state models. IEEE Trans. Softw. Eng. 17(6), 591–603 (1991)CrossRefGoogle Scholar
  23. 23.
    Hagerer, A., Hungar, H., Niese, O., Steffen, B.: Model generation by moderated regular extrapolation. In: Kutsche, R.-D., Weber, H. (eds.) FASE 2002. LNCS, vol. 2306, pp. 80–95. Springer, Heidelberg (2002). doi: 10.1007/3-540-45923-5_6 CrossRefGoogle Scholar
  24. 24.
    Hagerer, A., Margaria, T., Niese, O., Steffen, B., Brune, G., Ide, H.-D.: Efficient regression testing of CTI-systems: testing a complex call-center solution. Ann. Rev. Commun. Int. Eng. Consortium (IEC) 55, 1033–1040 (2001)Google Scholar
  25. 25.
    Howar, F., Steffen, B., Merten, M.: Automata learning with automated alphabet abstraction refinement. In: Jhala, R., Schmidt, D. (eds.) VMCAI 2011. LNCS, vol. 6538, pp. 263–277. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-18275-4_19 CrossRefGoogle Scholar
  26. 26.
    Isberner, M.: Foundations of active automata learning: an algorithmic perspective. Ph.D. thesis. TU Dortmund University, October 2015Google Scholar
  27. 27.
    Isberner, M., Howar, F., Steffen, B.: Inferring automata with state-local alphabet abstractions. In: Brat, G., Rungta, N., Venet, A. (eds.) NFM 2013. LNCS, vol. 7871, pp. 124–138. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-38088-4_9 CrossRefGoogle Scholar
  28. 28.
    Isberner, M., Howar, F., Steffen, B.: Learning register automata: from languages to program structures. Mach. Learn. 96, 1–34 (2013)MathSciNetzbMATHGoogle Scholar
  29. 29.
    Isberner, M., Howar, F., Steffen, B.: The TTT algorithm: a redundancy-free approach to active automata learning. In: Bonakdarpour, B., Smolka, S.A. (eds.) RV 2014. LNCS, vol. 8734, pp. 307–322. Springer, Cham (2014). doi: 10.1007/978-3-319-11164-3_26 Google Scholar
  30. 30.
    Isberner, M., Howar, F., Steffen, B.: The open-source learnlib. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 487–495. Springer, Cham (2015). doi: 10.1007/978-3-319-21690-4_32 CrossRefGoogle Scholar
  31. 31.
    Maler, O., Mens, I.-E.: Learning regular languages over large alphabets. In: Ábrahám, E., Havelund, K. (eds.) TACAS 2014. LNCS, vol. 8413, pp. 485–499. Springer, Heidelberg (2014). doi: 10.1007/978-3-642-54862-8_41 CrossRefGoogle Scholar
  32. 32.
    Margaria, T., Steffen, B.: Simplicity as a driver for agile innovation. Computer 43(6), 90–92 (2010)CrossRefGoogle Scholar
  33. 33.
    Mealy, G.H.: A method for synthesizing sequential circuits. Bell Syst. Tech. J. 34(5), 1045–1079 (1955)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Meinke, K., Sindhu, M.A.: Lbtest: a learning-based testing tool for reactive systems. In: Sixth IEEE International Conference on Software Testing, Verification and Validation, ICST 2013, Luxembourg, 18–22 March 2013, pp. 447–454 (2013)Google Scholar
  35. 35.
    Merten, M., Steffen, B., Howar, F., Margaria, T.: Next generation learnlib. In: Abdulla, P.A., Leino, K.R.M. (eds.) TACAS 2011. LNCS, vol. 6605, pp. 220–223. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-19835-9_18 CrossRefGoogle Scholar
  36. 36.
    Raffelt, H., Merten, M., Steffen, B., Margaria, T.: Dynamic testing via automata learning. Int. J. Softw. Tools Technol. Trans. (STTT) 11(4), 307–324 (2009)CrossRefGoogle Scholar
  37. 37.
    Raffelt, H., Steffen, B., Berg, T., Margaria, T.: LearnLib: a framework for extrapolating behavioral models. Int. J. Softw. Tools Technol. Trans. (STTT) 11(5), 393–407 (2009)CrossRefGoogle Scholar
  38. 38.
    Shahbaz, M., Groz, R.: Analysis and testing of black-box component-based systems by inferring partial models. Softw. Test. Verif. Reliab. 24(4), 253–288 (2014)CrossRefGoogle Scholar
  39. 39.
    Steffen, B., Howar, F., Merten, M.: Introduction to active automata learning from a practical perspective. In: Bernardo, M., Issarny, V. (eds.) SFM 2011. LNCS, vol. 6659, pp. 256–296. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-21455-4_8 CrossRefGoogle Scholar
  40. 40.
    Timmer, M., Brinksma, E., Stoelinga, M.: Model-based testing. In: Software and Systems Safety - Specification and Verification, pp. 1–32. IOS Press (2011)Google Scholar
  41. 41.
    Tretmans, J.: Model-based testing and some steps towards test-based modelling. In: Bernardo, M., Issarny, V. (eds.) SFM 2011. LNCS, vol. 6659, pp. 297–326. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-21455-4_9 CrossRefGoogle Scholar
  42. 42.
    Tretmans, J., Brinksma, E.: Torx: automated model-based testing. In: Hartman, A., Dussa-Ziegler, K. (ed.) First European Conference on Model-Driven Software Engineering, pp. 31–43, December 2003Google Scholar
  43. 43.
    Vaandrager, F.: Model learning. Commun. ACM 60(2), 86–95 (2017)CrossRefGoogle Scholar
  44. 44.
    Windmüller, S., Neubauer, J., Steffen, B., Howar, F., Bauer, O.: Active continuous quality control. In: Proceedings of the 16th International ACM Sigsoft Symposium on Component-based Software Engineering, CBSE 2013, pp. 111–120. ACM New York, NY, USA (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Dortmund University of TechnologyDortmundGermany

Personalised recommendations