Predicting and Testing Latencies with Deep Learning: An IoT Case Study

  • Bernhard K. Aichernig
  • Franz Pernkopf
  • Richard SchumiEmail author
  • Andreas Wurm
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11823)


The Internet of things (IoT) is spreading into the everyday life of millions of people. However, the quality of the underlying communication technologies is still questionable. In this work, we are analysing the performance of an implementation of MQTT, which is a major communication protocol of the IoT. We perform model-based test-case generation to generate log data for training a neural network. This neural network is applied to predict latencies depending on different features, like the number of active clients. The predictions are integrated into our initial functional model, and we exploit the resulting timed model for statistical model checking. This allows us to answer questions about the expected performance for various usage scenarios. The benefit of our approach is that it enables a convenient extension of a functional model with timing aspects using deep learning. A comparison to our previous work with linear regression shows that deep learning needs less manual effort in data preprocessing and provides significantly better predictions.


Statistical model checking Model-based testing Neural networks Performance Latency Internet of things MQTT EMQ 


  1. 1.
    Agha, G., Palmskog, K.: A survey of statistical model checking. ACM Trans. Model. Comput. Simul. (TOMACS) 28(1), 6:1–6:39 (2018)MathSciNetGoogle Scholar
  2. 2.
    Aichernig, B.K., et al.: Learning and statistical model checking of system response times. Softw. Qual. J. 27, 757–795 (2019)Google Scholar
  3. 3.
    Aichernig, B.K., Kann, S., Schumi, R.: Statistical model checking of response times for different system deployments. In: Feng, X., Müller-Olm, M., Yang, Z. (eds.) SETTA 2018. LNCS, vol. 10998, pp. 153–169. Springer, Cham (2018). Scholar
  4. 4.
    Aichernig, B.K., Schumi, R.: Property-based testing with FsCheck by deriving properties from business rule models. In: ICSTW, pp. 219–228. IEEE (2016)Google Scholar
  5. 5.
    Aichernig, B.K., Schumi, R.: Property-based testing of web services by deriving properties from business-rule models. Softw. Syst. Model. 18, 889–911 (2019)Google Scholar
  6. 6.
    Aichernig, B.K., Schumi, R.: Statistical model checking meets property-based testing. In: ICST, pp. 390–400. IEEE (2017)Google Scholar
  7. 7.
    Aichernig, B.K., Schumi, R.: How fast is MQTT? In: McIver, A., Horvath, A. (eds.) QEST 2018. LNCS, vol. 11024, pp. 36–52. Springer, Cham (2018). Scholar
  8. 8.
    Alur, R., Dill, D.L.: A theory of timed automata. Theor. Comput. Sci. 126(2), 183–235 (1994)MathSciNetzbMATHGoogle Scholar
  9. 9.
    Arts, T.: On shrinking randomly generated load tests. In: Erlang 2014, pp. 25–31. ACM (2014)Google Scholar
  10. 10.
    Ballarini, P., Bertrand, N., Horváth, A., Paolieri, M., Vicario, E.: Transient analysis of networks of stochastic timed automata using stochastic state classes. In: Joshi, K., Siegle, M., Stoelinga, M., D’Argenio, P.R. (eds.) QEST 2013. LNCS, vol. 8054, pp. 355–371. Springer, Heidelberg (2013). Scholar
  11. 11.
    Banks, A., Gupta, R.: MQTT version 3.1.1. OASIS Standard, December 2014Google Scholar
  12. 12.
    Becker, S., Koziolek, H., Reussner, R.H.: The Palladio component model for model-driven performance prediction. J. Syst. Softw. 82(1), 3–22 (2009)Google Scholar
  13. 13.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, New York (2006)zbMATHGoogle Scholar
  14. 14.
    Book, M., Gruhn, V., Hülder, M., Köhler, A., Kriegel, A.: Cost and response time simulation for web-based applications on mobile channels. In: QSIC, pp. 83–90. IEEE (2005)Google Scholar
  15. 15.
    Bulychev, P.E., et al.: UPPAAL-SMC: statistical model checking for priced timed automata. In: QAPL. EPTCS, vol. 85, pp. 1–16. Open Publishing Association (2012)Google Scholar
  16. 16.
    Chen, X., Mohapatra, P., Chen, H.: An admission control scheme for predictable server response time for web accesses. In: WWW, pp. 545–554. ACM (2001)Google Scholar
  17. 17.
    Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
  18. 18.
    Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: ICFP, pp. 268–279. ACM (2000)Google Scholar
  19. 19.
    Collina, M., Corazza, G.E., Vanelli-Coralli, A.: Introducing the QEST broker: scaling the IoT by bridging MQTT and REST. In: PIMRC, pp. 36–41. IEEE (2012)Google Scholar
  20. 20.
    Deng, L., Yu, D.: Deep learning: methods and applications. Found. Trends Sig. Process. 7(3–4), 197–387 (2014)MathSciNetzbMATHGoogle Scholar
  21. 21.
    Draheim, D., Grundy, J.C., Hosking, J.G., Lutteroth, C., Weber, G.: Realistic load testing of web applications. In: CSMR, pp. 57–70. IEEE (2006)Google Scholar
  22. 22.
    Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS. JMLR Proceedings, vol. 9, pp. 249–256. (2010)Google Scholar
  23. 23.
    Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: AISTATS. JMLR Proceedings, vol. 15, pp. 315–323. (2011)Google Scholar
  24. 24.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)zbMATHGoogle Scholar
  25. 25.
    Grinchtein, O.: Learning of Timed Systems. Ph.D. thesis, Uppsala University (2008)Google Scholar
  26. 26.
    Hawkins, D.M.: The problem of overfitting. J. Chem. Inf. Model. 44(1), 1–12 (2004)MathSciNetGoogle Scholar
  27. 27.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)Google Scholar
  28. 28.
    Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963)MathSciNetzbMATHGoogle Scholar
  29. 29.
    Houimli, M., Kahloul, L., Benaoun, S.: Formal specification, verification and evaluation of the MQTT protocol in the Internet of Things. In: ICMIT, pp. 214–221. IEEE, December 2017Google Scholar
  30. 30.
    Hughes, J.: QuickCheck testing for fun and profit. In: Hanus, M. (ed.) PADL 2007. LNCS, vol. 4354, pp. 1–32. Springer, Heidelberg (2006). Scholar
  31. 31.
    Kalaji, A.S., Hierons, R.M., Swift, S.: Generating feasible transition paths for testing from an extended finite state machine. In: ICST, pp. 230–239. IEEE (2009)Google Scholar
  32. 32.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  33. 33.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)Google Scholar
  34. 34.
    Lee, S., Kim, H., Hong, D., Ju, H.: Correlation analysis of MQTT loss and delay according to QoS level. In: ICOIN, pp. 714–717. IEEE (2013)Google Scholar
  35. 35.
    Legay, A., Delahaye, B., Bensalem, S.: Statistical model checking: an overview. In: Barringer, H., et al. (eds.) RV 2010. LNCS, vol. 6418, pp. 122–135. Springer, Heidelberg (2010). Scholar
  36. 36.
    Legay, A., Sedwards, S.: On statistical model checking with PLASMA. In: TASE, pp. 139–145. IEEE (2014)Google Scholar
  37. 37.
    Lu, Y., Nolte, T., Bate, I., Cucu-Grosjean, L.: A statistical response-time analysis of real-time embedded systems. In: RTSS, pp. 351–362. IEEE (2012)Google Scholar
  38. 38.
    Nagelkerke, N.J.: A note on a general definition of the coefficient of determination. Biometrika 78(3), 691–692 (1991)MathSciNetzbMATHGoogle Scholar
  39. 39.
    Papadakis, M., Sagonas, K.: A PropEr integration of types and function specifications with property-based testing. In: Erlang 2011, pp. 39–50. ACM (2011)Google Scholar
  40. 40.
    Pearson, K.: Note on regression and inheritance in the case of two parents. Proc. R. Soc. London 58, 240–242 (1895)Google Scholar
  41. 41.
    Tyagi, R.S.: A comparative study of performance testing tools. Int. J. Adv. Res. Comput. Sci. Softw. Eng. IJARCSSE 3(5), 1300–1307 (2013)Google Scholar
  42. 42.
    Schmidt, J., Ghorbani, A., Hapfelmeier, A., Kramer, S.: Learning probabilistic real-time automata from multi-attribute event logs. Intell. Data Anal. 17(1), 93–123 (2013)Google Scholar
  43. 43.
    Schumi, R.: Predicting and testing system response-times with statistical model checking and property-based testing. Ph.D. thesis, Graz University of Technology (2018)Google Scholar
  44. 44.
    Schumi, R., Lang, P., Aichernig, B.K., Krenn, W., Schlick, R.: Checking response-time properties of web-service applications under stochastic user profiles. In: Yevtushenko, N., Cavalli, A.R., Yenigün, H. (eds.) ICTSS 2017. LNCS, vol. 10533, pp. 293–310. Springer, Cham (2017). Scholar
  45. 45.
    Tappler, M., Aichernig, B.K., Bloem, R.: Model-based testing IoT communication via active automata learning. In: ICST, pp. 276–287. IEEE (2017)Google Scholar
  46. 46.
    Thangavel, D., Ma, X., Valera, A.C., Tan, H., Tan, C.K.: Performance evaluation of MQTT and CoAP via a common middleware. In: ISSNIP, pp. 1–6. IEEE (2014)Google Scholar
  47. 47.
    Verwer, S., de Weerdt, M., Witteveen, C.: A likelihood-ratio test for identifying probabilistic deterministic real-time automata from positive data. In: Sempere, J.M., García, P. (eds.) ICGI 2010. LNCS (LNAI), vol. 6339, pp. 203–216. Springer, Heidelberg (2010). Scholar
  48. 48.
    Wald, A.: Sequential Analysis. Courier Corporation, North Chelmsford (1973)zbMATHGoogle Scholar
  49. 49.
    Wurm, A.: Predicting the latency of MQTT brokers using deep learning. Master’s thesis, Graz University of Technology (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Bernhard K. Aichernig
    • 1
  • Franz Pernkopf
    • 2
  • Richard Schumi
    • 1
    Email author
  • Andreas Wurm
    • 2
  1. 1.Institute of Software TechnologyGraz University of TechnologyGrazAustria
  2. 2.Signal Processing and Speech Communication LabGraz University of TechnologyGrazAustria

Personalised recommendations