Software Quality Journal

, Volume 27, Issue 2, pp 703–729 | Cite as

Testing TLS using planning-based combinatorial methods and execution framework

  • Dimitris E. SimosEmail author
  • Josip Bozic
  • Bernhard Garn
  • Manuel Leithner
  • Feng Duan
  • Kristoffer Kleine
  • Yu Lei
  • Franz Wotawa


The TLS protocol is the standard for secure Internet communication between two parties. Unfortunately, there have been recently successful attacks like DROWN, ROBOT, or BREACH that indicate the necessity for thoroughly testing TLS implementations. In our research work, we focus on automated test case generation and execution for the TLS security protocol, where the aim is to combine planning with combinatorial methods for providing test cases that ideally also reveal previously unknown attacks. This is made feasible by creating appropriate input parameter models for different messages that can appear in a TLS message sequence. In this paper, we present the resulting test case generation and execution framework together with the corresponding test oracle. Furthermore, we discuss in detail empirical results obtained via testing different TLS implementations.


Combinatorial testing Security testing Planning Security protocols TLS 



The research presented in the paper has been funded in part by the Austrian Research Promotion Agency (FFG) under grant 851205 (Security ProtocoL Interaction Testing in Practice - SPLIT) and the Austrian COMET Program (FFG).


  1. AlFardan, N., & Paterson, K.G. (2012). Plaintext-recovery attacks against datagram tls. In Network and distributed system security symposium (NDSS 2012).Google Scholar
  2. Berbecaru, D., & Lioy, A. (2007). On the robustness of applications based on the ssl and tls security protocol. In European Public key infrastructure workshop (pp. 248–264). Springer.Google Scholar
  3. Beurdouche, B., Bhargavan, K., Delignat-Lavaud, A., Fournet, C., Kohlweiss, M., Pironti, A., Strub, P.Y., Zinzindohoue, J.K. (2015). A messy state of the union: taming the composite state machines of tls. In Proceedings of the 36th IEEE symposium on security and privacy.Google Scholar
  4. Beurdouche, B., Delignat-Lavaud, A., Kobeissi, N., Pironti, A., Bhargavan, K. (2015). Flextls: a tool for testing tls implementations. In 9th USENIX Workshop on offensive technologies (WOOT’15).Google Scholar
  5. Bhargavan, K., & Leurent, G. (2016). Transcript collision attacks: breaking authentication in tls, ike, and ssh. In Network and distributed system security symposium–NDSS 2016.Google Scholar
  6. Bhargavan, K., Lavaud, A.D., Fournet, C., Pironti, A., Strub, P.Y. (2014). Triple handshakes and cookie cutters: breaking and fixing authentication over tls. In 2014 IEEE Symposium on security and privacy (SP) (pp. 98–113). IEEE.Google Scholar
  7. Blum, A., & Furst, M. (1995). Fast planning through planning graph analysis. In IJCAI95 (pp. 1636–1642).Google Scholar
  8. Bozic, J., & Wotawa, F. (2014). Plan it! automated security testing based on planning. In Proceedings of the 26th IFIP WG 6.1 international conference (ICTSS’14) (pp. 48–62).Google Scholar
  9. Bozic, J., & Wotawa, F. (2015). Purity: a planning-based security testing tool. In 2015 IEEE International conference on software quality, reliability and security-companion (QRS-C) (pp. 46–55).Google Scholar
  10. Bozic, J., Kleine, K., Simos, D.E., Wotawa, F. (2017). Planning-based security testing of the SSL/TLS protocol. In Proceedings of the IEEE international conference on software testing, verification and validation workshops (ICSTW).Google Scholar
  11. Brubaker, C., Jana, S., Ray, B., Khurshid, S., Shmatikov, V. (2014). Using frankencerts for automated adversarial testing of certificate validation in ssl/tls implementations. In Proceedings of the 2014 IEEE symposium on security and privacy.Google Scholar
  12. de Ruiter, J., & Poll, E. (2015). Protocol state fuzzing of tls implementations. In 24th USENIX Security Symposium (USENIX Security 15) (pp. 193–206).Google Scholar
  13. Dierks, T., & Rescorla, E. (2008). Rfc 5246: the transport layer security (tls) protocol. The Internet Engineering Task Force.Google Scholar
  14. Dowling, B., Fischlin, M., Günther, F., Stebila, D. (2015). A cryptographic analysis of the tls 1.3 handshake protocol candidates. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security (pp. 1197–1210). ACM.Google Scholar
  15. Duan, F., Lei, Y., Yu, L., Kacker, R.N., Kuhn, D.R. (2017). Optimizing ipog’s vertical growth with constraints based on hypergraph coloring. In 2017 IEEE International Conference on software testing, verification and validation workshops (ICSTW) (pp. 181–188). IEEE.Google Scholar
  16. Galler, S.J., Zehentner, C., Wotawa, F. (2010). Aiana: an ai planning system for test data generation. In 1st Workshop on testing object-oriented software systems (pp. 30–37).Google Scholar
  17. Hollenbeck, S. (2004). Transport layer security protocol compression methods.Google Scholar
  18. Jager, T., Schwenk, J., Somorovsky, J. (2015). Practical invalid curve attacks on tls-ecdh. In European Symposium on research in computer security (pp. 407–425). Springer.Google Scholar
  19. Kleine, K., & Simos, D.E. (2017). Coveringcerts: combinatorial methods for x.509 certificate testing. In 2017 IEEE International conference on software testing, verification and validation (ICST) (pp. 69–79).Google Scholar
  20. Krawczyk, H., Paterson, K.G., Wee, H. (2013). On the security of the tls protocol: a systematic analysis. In CRYPTO.Google Scholar
  21. Kuhn, R., Lei, Y., Kacker, R. (2008). Practical combinatorial testing: beyond pairwise. It Professional 10(3).Google Scholar
  22. Kuhn, D.R., Bryce, R., Duan, F., Ghandehari, L.S., Lei, Y., Kacker, R.N. (2015). Chapter one-combinatorial testing: theory and practice. Advances in Computers, 99, 1–66.CrossRefGoogle Scholar
  23. Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J. (2008). Ipog/ipog-d: efficient test generation for multi-way combinatorial testing. Software Testing, Verification and Reliability, 18(3), 125–148.CrossRefGoogle Scholar
  24. Leitner, A., & Bloem, R. (2005). Automatic testing through planning. Tech. rep. Technische Universität Graz, Institute for Software Technology.Google Scholar
  25. Mavrogiannopoulos, N., Vercauteren, F., Velichkov, V., Preneel, B. (2012). A cross-protocol attack on the tls protocol. In ACM CCS 12: 19th Conference on computer and communications security.Google Scholar
  26. McDermott, D., Ghallab, M., Howe, A., Knoblock, C., Ram, A., Veloso, M., Weld, D., Wilkins, D. (1998). Pddl - the planning domain definition language. In The AIPS-98 planning competition comitee.Google Scholar
  27. Memon, A.M., Pollack, M.E., Soffa, M.L. (2000). A planning-based approach to gui testing. In Proceedings of the 13th international software / internet quality week (QW’00).Google Scholar
  28. Meyer, C., Somorovsky, J., Weiss, E., Schwenk, J., Schinzel, S., Tews, E. (2014). Revisiting ssl/tls implementations: new bleichenbacher side channels and attacks. In USENIX Security (Vol. 14, pp. 733–748).Google Scholar
  29. Morais, A., Martins, E., Cavalli, A., Jimenez, W. (2009). Security protocol testing using attack trees. In CSE (2), IEEE Computer Society (2009) (pp. 690–697).Google Scholar
  30. Raffelt, H., Steffen, B., Berg, T. (2005). Learnlib: a library for automata learning and experimentation. In Proceedings of the 10th international workshop on formal methods for industrial critical systems (FMICS’05) (pp. 62–71).Google Scholar
  31. Russell, S.J., & Norvig, P. (1995). Artificial intelligence: a modern approach. Prentic Hall.Google Scholar
  32. Shmaryahu, D., Shani, G., Hoffmann, J., Steinmetz, M. (2017). Partially observable contingent planning for penetration testing. In: Proceedings of first international workshop on AI in security (IWAIse).Google Scholar
  33. Simos, D.E., Kuhn, R., Voyiatzis, A.G., Kacker, R. (2016). Combinatorial methods in security testing. IEEE Computer, 49, 40–43.CrossRefGoogle Scholar
  34. Simos, D.E., Bozic, J., Duan, F., Garn, B., Kleine, K., Lei, Y., Wotawa, F. (2017). Testing tls using combinatorial methods and execution framework. In Proceedings of the IFIP international conference on testing software and systems (ICTSS’17).Google Scholar
  35. Somorovsky, J. (2016). Systematic fuzzing and testing of tls libraries. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security (CCS’16).Google Scholar
  36. Walz, A., & Sikora, A. (2017). Exploiting dissent: towards fuzzing-based differential black box testing of tls implementations. IEEE Transactions on Dependable and Secure Computing, 99, 1–1.CrossRefGoogle Scholar
  37. Yadav, N., Thangarajah, J., Sardina, S. (2017). Agent design consistency checking via planning. In Proceedings of the twenty-sixth international joint conference on artificial intelligence (IJCAI-17).Google Scholar
  38. Yu, L., Lei, Y., Kacker, R.N., Kuhn, D.R. (2013). Acts: a combinatorial test generation tool. In 2013 IEEE Sixth International Conference on software testing, verification and validation (ICST) (pp. 370–375). IEEE.Google Scholar
  39. Yu, L., Lei, Y., Nourozborazjany, M., Kacker, R.N., Kuhn, D.R. (2013). An efficient algorithm for constraint handling in combinatorial test generation. In 2013 IEEE Sixth International Conference on software testing, verification and validation (ICST) (pp. 242–251). IEEE.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.SBA ResearchViennaAustria
  2. 2.Institute for Software TechnologyGraz University of TechnologyGrazAustria
  3. 3.University of Texas at ArlingtonArlingtonUSA

Personalised recommendations