Security Testing for Chatbots

  • Josip BozicEmail author
  • Franz Wotawa
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11146)


Services like chatbots that provide information to customers in real-time are of increasing importance for the online market. Chatbots offer an intuitive interface to answer user requests in an interactive manner. The inquiries are of wide-range and include information about specific goods and services but also financial issues and personal advices. The notable advantages of these programs are the simplicity of use and speed of the search process. In some cases, chatbots have even surpassed classical web, mobile applications, and social networks. Chatbots might have access to huge amount of data or personal information. Therefore, they might be a valuable target for hackers, and known web application vulnerabilities might be a security issue for chatbots as well. In this paper, we discuss the challenges of security testing for chatbots. We provide an overview about an automated testing approach adapted to chatbots, and first experimental results.


Adaptive systems security testing chatbots 



The research presented in the paper has been funded in part by the Cooperation Programme Interreg V-A Slovenia-Austria under the project AS-IT-IC (Austrian-Slovenian Intelligent Tourist Information Center).


  1. 1.
    Botium - new generation testing., accessed: 2018–05-07
  2. 2.
    BotMan - The PHP messaging and chatbot library., accessed: 2018–05-20
  3. 3. - We Mill Bots and Create Bot Milling Tools!, accessed: 2018–05-22
  4. 4.
    CharlieBot., accessed: 2018–05-22
  5. 5.
    Chatbot Market Size And Share Analysis, Industry Report, 2014–2025., accessed: 2018–05-07
  6. 6.
    Chatbottest., accessed: 2018–05-07
  7. 7.
    Gartner Top Strategic Predictions for 2018 and Beyond., accessed: 2018–05-07
  8. 8.
    jsoup: Java HTML Parser., accessed: 2018–02-02
  9. 9.
    OWASP Top Ten Project., accessed: 2018–01-31
  10. 10.
    Program O AI Chatbot - The Friendly Open Source PHP, MySQL, AIML Chatbot., accessed: 2018–02-04
  11. 11.
    QMetry BOT Tester., accessed: 2018–05-07
  12. 12.
    Bozic, J., Wotawa, F.: Security Testing Based on Attack Patterns. In: Proceedings of the 5th International Workshop on Security Testing (SECTEST’14) (2014)Google Scholar
  13. 13.
    Clarke, J., Fowler, K., Oftedal, E., Alvarez, R.M., Hartley, D., Kornbrust, A., O’Leary-Steele, G., Revelli, A., Siddharth, S., Slaviero, M.: SQL Injection Attacks and Defense, 2nd edn. Syngress, (2012)Google Scholar
  14. 14.
    Colby, K.: Artificial Paranoia: A Computer Simulation of Paranoid Process. Pergamon Press, New York (1975)Google Scholar
  15. 15.
    Duchene, F., Rawat, S., Richier, J.L., Groz, R.: KameleonFuzz: Evolutionary Fuzzing for Black-Box XSS Detection. In: CODASPY. pp. 37–48. ACM (2014)Google Scholar
  16. 16.
    Fogie, S., Grossman, J., Hansen, R., Rager, A., Petkov, P.D.: XSS Attacks: Cross Site Scripting Exploits and Defense. Syngress, (2007)Google Scholar
  17. 17.
    Halfond, W.G.J., Viegas, J., Orso, A.: A Classification of SQL Injection Attacks and Countermeasures. In: Proceedings of the IEEE International Symposium on Secure Software Engineering. Arlington, VA, USA (2006)Google Scholar
  18. 18.
    Liu, G., Liu, Q., Zhang, W.: Model-Based Testing and Validation on Artificial Intelligence Systems. In: Second International Multisymposium on Computer and Computational Sciences (2007)Google Scholar
  19. 19.
    Lowe, R., Noseworthy, M., Serban, I.V., Angelard-Gontier, N., Bengio, Y., Pineau, J.: Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses. In: Proceedings of the 5th International Conference on Learning Representations (ICLR) Workshop. Toulon, France (2017)Google Scholar
  20. 20.
    Martin, M., Lam, M.S.: Automatic Generation of XSS and SQL Injection Attacks with Goal-Directed Model Checking. In: 17th USENIX Security Symposium (2008)Google Scholar
  21. 21.
    McCarthy, J., Hayes, P.J.: Some Philosophical Problems from the Standpoint of Artificial Intelligence. In: Meltzer, B., Michie, D. (eds.) Machine Intelligence 4, pp. 463–502. Edinburgh University Press (1969), reprinted in McC90Google Scholar
  22. 22.
    Rushby, J.: Quality Measures and Assurance for AI Software. In: NASA Contract Report 4187, Washington DC (1988)Google Scholar
  23. 23.
    Shawar, B.A., Atwell, E.: Using corpora in machine-learning chatbot systems. In: International Journal of Corpus Linguistics, vol. 10 (2005)CrossRefGoogle Scholar
  24. 24.
    Vasconcelos, M., Candello, H., Pinhanez, C., dos Santos, T.: Bottester: Testing Conversational Systems with Simulated Users. In: IHC 2017: Proceedings of the XVI Brazilian Symposium on Human Factors in Computing Systems (2017)Google Scholar
  25. 25.
    Wallace, R.S.: The Elements of AIML Style. In: ALICE A.I. Foundation (2003)Google Scholar
  26. 26.
    Wallace, R.S.: The Anatomy of A.L.I.C.E. In: ALICE A.I. Foundation (2004)Google Scholar
  27. 27.
    Weizenbaum, J.: ELIZA-A Computer Program For the Study of Natural Language Communication Between Man and Machine. In: Communications of the ACM Volume 9, Number 1 (January 1966) (1966)CrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2018

Authors and Affiliations

  1. 1.Graz University of Technology, Institute for Software TechnologyGrazAustria

Personalised recommendations