Why Johnny Can’t Pentest: An Analysis of Black-Box Web Vulnerability Scanners

  • Adam Doupé
  • Marco Cova
  • Giovanni Vigna
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6201)


Black-box web vulnerability scanners are a class of tools that can be used to identify security issues in web applications. These tools are often marketed as “point-and-click pentesting” tools that automatically evaluate the security of web applications with little or no human support. These tools access a web application in the same way users do, and, therefore, have the advantage of being independent of the particular technology used to implement the web application. However, these tools need to be able to access and test the application’s various components, which are often hidden behind forms, JavaScript-generated links, and Flash applications.

This paper presents an evaluation of eleven black-box web vulnerability scanners, both commercial and open-source. The evaluation composes different types of vulnerabilities with different challenges to the crawling capabilities of the tools. These tests are integrated in a realistic web application. The results of the evaluation show that crawling is a task that is as critical and challenging to the overall ability to detect vulnerabilities as the vulnerability detection techniques themselves, and that many classes of vulnerabilities are completely overlooked by these tools, and thus research is required to improve the automated detection of these flaws.


USENIX Security Symposium Directory Traversal Vulnerability Scanner Injection Vulnerability Initial Initial 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    AnantaSec: Web Vulnerability Scanners Evaluation (January 2009),
  2. 2.
    Balzarotti, D., Cova, M., Felmetsger, V., Vigna, G.: Multi-module Vulnerability Analysis of Web-based Applications. In: Proceedings of the ACM conference on Computer and Communications Security (CCS), pp. 25–35 (2007)Google Scholar
  3. 3.
    Curphey, M., Araujo, R.: Web Application Security Assessment Tools. IEEE Security and Privacy 4(4), 32–41 (2006)CrossRefGoogle Scholar
  4. 4.
    CVE: Common Vulnerabilities and Exposures,
  5. 5.
    Foundstone: Hacme Bank v2.0 (May 2006),
  6. 6.
    Grossman, J.: Challenges of Automated Web Application Scanning. In: BlackHat Windows Security Conference (2004)Google Scholar
  7. 7.
    Kals, S., Kirda, E., Kruegel, C., Jovanovic, N.: SecuBat: A Web Vulnerability Scanner. In: Proceedings of the International World Wide Web Conference (2006)Google Scholar
  8. 8.
    McAllister, S., Kruegel, C., Kirda, E.: Leveraging User Interactions for In-Depth Testing of Web Applications. In: Proceedings of the Symposium on Recent Advances in Intrusion Detection (2008)Google Scholar
  9. 9.
    Open Security Foundation: OSF DataLossDB: Data Loss News, Statistics, and Research,
  10. 10.
    Open Web Application Security Project (OWASP): OWASP SiteGenerator,
  11. 11.
    Open Web Application Security Project (OWASP): OWASP WebGoat Project,
  12. 12.
    Open Web Application Security Project (OWASP): Web Input Vector Extractor Teaser,
  13. 13.
    Open Web Application Security Project (OWASP): OWASP Top Ten Project (2010),
  14. 14.
    OpenID Foundation: OpenID,
  15. 15.
    PCI Security Standards Council: PCI DDS Requirements and Security Assessment Procedures, v1.2 (October 2008)Google Scholar
  16. 16.
    Peine, H.: Security Test Tools for Web Applications. Tech. Rep. 048.06, Fraunhofer IESE (January 2006)Google Scholar
  17. 17.
    Provos, N., Mavrommatis, P., Rajab, M., Monrose, F.: All Your iFRAMEs Point to Us. In: Proceedings of the USENIX Security Symposium, pp. 1–16 (2008)Google Scholar
  18. 18.
    RSnake: Sql injection cheat sheet,
  19. 19.
    RSnake: XSS (Cross Site Scripting) Cheat Sheet,
  20. 20.
    Small, S., Mason, J., Monrose, F., Provos, N., Stubblefield, A.: To Catch a Predator: A Natural Language Approach for Eliciting Malicious Payloads. In: Proceedings of the USENIX Security Symposium (2008)Google Scholar
  21. 21.
    Suto, L.: Analyzing the Effectiveness and Coverage of Web Application Security Scanners (October 2007) (case Study) Google Scholar
  22. 22.
    Suto, L.: Analyzing the Accuracy and Time Costs of Web Application Security Scanners (Feburary 2010)Google Scholar
  23. 23.
    Vieira, M., Antunes, N., Madeira, H.: Using Web Security Scanners to Detect Vulnerabilities in Web Services. In: Proceedings of the Conference on Dependable Systems and Networks (2009)Google Scholar
  24. 24.
    Wiegenstein, A., Weidemann, F., Schumacher, M., Schinzel, S.: Web Application Vulnerability Scanners—a Benchmark. Tech. rep., Virtual Forge GmbH (October 2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Adam Doupé
    • 1
  • Marco Cova
    • 1
  • Giovanni Vigna
    • 1
  1. 1.University of CaliforniaSanta Barbara

Personalised recommendations