Web Scanner Detection Based on Behavioral Differences

Abstract

Web scanners will not only take up the bandwidth of the server, but also collect sensitive information of websites and probe vulnerabilities of the system, which seriously threaten the security of websites. Accurate detection of Web scanners can effectively mitigate this kind of thread. Existing scanner detection methods extract features from log and differentiate between scanners and legal users with machine learning. However, these methods are unable to block scanning due to lack of behavior information of clients. To solve this problem, a Web scanner detection method based on behavioral differences is proposed. It collects request information and behavior information of clients by three modules named Passive Detection, Active Injection and Active Detection. Then, six kinds of features including fingerprint of scanners and execution ability of JavaScript code are extracted to detect whether a client is a scanner. This method makes full use of the behavior characteristics of clients and the behavioral differences between scanners and legal users. The experimental results showed the method is efficient and fast in scanner detection.

Appendices

A Fingerprint Information of Common Scanners

Name	Fingerprint	Location
wpscan	wpscan	User-Agent
SQLmap	Sqlmap/version/#stable(http://sqlmap.org)	User-Agent
AppScan	APPSCAN	Requset parameter, URL
AWVS	Acunetix-Aspect	Request header
W3af	w3af.org	User-Agent
Burpsuite	burpcollaborator.net	Request header, parameter, URL
WebCruiser	WebCruiser, HEAD method	User-Agent
NetSpark	X-Scanner:Netsparker or Netsparker	Request header, parameter
FileSensor	Scrapy/1.4.0 (+http://scrapy.org)	User-Agent
Yujian	HEAD method, User-Agent:-	User-Agent, RM
BBScan	BBScan/version	User-Agent
DirBrute	whoami=wyscan_dirfuzz	Cookie
Nikto	(Nikto/version) (Evasions:None) (Test:map_codes)	User-Agent

B Testing Results of Scanners

Scanners	Visiting frequency	Fingerprint	Resource file	Carrying cookie	JS execution	Mouse click	All methods
Appscan	$\surd$	$\surd$	$\surd$	$\times$	$\times$	$\surd$	$\surd$
w3af	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$
Burpsuite	$\surd$	$\surd$	$\surd$	$\times$	$\surd$	$\surd$	$\surd$
AWVS	$\surd$	$\surd$	$\surd$	$\times$	$\surd$	$\surd$	$\surd$
netsparker	$\surd$	$\surd$	$\surd$	$\times$	$\surd$	$\surd$	$\surd$
htpwdScan	$\surd$	$\times$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$
NagaScan	$\surd$	$\times$	$\times$	$\times$	$\times$	$\times$	$\surd$
WebCrusier	$\surd$	$\surd$	$\times$	$\surd$	$\surd$	$\surd$	$\surd$
Yujian directory scanning	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$
Yujian website identification	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$
SQLmap	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$
BBScan	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$
Cangibrina	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$
BruteXSS	$\times$	$\surd$	$\times$	$\times$	$\surd$	$\times$	$\surd$
Shuriken	$\times$	$\times$	$\times$	$\times$	$\surd$	$\times$	$\surd$
Weakfilescan	$\surd$	$\times$	$\surd$	$\times$	$\surd$	$\surd$	$\surd$
Dirsearch	$\surd$	$\times$	$\times$	$\surd$	$\surd$	$\surd$	$\surd$
Pentestdb	$\times$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$
Lcyscan	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$
DirBrute	$\surd$	$\surd$	$\surd$	$\times$	$\surd$	$\surd$	$\surd$
wpscan	$\times$	$\surd$	$\times$	$\times$	$\surd$	$\surd$	$\surd$