Semantic Features from Web-Traffic Streams

Abstract

We describe a method to convert web-traffic textual streams into a set of documents in a corpus to allow use of established linguistic tools for the study of semantics, topic evolution, and token-combination signatures. A novel web-document corpus is also described which represents semantic features from each batch for subsequent analysis. A (American-English) lexicon is used to create a canonical representation of each corpus whereby there is a consistent mapping of each TermID to the corresponding lexicon-word or token. Finally, representation of a corpus member as a ‘document’ is accomplished by combining the (http) request string with the concatenation of all responses to it. This representation thus allows association of the request string tokens with the resulting content, for consumption by document classification and comparison algorithms.

Appendices

Appendix: Sample Representations

The following excerpt illustrates typical web-traffic captured by Snort, TCPdump, and other string-oriented capture tools. These tools often add line-feeds following header records to enhance human readability. Other, binary data are rendered as ASCII characters, or as ‘.’ when the corresponding byte is not a printable character (Fig. 1).

Fig. 1

Various lexicons used in our process to represent and analyze web-stream content. Three of the lexicons are (mostly) removed using stop-word lists. These sub-lexicons (II, III, V) would distract from semantic analyses that use T:F representations

Raw Data (from TCPdump)

Raw Traffic Strings After Tokenization

After Stop-Word Removal and Mapping to the Lexicon TermIDs

A validation display of token expansion from (T:F) back to lexicon word[T] per document. Documents #25 and #26 are shown:

WebStops

The webstops list contains many word tokens that comprise HTML markup and containers in web pages, such as tables, JavaScript functions, list structures, and style sheets. These words relate to the construction of such containers common to all web pages, and hence, are devoid of semantic content. These tokens, even though colliding with some lexicon words, must be removed so that the high frequency of these tokens does not dominate (T:F) sensitive semantic analysis algorithms like latent Dirichlet allocation.

academic	block	chars
accent	body	check
accept	bold	class
action	border	click
agent	bottom	clip
align	bounding	close
alive	box	color
application	boxes	colorful
author	browse	comma
auto	browser	common
background	bundle	compatibility
banner	button	compatible
batch	buttons	connection
before	bytes	console
begin	cache	content
bind	cancel	continue
blackboard	center	control
blank	char	cookie
	character	cookies
	characters	copy . . . .

CustomStopWords

Modern web pages contain other non-lexicon words associated with JavaScript code, variable names and values. JavaScripting markup attributes also typically contain numerous key-value pairs. We observe that the values of KVPs most often contain semantically interesting content. The current process tries to retain these values, as well as other named entities, while removing non-lexicon keys and variable names. The following custom stopword list was formed after examination of a small set of web-pages, and was labeled by the author for subsequent use on this dataset. We also suggest that a more accurate and dynamic result should process the KVPs early in tokenization by splitting on the equals character (“=”). HTML and JavaScript keywords can be formally enumerated and removed. Variable names will be more difficult to determine precisely; it is likely partial-word stemming of segments may yield satisfactory performance since most often, variables consist of concatenated words, sometimes camel-cased, for self documentation. Entropy measures of the discovered components could also improve recognition of variable names.

bbnj	puvq	carin
pbtpid	panose	callout
errorh	btngradientopacity	imcspan
yvlq	abpay	unexpectedtype
validatedelete	pubi	headerbgcolor
logout	rssheadlinecell	colheader
classe	brea	codebase
iptg	serv	privacypolicy
sfri	offborder	jrskl
emihidden	regexpmatch	pollcometinterval
pickname	fieldcaption	reqrevision
headgrade	playlists	baccentmedium
mathfont	getmenubyname	substr
nbsp	clickimage	bord
sethttpmethod	nprmodpipe	active . . . .

Semantic Analysis After LDA (Latent Dirichlet Allocation)

Treating such vectors in their pre-converted form would preserve anonymity while allowing trending, differentiation and anomaly analysis and comparisons. The following output has been expanded to the original words as a validation step to show correspondence to the original corpus documents.