, Volume 41, Issue 1, pp 107-128
Date: 11 Apr 2012

Recognizing malicious software behaviors with tree automata inference

Rent the article at a discount

Rent now

* Final gross prices may vary according to local VAT.

Get Access


We explore how formal methods and tools of the verification trade could be used for malware detection and analysis. In particular, we propose a new approach to learning and generalizing from observed malware behaviors based on tree automata inference. Our approach infers k-testable tree automata from system call dataflow dependency graphs. We show how inferred automata can be used for malware recognition and classification.

This paper is an extended journal version of [3]. The extensions include additional experimental results and a more thorough discussion.
This material is based upon work partially supported by the National Science Foundation under Grants No. 0832943, 0842694, 0842695, 0831501, 0424422, by the Air Force Research Laboratory under Grant No. P010071555, by the Office of Naval Research under MURI Grant No. N000140911081, and by the MURI program under AFOSR Grants No. FA9550-08-1-0352 and FA9550-09-1-0539. The work of the first author is also supported by the Natural Sciences and Engineering Research Council of Canada PDF fellowship.