Article

Formal Methods in System Design

, Volume 41, Issue 1, pp 107-128

Recognizing malicious software behaviors with tree automata inference

  • Domagoj BabićAffiliated withComputer Science Division, University of California Email author 
  • , Daniel ReynaudAffiliated withComputer Science Division, University of California
  • , Dawn SongAffiliated withComputer Science Division, University of California

Rent the article at a discount

Rent now

* Final gross prices may vary according to local VAT.

Get Access

Abstract

We explore how formal methods and tools of the verification trade could be used for malware detection and analysis. In particular, we propose a new approach to learning and generalizing from observed malware behaviors based on tree automata inference. Our approach infers k-testable tree automata from system call dataflow dependency graphs. We show how inferred automata can be used for malware recognition and classification.

Keywords

Tree automata inference Behavioral malware detection