SKALD: A Scalable Architecture for Feature Extraction, Multi-user Analysis, and Real-Time Information Sharing

  • George D. WebsterEmail author
  • Zachary D. Hanif
  • Andre L. P. Ludwig
  • Tamas K. Lengyel
  • Apostolis Zarras
  • Claudia Eckert
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9866)


The inability of existing architectures to allow corporations to quickly process information at scale and share knowledge with peers makes it difficult for malware analysis researchers to present a clear picture of criminal activity. Hence, analysis is limited in effectively and accurately identify the full scale of adversaries’ activities and develop effective mitigation strategies. In this paper, we present Skald: a novel architecture which guides the creation of analysis systems to support the research of malicious activities plaguing computer systems. Our design provides the scalability, flexibility, and robustness needed to process current and future volumes of data. We show that our prototype is able to process millions of samples in only few milliseconds per sample with zero critical errors. Additionally, Skald enables the development of new methodologies for information sharing, enabling analysis across collective knowledge. Consequently, defenders can perform accurate investigations and real-time discovery, while reducing mitigation time and infrastructure cost.


Feature Extraction Service Orient Architecture Message Queue Investigation Planner Industry Peer 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



We would like to thank the Technical University of Munich for providing ample infrastructure to support our prototype development. We would also like to thank the United States Air Force for sponsoring George Webster in his academic pursuit. In addition, we thank the German Federal Ministry of Education and Research for providing funding for hardware under grant 16KIS0328 (IUNO). Lastly, we would like to thank the members of VirusTotal, Yara Exchange, and DARPA for their valuable discussions and support.


  1. 1.
    Alvarez, V.M.: Yara 3.3.0. VirusTotal (Google Inc.) (2015).
  2. 2.
    Barack, O.: Executive Order No. 13691. Promoting Private Sector Cybersecurity Information Sharing (2015)Google Scholar
  3. 3.
    Bu, Z., Dirro, T., Greve, P., Lin, Y., Marcus, D., Paget, F., Pogulievsky, V., Schmugar, C., Shah, J., Sommer, D., et al.: McAfee Threats Report: Second Quarter 2012 (2012)Google Scholar
  4. 4.
    Choo, K.-K.R.: The cyber threat landscape: challenges and future research directions. Comput. Secur. 30(8), 719–731 (2011)CrossRefGoogle Scholar
  5. 5.
    Cristian, F.: Understanding fault-tolerant distributed systems. Commun. ACM 34(2), 56–78 (1991)CrossRefGoogle Scholar
  6. 6.
    DARPA: Cyber Information Sharing - DARPA Cyber Forum, October 2015Google Scholar
  7. 7.
    Estublier, J.: Software configuration management: a roadmap. In: Conference on the Future of Software Engineering (2000)Google Scholar
  8. 8.
    Google: Protocol Buffers, November 2015.
  9. 9.
    Grobauer, B., Berger, S., Göbel, J., Schreck, T., Wallinger, J.: The MANTIS Framework: Cyber Threat Intelligence Management for CERTs, Boston, USA, June 2014Google Scholar
  10. 10.
    Guarnieri, C., Tanasi, A., Bremer, J., Schloesser, M.: The Cuckoo Sandbox (2012).
  11. 11.
    Hanif, Z., Calhoun, T., Trost, J.: BinaryPig: scalable static binary analysis over Hadoop. In: Black Hat USA (2013)Google Scholar
  12. 12.
    HiveMQ: MQTT Essentials Part 6: Quality of Service 0, 1 & 2 (2015).
  13. 13.
    Jang, J., Brumley, D., Venkataraman, S.: BitShred: feature hashing malware for scalable triage and semantic analysis. In: Conference on Computer and Communications Security, CCS (2011)Google Scholar
  14. 14.
    Kolosnjaji, B., Zarras, A., Lengyel, T., Webster, G., Eckert, C.: Adaptive semantics-aware malware classification. In: Caballero, J., Zurutuza, U., Rodríguez, R.J. (eds.) DIMVA 2016. LNCS, vol. 9721, pp. 419–439. Springer, Heidelberg (2016). doi: 10.1007/978-3-319-40667-1_21 CrossRefGoogle Scholar
  15. 15.
    Krafzig, D., Banke, K., Slama, D.: Enterprise SOA: Service-Oriented Architecture Best Practices. Prentice Hall Professional, Indianapolis (2005)Google Scholar
  16. 16.
    Lakshman, A., Malik, P.: Cassandra: a decentralized structured storage system. ACM SIGOPS Oper. Syst. Rev. 44(2), 35–40 (2010)CrossRefGoogle Scholar
  17. 17.
    Lengyel, T.K., Maresca, S., Payne, B.D., Webster, G.D., Vogl, S., Kiayias, A.: Scalability, fidelity and stealth in the DRAKVUF dynamic malware analysis system. In: Annual Computer Security Applications Conference, ACSAC (2014)Google Scholar
  18. 18.
    Ollmann, G.: Behind todays crimeware installation lifecycle: how advanced malware morphs to remain stealthy and persistent. Technical report, Damballa (2011)Google Scholar
  19. 19.
    Papazoglou, M.P., Van Den Heuvel, W.-J.: Service oriented architectures: approaches, technologies and research issues. VLDB J. 16(3), 389–415 (2007)CrossRefGoogle Scholar
  20. 20.
    Parkour, M., DiMino, A.: Deepend Research - Yara Exchange, May 2015.
  21. 21.
    Shields, W.: Problems with PEHash Implementations, September 2014.
  22. 22.
    Stamos, A.: The Failure of the Security Industry, April 2015.
  23. 23.
    The MITRE Corporation: Collaborative Research Into Threats (CRITs), June 2014.
  24. 24.
    Verma, A., Pedrosa, L., Korupolu, M.R., Oppenheimer, D., Tune, E., Wilkes, J.: Large-scale cluster management at Google with Borg. In: European Conference on Computer Systems, EuroSys (2015)Google Scholar
  25. 25.
    VirusTotal: File Statistics, May 2015.
  26. 26.
    Vixie, P.: Internet Security Marketing: Buyer Beware, April 2015.
  27. 27.
    Wicherski, G.: PEHash: a novel approach to fast malware clustering. In: USENIX Workshop on Large-Scale Exploits and Emergent Threats, LEET (2009)Google Scholar
  28. 28.
    Zeltser, L.: SANS - Managing and Exploring Malware Samples with Viper, June 2014.

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • George D. Webster
    • 1
    Email author
  • Zachary D. Hanif
    • 1
  • Andre L. P. Ludwig
    • 1
  • Tamas K. Lengyel
    • 1
  • Apostolis Zarras
    • 1
  • Claudia Eckert
    • 1
  1. 1.Technical University of MunichGarchingGermany

Personalised recommendations