Unsupervised Detection of APT C&C Channels using Web Request Graphs

  • Pavlos Lamprakis
  • Ruggiero Dargenio
  • David Gugelmann
  • Vincent Lenders
  • Markus Happe
  • Laurent VanbeverEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10327)


HTTP is the main protocol used by attackers to establish a command and control (C&C) channel to infected hosts in a network. Identifying such C&C channels in network traffic is however a challenge because of the large volume and complex structure of benign HTTP requests emerging from regular user browsing activities. A common approach to C&C channel detection has been to use supervised learning techniques which are trained on old malware samples. However, these techniques require large training datasets which are generally not available in the case of advanced persistent threats (APT); APT malware are often custom-built and used against selected targets only, making it difficult to collect malware artifacts for supervised machine learning and thus rendering supervised approaches ineffective at detecting APT traffic.

In this paper, we present a novel and highly effective unsupervised approach to detect C&C channels in Web traffic. Our key observation is that APT malware typically follow a specific communication pattern that is different from regular Web browsing. Therefore, by reconstructing the dependencies between Web requests, that is the Web request graphs, and filtering away the nodes pertaining to regular Web browsing, we can identify malware requests without training a malware model.

We evaluated our approach on real Web traces and show that it can detect the C&C requests of nine APTs with a true positive rate of 99.5–100% and a true negative rate of 99.5–99.7%. These APTs had been used against several hundred organizations for years without being detected.


Malware detection Web request graph Command and control channel Click detection Graph analysis Advanced persistent threat 


  1. 1.
  2. 2.
    Contagiodump Blog. Accessed Jan 2017
  3. 3.
  4. 4.
    HTTP Method Definitions. Accessed Jan 2017
  5. 5.
    Malware Capture Facility Project. Accessed Jan 2017
  6. 6.
    Malware-Traffic-Analysis Blog. Accessed Jan 2017
  7. 7.
    pcapanalysis. Accessed Jan 2017
  8. 8.
    Bilge, L., Balzarotti, D., Robertson, W., Kirda, E., Kruegel, C.: Disclosure: detecting botnet command and control servers through large-scale netflow analysis. In: Proceedings of the Annual Computer Security Applications Conference, ACSAC 2012, pp. 129–138. ACM (2012)Google Scholar
  9. 9.
    Bugzilla: Bug 1282878. Accessed Feb 2017
  10. 10.
    Burghouwt, P., Spruit, M., Sips, H.: Detection of covert botnet command and control channels by causal analysis of traffic flows. In: Wang, G., Ray, I., Feng, D., Rajarajan, M. (eds.) CSS 2013. LNCS, vol. 8300, pp. 117–131. Springer, Cham (2013). doi: 10.1007/978-3-319-03584-0_10 CrossRefGoogle Scholar
  11. 11.
    Chen, P., Desmet, L., Huygens, C.: A study on advanced persistent threats. In: Decker, B., Zúquete, A. (eds.) CMS 2014. LNCS, vol. 8735, pp. 63–72. Springer, Heidelberg (2014). doi: 10.1007/978-3-662-44885-4_5 Google Scholar
  12. 12.
  13. 13.
  14. 14.
    FireEye: To Russia With Targeted Attack. Accessed Feb 2017
  15. 15.
    Gu, G., Perdisci, R., Zhang, J., Lee, W.: Botminer: clustering analysis of network traffic for protocol- and structure-independent botnet detection. In: Proceedings of the USENIX Security Symposium. USENIX Security 2008 (2008)Google Scholar
  16. 16.
    Gu, G., Zhang, J., Lee, W.: Botsniffer: detecting botnet command and control channels in network traffic. In: Proceedings of the Network and Distributed System Security Symposium (NDSS 2008) (2008)Google Scholar
  17. 17.
    Gugelmann, D., Gasser, F., Ager, B., Lenders, V.: Hviz: Http(s) traffic aggregation and visualization for network forensics. In: Proceedings of the DFRWS Europe (DFRWS 2015 Europe) Digital Investigation 12, Supplement 1, pp. 1–11 (2015)Google Scholar
  18. 18.
    IETF: Online Certificate Status Protocol - OCSP. Accessed Feb 2017
  19. 19.
    Jacob, G., Hund, R., Kruegel, C., Holz, T.: Jackstraws: picking command and control connections from bot traffic. In: Proceedings of the USENIX Security Symposium. USENIX Security 2011 (2011)Google Scholar
  20. 20.
    Jones, M.: Protecting privacy with referrers (2010). Accessed Feb 2017
  21. 21.
  22. 22.
    Kim, S.J., Lee, S., Bae, B.: Has-analyzer: detecting http-based c&c based on the analysis of http activity sets. TIIS 8(5), 1801–1816 (2014)CrossRefGoogle Scholar
  23. 23.
    Lab, K.: The Darkhotel APT, a story of unusual hospitality. Accessed Feb 2017
  24. 24.
    Mandiant: APT1 - Exposing One of China’s Cyber Espionage Units. Accessed Feb 2017
  25. 25.
    Neasbitt, C., Perdisci, R., Li, K., Nelms, T.: Clickminer: towards forensic reconstruction of user-browser interactions from network traces. In: Proceedings of the ACM CCS 2014, pp. 1244–1255. ACM (2014)Google Scholar
  26. 26.
    Nelms, T., Perdisci, R., Ahamad, M.: Execscent: mining for new c&c domains in live networks with adaptive control protocol templates. In: Proceedings of the USENIX Security Symposium, pp. 589–604. USENIX, Washington, D.C. (2013)Google Scholar
  27. 27.
    Nelms, T., Perdisci, R., Antonakakis, M., Ahamad, M.: Webwitness: investigating, categorizing, and mitigating malware download paths. In: Proceedings of the USENIX Security Symposium, pp. 1025–1040. USENIX (2015)Google Scholar
  28. 28.
    NIST: Managing Information Security Risk., nIST Special Publication 800–39
  29. 29.
  30. 30.
    Oprea, A., Li, Z., Yen, T.F., Chin, S.H., Alrwais, S.: Detection of early-stage enterprise infection by mining large-scale log data. In: Proceedings of the IEEE/IFIP Int. Conf. on Dependable Systems and Networks, DSN 2015, pp. 45–56. IEEE Computer Society (2015)Google Scholar
  31. 31.
    Paxson, V.: Bro: a system for detecting network intruders in real-time. Comput. Netw. 31(23–24), 2435–2463 (1999)CrossRefGoogle Scholar
  32. 32.
    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)MathSciNetzbMATHGoogle Scholar
  33. 33.
    Perdisci, R., Ariu, D., Giacinto, G.: Scalable fine-grained behavioral clustering of http-based malware. Comput. Netw. 57(2), 487–500 (2013)CrossRefGoogle Scholar
  34. 34.
    Proofpoint: Nettraveler apt targets russian, european interests. Accessed Jan 2017
  35. 35.
    Security, F.: Looking at the Sky for a DarkComet. Accessed Feb 2017
  36. 36.
    SeleniumHQ: Accessed Jan 2017
  37. 37.
    Symantec: Internet security threat report. Technical Report 21, Symantec, April 2016.
  38. 38.
    Tegeler, F., Fu, X., Vigna, G., Kruegel, C.: Botfinder: finding bots in network traffic without deep packet inspection. In: Proceedings of the International Conference on Emerging Networking Experiments and Technologies (CoNEXT), pp. 349–360. ACM (2012)Google Scholar
  39. 39.
  40. 40.
    Vassio, L., Drago, I., Mellia, M.: Detecting user actions from HTTP traces: toward an automatic approach. In: International Wireless Communications and Mobile Computing Conference (IWCMC), pp. 50–55 (2016)Google Scholar
  41. 41.
    W3C: Referer Policy. Accessed Feb 2017
  42. 42.
    Xie, G., Iliofotou, M., Karagiannis, T., Faloutsos, M., Jin, Y.: Resurf: reconstructing web-surfing activity from network traffic. In: Proceedings of the International Conference on Networking, IFIP (2013)Google Scholar
  43. 43.
    Zhang, H., Banick, W., Yao, D., Ramakrishnan, N.: User intention-based traffic dependence analysis for anomaly detection. In: IEEE Symposium on Security and Privacy Workshops, pp. 104–112, May 2012Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Pavlos Lamprakis
    • 1
  • Ruggiero Dargenio
    • 1
  • David Gugelmann
    • 1
  • Vincent Lenders
    • 2
  • Markus Happe
    • 1
  • Laurent Vanbever
    • 1
    Email author
  1. 1.ETH ZurichZurichSwitzerland
  2. 2.ArmasuisseThunSwitzerland

Personalised recommendations