Perception Mining of Network Protocol’s Stealth Attack Behaviors

  • Yan-Jing Hu
  • Xu An WangEmail author
Conference paper
Part of the Lecture Notes in Networks and Systems book series (LNNS, volume 97)


Unknown network protocol’s stealth attack behavior is becoming a new type of attack, which greatly harms the cyber space security. The stealth behaviors are not easy to be detected by existing security measures. Starting with the implementation of the instructions of the protocol programs, the normal behavior instruction sequences are captured by dynamic binary analysis. The algorithm of instruction clustering and feature distance computation is designed to mine the potential stealth attack behavior instruction sequences. The mined stealth attack behavior instruction sequences (for inline assembly) are loaded into the general executing framework. A virtual protocol behavior analysis platform HiddenDisc has been developed, and the Dynamic analysis is implemented on the platform. Then the protocol execution security evaluation scheme is proposed and implemented. Using the stealth transformation method designed by ourselves, the stealth attack behaviors are transformed. We successfully attacked the virtual target machine by using the transformed stealth attack behaviors, but the stealth behaviors were not captured. The experimental results show that the present method can accurately and efficiently perception mining unknown protocol’s stealth attack behaviors, transform and use of stealth attack behavior can also enhance our information offensive and defensive capabilities.


Protocol reverse Stealth attack behavior Instruction clustering 



This work is supported by the National Key Research and Development Program of China Under Grants No. 2017YFB0802000, National Cryptography Development Fund of China Under Grants No. MMJJ20170112, the Natural Science Basic Research Plan in Shaanxi Province of china (Grant Nos. 2018JM6028), National Nature Science Foundation of China (Grant Nos. 61772550, 61572521, U1636114, 61402531, 61103178, 61373170, 61402530, 61309022 and 61309008.), Engineering University of PAP’s Funding for Scientific Research Innovation Team (grant no. KYTD201805).


  1. 1.
    Harale, A., Tambe, S.: Detection and analysis of network & application layer attacks using honey pot with system security features. Int. J. Adv. Res. Ideas Innov. Technol. 3, 1–4 (2017)Google Scholar
  2. 2.
    Meng, B., et al.: DDOS attack detection system based on analysis of users’ behaviors for application layer. In: 2017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC) 2017, pp. 596–599 (2017)Google Scholar
  3. 3.
    Wang, Y., Yang, J.: Ethical hacking and network defense: choose your best network vulnerability scanning tool. In: 31st International Conference on Advanced Information Networking and Applications Workshops (WAINA) 2017. IEEE Conference Publications, pp. 110–113 (2017)Google Scholar
  4. 4.
    Bateman, W.M., Amaya, A., Fenstermaker, J.: Securing the grid and your critical utility functions. In: 2017 IEEE Rural Electric Power Conference (REPC) 2017, pp. 29–37 (2017)Google Scholar
  5. 5.
    Dooley, M., Rooney, T.: DNS vulnerabilities. In: DNS Security Management 2017, p. 324. Wiley-IEEE Press (2017)Google Scholar
  6. 6.
    Almubairik, N.A., Wills, G.: Automated penetration testing based on a threat model. In: 11th International Conference for Internet Technology and Secured Transactions (ICITST) 2016, pp. 413–414. IEEE Conference Publications (2016)Google Scholar
  7. 7.
    Narayan, J., Shukla, S.K., Clancy, T.C.: A survey of automatic protocol reverse engineering tools. ACM Comput. Surv. 48(3), 1–26 (2015)CrossRefGoogle Scholar
  8. 8.
    Zhang Zhao, W.Q.-Y., Wen, T.: Survey of mining protocol specifications. Comput. Eng. Appl. 49, 1–9 (2013)Google Scholar
  9. 9.
    Luo, X., et al.: A type-aware approach to message clustering for protocol reverse engineering. Sensors 19(3), 716 (2019)CrossRefGoogle Scholar
  10. 10.
    Votipka, D., et al.: An observational investigation of reverse engineers’ process and mental models. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems 2019, pp. 1–6. ACM, Glasgow (2019)Google Scholar
  11. 11.
    Li, P., Mao, K.: Knowledge-oriented convolutional neural network for causal relation extraction from natural language texts. Expert Syst. Appl. 115, 512–523 (2019)CrossRefGoogle Scholar
  12. 12.
    Bossert, G., Guihéry, F., Hiet, G.: Towards automated protocol reverse engineering using semantic information. In: Proceedings of the 9th ACM Symposium on Information, Computer and Communications Security 2014, pp. 51–62. ACM, Kyoto (2014)Google Scholar
  13. 13.
    Koganti, V.S., Galla, L.K., Nuthalapati, N.: Internet worms and its detection. In: International Conference on Control, Instrumentation, Communication and Computational Technologies (ICCICCT) 2016, pp. 64–73. IEEE Conference Publications (2016)Google Scholar
  14. 14.
    Pawlowski, A., Contag, M., Holz, T.: Probfuscation: an obfuscation approach using probabilistic control flows. In: Caballero, J., Zurutuza, U., Rodríguez, R. (eds.) Detection of Intrusions and Malware, and Vulnerability Assessment: Proceedings of the 13th International Conference, DIMVA 2016, San Sebastián, Spain, 7–8 July 2016, pp. 165–185. Springer, Cham (2016)CrossRefGoogle Scholar
  15. 15.
    Xie, X., et al.: Mixed obfuscation of overlapping instruction and self-modify code based on hyper-chaotic opaque predicates. In: Tenth International Conference on Computational Intelligence and Security 2014, pp. 524–528. IEEE Conference Publications (2014)Google Scholar
  16. 16.
    Payer, M.: HexPADS: a platform to detect “stealth” attacks. In: Caballero, J., Bodden, E., Athanasopoulos, E. (eds.) Engineering Secure Software and Systems: Proceedings of the 8th International Symposium, ESSoS 2016, London, UK, 6–8 April 2016, pp. 138–154. Springer, Cham (2016)Google Scholar
  17. 17.
    Karim, A., et al.: Botnet detection techniques: review, future trends, and issues. J. Zhejiang Univ. Sci. C 15(11), 943–983 (2014)CrossRefGoogle Scholar
  18. 18.
    Abul Hasan, M.J., Ramakrishnan, S.: A survey: hybrid evolutionary algorithms for cluster analysis. Artif. Intell. Rev. 36(3), 179–204 (2011)CrossRefGoogle Scholar
  19. 19.
    Lim, J., Reps, T., Liblit, B.: Extracting output formats from executables. In: Proceedings of the Working Conference on Reverse Engineering, Benevento, Italy (2006)Google Scholar
  20. 20.
    Egele, M., et al.: A survey on automated dynamic malware-analysis techniques and tools. ACM Comput. Surv. 44(2), 1–42 (2012)CrossRefGoogle Scholar
  21. 21.
    Caballero, J., Yin, H., Liang, Z., Dawn, S.: Polyglot: automatic extraction of protocol message format using dynamic binary analysis. In: Proceedings of the 14th ACM Conference on Computer and Communications Security, pp. 317–329 (2007)Google Scholar
  22. 22.
    Caballero, J., Poosankam, P., Kreibich, C., Song, D.: Dispatcher: enabling active botnet infiltration using automatic protocol reverse-engineering. In: Proceedings of the 16th ACM Conference on Computer and Communications Security, pp. 621–634 (2009)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Network and Information Security Key LaboratoryEngineering University of the Armed Police ForceXi’anChina
  2. 2.National Key Laboratory of Integrated Services NetworksXidian UniversityXi’anChina

Personalised recommendations