Insider Threat Detection: Machine Learning Way

  • Mehul S. RavalEmail author
  • Ratnik Gandhi
  • Sanjay Chaudhary
Part of the Advances in Information Security book series (ADIS, volume 72)


The chapter aims to cover and analyse contributions from machine learning to detect an insider threat. It presents various launch mechanisms and details impact of an insider attack on various sectors. Presenting state-of-the-art for detecting insider threat based on psychology, criminology and game theory, the chapter also covers case studies showing use of Machine Learning for anomaly detection. In real life, malicious events are low in number. The chapter will showcase detection of such a low occurring anomaly from a large dataset accurately. The chapter specifically focuses on USB device insertion or removal event and apply linear regression followed by Cook’s and Mahalanobis distance to identify malicious activities of the user. Subsequently, it applies Neural Network and Support Vector Machine to login activities of a user to successfully demonstrates detection of an anomaly behaviour. It concludes discussing future directions that uses combination of methods from natural language processing, behavioural analysis, sentiment analysis, and machine learning for insider threat detection.



We would like to thank Ativ Joshi and Pratik Paladia for helping with the experiments. We also express gratitude to unknown reviewers for insightful comments in improving the quality of this chapter.


  1. 1.
  2. 2.
    Chris King, “Spotlight On: Malicious Insiders and Organized Crime Activity”, Technical note, CMU/SEI-2012-TN-001, Jan. 2012.Google Scholar
  3. 3.
    G. Fyffe, “Addressing insider threat,” Network Security, vol. 2008, no.3, pp. 11–14, 2008.CrossRefGoogle Scholar
  4. 4.
    S. L. Pfleeger and S. J. Stolfo, “Addressing the insider threat,” IEEE Security & Privacy, vol. 7, no. 6, pp. 10–13, 2009.CrossRefGoogle Scholar
  5. 5.
    E. Cole and S. Ring, Insider Threat: Protecting the Enterprise from Sabotage, Spying, and Theft: Protecting the Enterprise from Sabotage, Spying, and Theft. Syngress, 2005.Google Scholar
  6. 6.
  7. 7.
    Collins. Matthew, Theis. Michael, Trzeciak. Randall, Strozer. Jeremy, Clark. Jason, Costa. Daniel, Cassidy. Tracy, Albrethsen. Michael, and Moore. Andrew, “Common Sense Guide to Mitigating Insider Threats, 5th Edition,” Software Engineering Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, Technical Report CMU/SEI-2016-TR-015, 2016.
  8. 8.
    2017 data breach investigations report executive summary,
  9. 9.
    Adam Cummings, Todd Lewellen, David McIntire, Andrew P. Moore, Randall F. Trzeciak, “Insider Threat Study: Illicit Cyber Activity Involving Fraud in the U.S. Financial Services Sector,” Software Engineering Institute, CMU/SEI-2012-SR-004, July 2012.Google Scholar
  10. 10.
    A. Azaria, A. Richardson, S. Kraus and V. S. Subrahmanian, “Behavioral Analysis of Insider Threat: A Survey and Bootstrapped Prediction in Imbalanced Data,” in IEEE Transactions on Computational Social Systems, vol. 1, no. 2, pp. 135–155, June 2014.CrossRefGoogle Scholar
  11. 11.
    G. Magklaras and S. Furnell, “Insider threat prediction tool: Evaluating the probability of it misuse,” Computers & Security, vol. 21, no. 1, pp. 62–73, 2001.CrossRefGoogle Scholar
  12. 12.
    G. Jabbour and D. A. Menasce, “The insider threat security architecture: a framework for an integrated, inseparable, and uninterrupted self-protection mechanism,” in Computational Science and Engineering, 2009. CSE’09. International Conference on, vol. 3. IEEE, 2009, pp. 244–251.Google Scholar
  13. 13.
    J. Hunker and C. W. Probst, “Insiders and insider threats an overview of definitions and mitigation techniques,” Journal of Wireless Mobile Networks, Ubiquitous Computing, and Dependable Applications, vol. 2, no. 1, pp. 4–27, 2011.Google Scholar
  14. 14.
    S.Sinclair and S.W.Smith,“Preventative directions for insider threat mitigation via access control,” in Insider Attack and Cyber Security. Springer, 2008, pp. 165–194.Google Scholar
  15. 15.
    L. Greitzer and D. A. Frincke, “Combining traditional cyber security audit data with psychosocial data: towards predictive modeling for insider threat mitigation,” in Insider Threats in Cyber Security. Springer, 2010, pp. 85–113.Google Scholar
  16. 16.
    A. Liu, C. Martin, T. Hetherington, and S. Matzner, “A comparison of system call feature representations for insider threat detection,” in Information Assurance Workshop, 2005. IAW’05. Proceedings from the Sixth Annual IEEE SMC. IEEE, 2005, pp. 340–347.Google Scholar
  17. 17.
    M. A. Maloof and G. D. Stephens, “ELICIT: A system for detecting insiders who violate need-to-know,” in Recent Advances in Intrusion Detection. Springer, 2007, pp. 146–166.Google Scholar
  18. 18.
    Majumder, Navonil, Soujanya Poria, Alexander Gelbukh, and Erik Cambria, “Deep Learning-Based Document Modeling for Personality Detection from Text.” IEEE Intelligent Systems 32.2 (2017): 74–79.CrossRefGoogle Scholar
  19. 19.
    Tuor, Aaron, Samuel Kaplan, Brian Hutchinson, Nicole Nichols, and Sean Robinson. “Deep Learning for Unsupervised Insider Threat Detection in Structured Cybersecurity Data Streams.”, AAAI-17 Workshop on Artificial intelligence for cyber security, pp. 224–230, 2017.Google Scholar
  20. 20.
    Ryan, Jake, Meng-Jang Lin, and Risto Miikkulainen. “Intrusion detection with neural networks.” In Advances in neural information processing systems, pp. 943–949. 1998.Google Scholar
  21. 21.
    Debar, Herve, Monique Becker, and Didier Siboni. “A neural network component for an intrusion detection system.” In Research in Security and Privacy, 1992. Proceedings., 1992 IEEE Computer Society Symposium on, pp. 240–250. IEEE, 1992.Google Scholar
  22. 22.
    Veeramachaneni, Kalyan, Ignacio Arnaldo, Vamsi Korrapati, Constantinos Bassias, and Ke Li. “AI2: training a big data machine to defend.” In Big Data Security on Cloud (Big Data Security), IEEE International Conference on High Performance and Smart Computing (HPSC), and IEEE International Conference on Intelligent Data and Security (IDS), 2016 IEEE 2nd International Conference on, pp. 49–54. IEEE, 2016.Google Scholar
  23. 23.
    Nousiainen, Sami, Jorma Kilpi, Paula Silvonen, and Mikko Hiirsalmi. Anomaly detection from server log data. Technical report, 2009.Google Scholar
  24. 24.
    Rodriguez, Aitor, and Mario de los Mozos. “Improving network security through traffic log anomaly detection using time series analysis.” Computational Intelligence in Security for Information Systems 2010 (2010): 125–133.Google Scholar
  25. 25.
    Zhu, Xia. Resilient control and intrusion detection for scada systems. University of California, Berkeley, 2011.Google Scholar
  26. 26.
    Andrysiak, Tomasz, Łukasz Saganowski, Michał Choraś, and Rafał Kozik. “Network traffic prediction and anomaly detection based on ARFIMA model” In International Joint Conference SOCO’14-CISIS’14-ICEUTE’14, pp. 545–554. Springer, Cham, 2014.Google Scholar
  27. 27.
    Model, ARIMA-GARCH. “Detection of Network Attacks Using Hybrid.” In Dependability Problems and Complex Systems: Proceedings of the Twelfth International Conference on Dependability and Complex Systems DepCoS-RELCOMEX. July 2–6, 2017, Brunów, Poland, vol. 582, p. 1. Springer, 2017.Google Scholar
  28. 28.
  29. 29.
  30. 30.
  31. 31.
    Predd, Joel, Shari Lawrence Pfleeger, Jeffrey Hunker, and Carla Bulford. “Insiders behaving badly.” IEEE Security & Privacy 6, no. 4, pp.66–70, 2008.CrossRefGoogle Scholar
  32. 32.
  33. 33.
  34. 34.
    Epstein, Jeremy. “Security lessons learned from Société Générale” IEEE Security & Privacy 6, no. 3, pp. 80–82, 2008.Google Scholar
  35. 35.
    Rost, Johann. “Political reasons for failed software projects” IEEE Software 21, no. 6, pp. 103–104, 2004.CrossRefGoogle Scholar
  36. 36.
    Thompson, Hugh. “The human element of information security” IEEE Security & Privacy 11, no. 1 pp. 32–35, 2013.CrossRefGoogle Scholar
  37. 37.
    Duran, Felicia, Stephen H. Conrad, Gregory N. Conrad, David P. Duggan, and Edward Bruce Held. “Building a system for insider security.” IEEE Security & Privacy 7, no. 6, pp. 30–38, 2009.CrossRefGoogle Scholar
  38. 38.
    Band, S.R., Cappelli, D.M., Fischer, L.F., Moore, A.P., Shaw, E.D. and Trzeciak, R.F., 2006. Comparing insider IT sabotage and espionage: A model-based analysis (No. CMU/SEI-2006-TR-026). CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE ENGINEERING INST.Google Scholar
  39. 39.
    Herbig, K. “Changes in espionage by Americans 1947–2007,” Monterey, CA, Defense Personnel Security Research Center. 2008.Google Scholar
  40. 40.
    Turner, James T., and Michael Gelles. Threat assessment: A risk management approach. Routledge, 2012.Google Scholar
  41. 41.
    “Insider Analysis”, Module 23, The 19th International training course, SAND2006-1987C, Sandia National laboratories, 2006, pp. 214–287.Google Scholar
  42. 42.
    Greitzer, Frank L., Andrew P. Moore, Dawn M. Cappelli, Dee H. Andrews, Lynn A. Carroll, and Thomas D. Hull. “Combating the insider cyber threat.” IEEE Security & Privacy 6, no. 1, pp. 61–64, 2008.CrossRefGoogle Scholar
  43. 43.
    Legg, Philip A., Oliver Buckley, Michael Goldsmith, and Sadie Creese. “Automated insider threat detection system using user and role-based profile assessment.” IEEE Systems Journal 11, no. 2 (2017): 503–512.CrossRefGoogle Scholar
  44. 44.
    Koch, Robert, Mario Golling, and Gabi Dreo Rodosek. “Behavior-based intrusion detection in encrypted environments.” IEEE Communications Magazine 52, no. 7 (2014): 124–131.CrossRefGoogle Scholar
  45. 45.
    Bowen, Brian, Malek Ben Salem, Shlomo Hershkop, Angelos Keromytis, and Salvatore Stolfo. “Designing host and network sensors to mitigate the insider threat.” IEEE Security & Privacy 7, no. 6 (2009): 22–29.CrossRefGoogle Scholar
  46. 46.
    Böse, Brock, Bhargav Avasarala, Srikanta Tirthapura, Yung-Yu Chung, and Donald Steiner. “Detecting Insider Threats Using RADISH: A System for Real-Time Anomaly Detection in Heterogeneous Data Streams.” IEEE Systems Journal (2017).Google Scholar
  47. 47.
    Almehmadi, Abdulaziz, and Khalil El-Khatib. “On the possibility of insider threat prevention using intent-based access control (IBAC).” IEEE Systems Journal 11, no. 2 (2017): 373–384.CrossRefGoogle Scholar
  48. 48.
    Chen, You, Steve Nyemba, and Bradley Malin. “Detecting anomalous insiders in collaborative information systems.” IEEE transactions on dependable and secure computing 9, no. 3 (2012): 332–344.CrossRefGoogle Scholar
  49. 49.
    Mills, Jennifer U., Steven MF Stuban, and Jason Dever. “Predict insider threats using human behaviors.” IEEE Engineering Management Review 45, no. 1 (2017): 39–48.CrossRefGoogle Scholar
  50. 50.
    Theoharidou, Marianthi, Spyros Kokolakis, Maria Karyda, and Evangelos Kiountouzis. “The insider threat to information systems and the effectiveness of ISO17799.” Computers & Security 24, no. 6 (2005): 472–484.CrossRefGoogle Scholar
  51. 51.
    Caputo, Deanna, Marcus Maloof, and Gregory Stephens. “Detecting insider theft of trade secrets.” IEEE Security & Privacy 7, no. 6 (2009): 14–21.CrossRefGoogle Scholar
  52. 52.
    Jajodia, Sushil, Anup K. Ghosh, V. S. Subrahmanian, Vipin Swarup, Cliff Wang, and X. Sean Wang, eds. Moving Target Defense II: Application of Game Theory and Adversarial Modeling. Vol. 100. Springer Science & Business Media, 2012.Google Scholar
  53. 53.
    Pita, James, Manish Jain, Milind Tambe, Fernando Ordóñez, and Sarit Kraus. “Robust solutions to Stackelberg games: Addressing bounded rationality and limited observations in human cognition.” Artificial Intelligence 174, no. 15 (2010): 1142–1171.MathSciNetCrossRefGoogle Scholar
  54. 54.
    Roy, Sankardas, Charles Ellis, Sajjan Shiva, Dipankar Dasgupta, Vivek Shandilya, and Qishi Wu. “A survey of game theory as applied to network security.” In System Sciences (HICSS), 2010 43rd Hawaii International Conference on, pp. 1–10. IEEE, 2010.Google Scholar
  55. 55.
    Alpcan, Tansu, and Tamer Basar. “A game theoretic approach to decision and analysis in network intrusion detection.” In Decision and Control, 2003. Proceedings. 42nd IEEE Conference on, vol. 3, pp. 2595–2600. IEEE, 2003.Google Scholar
  56. 56.
    Liu, Debin, XiaoFeng Wang, and Jean Camp. “Game-theoretic modeling and analysis of insider threats.” International Journal of Critical Infrastructure Protection 1 (2008): 75–80.CrossRefGoogle Scholar
  57. 57.
    Rich, Eliot, Ignacio J. Martinez-Moyano, Stephen Conrad, Dawn M. Cappelli, Andrew P. Moore, Timothy J. Shimeall, David F. Andersen et al. “Simulating insider cyber-threat risks: a model-based case and a case-based model.” In Proceedings of the 23rd International Conference of the System dynamics Society, pp. 17–21. The System Dynamics Society, 2005.Google Scholar
  58. 58.
    Kraus, Sarit, Penina Hoz-Weiss, Jonathan Wilkenfeld, David R. Andersen, and Amy Pate. “Resolving crises through automated bilateral negotiations.” Artificial Intelligence 172, no. 1 (2008): 1–18.MathSciNetCrossRefGoogle Scholar
  59. 59.
    Rosenfeld, Avi, Inon Zuckerman, Amos Azaria, and Sarit Kraus. “Combining psychological models with machine learning to better predict people’s decisions.” Synthese 189, no. 1 (2012): 81–93.CrossRefGoogle Scholar
  60. 60.
    Nguyen, Thanh Hong, Rong Yang, Amos Azaria, Sarit Kraus, and Milind Tambe. “Analyzing the Effectiveness of Adversary Modeling in Security Games.” In AAAI. 2013.Google Scholar
  61. 61.
    Kantzavelou, Ioanna, and Sokratis Katsikas. “A game-based intrusion detection mechanism to confront internal attackers.” Computers & Security 29, no. 8 (2010): 859–874.CrossRefGoogle Scholar
  62. 62.
    Yang, Rong, Albert Xin Jiang, Milind Tambe, and Fernando Ordonez. “Scaling-up Security Games with Boundedly Rational Adversaries: A Cutting-plane Approach.” In IJCAI, pp. 404–410. 2013.Google Scholar
  63. 63.
    Tang, Ke, Mingyuan Zhao, and Mingtian Zhou. “Cyber insider threats situation awareness using game theory and information fusion-based user behavior predicting algorithm.” Journal of Information & Computational Science 8, no. 3 (2011): 529–545.Google Scholar
  64. 64.
    Pratt, Travis C., and Francis T. Cullen. “The empirical status of Gottfredson and Hirschi's general theory of crime: A meta-analysis.” Criminology 38, no. 3 (2000): 931–964.Google Scholar
  65. 65.
    Hirschi, Travis. “Social bond theory.” Criminological theory: Past to present. Los Angeles: Roxbury (1998).Google Scholar
  66. 66.
    Ajzen, Icek. “From intentions to actions: A theory of planned behavior.” In Action control, pp. 11–39. Springer Berlin Heidelberg, 1985.CrossRefGoogle Scholar
  67. 67.
    Clarke, Ronald VG. “Situational” “Crime Prevention: Theory and Practice.” The British Journal of Criminology 20, no. 2 (1980): 136–147.CrossRefGoogle Scholar
  68. 68.
  69. 69.
    Myerson, Roger B. Game theory. Harvard university press, 2013.Google Scholar
  70. 70.
    Krawczyk, Bartosz. “Learning from imbalanced data: open challenges and future directions.” Progress in Artificial Intelligence 5, no. 4 (2016): 221–232.CrossRefGoogle Scholar
  71. 71.
    Haixiang, G., Yijing, L., Shang, J., Mingyun, G., Yuanyue, H., & Bing, G. (2017). Learning from class-imbalanced data: Review of methods and applications. Expert Systems with Applications, 73, 220–239.CrossRefGoogle Scholar
  72. 72.
    Azaria, Amos, Ariella Richardson, Sarit Kraus, and V. S. Subrahmanian. “Behavioral analysis of insider threat: A survey and bootstrapped prediction in imbalanced data.” IEEE Transactions on Computational Social Systems 1, no. 2 (2014): 135–155.CrossRefGoogle Scholar
  73. 73.
    Cook, R. D. (1977). Detection of influential observation in linear regression. Technometrics, 19(1), 15–18.MathSciNetzbMATHGoogle Scholar
  74. 74.
    Mahalanobis, Prasanta Chandra (1936). “On the generalised distance in statistics”. Proceedings of the National Institute of Sciences of India. 2 (1): 49–55.Google Scholar
  75. 75.
    Ratnik Gandhi, Mehul S Raval, and Sanjay Chaudhary, “Pattern Discovery for Insider Threat”, CSI Communications 42, No. 2 (2018): 31–33.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Mehul S. Raval
    • 1
    Email author
  • Ratnik Gandhi
    • 1
  • Sanjay Chaudhary
    • 1
  1. 1.School of Engineering and Applied ScienceAhmedabad UniversityAhmedabadIndia

Personalised recommendations