A Novel Metric for Measuring Operational Effectiveness of a Cybersecurity Operations Center



Cybersecurity threats are on the rise with evermore digitization of the information that many day-to-day systems depend upon. The demand for cybersecurity analysts outpaces supply, which calls for optimal management of the analyst resource. In this chapter, a new notion of cybersecurity risk is defined, which arises when alerts from intrusion detection systems remain unanalyzed at the end of a work-shift. The above risk poses a security threat to the organization, which in turn impacts the operational effectiveness of the cybersecurity operations center (CSOC). The chapter considers four primary analyst resource parameters that influence risk. For a given risk threshold, the parameters include (1) number of analysts in a work-shift, and in turn within the organization, (2) expertise mix of analysts in a work-shift to investigate a wide range of alerts, (3) optimal sensor to analyst allocation, and (4) optimal scheduling of analysts that guarantees both number and expertise mix of analysts in every work-shift. The chapter presents a thorough treatment of risk and the role it plays in analyst resource management within a CSOC under varying alert generation rates from sensors. A simulation framework to measure risk under various model parameter settings is developed, which can also be used in conjunction with an optimization model to empirically validate the optimal settings of the above model parameters. The empirical results, sensitivity study, and validation study confirms the viability of the framework for determining the optimal management of the analyst resource that minimizes risk under the uncertainty of alert generation and model constraints.



The authors would like to thank Dr. Cliff Wang of the Army Research Laboratory for suggesting this problem to us. Ganesan, Jajodia, and Shah were partially supported by the Army Research Office under grants W911NF-13-1-0421 and W911NF-15-1-0576 and by the Office of Naval Research grant N00014-15-1-2007.


  1. 1.
    M. Albanese, C. Molinaro, F. Persia, A. Picariello, V.S. Subrahmanian, Discovering the top-k unexplained sequences in time-stamped observation data. IEEE Trans. Knowl. Data Eng. 26(3), 577–594 (2014)CrossRefGoogle Scholar
  2. 2.
    J.P. Anderson, Computer security threat monitoring and surveillance. Tech. Rep. James P. Anderson Co., Fort Washington, PA (1980)Google Scholar
  3. 3.
    M.E. Aydin, E. Oztemel, Dynamic job-shop scheduling using reinforcement learning agents. Robot. Auton. Syst. 33(2), 169–178 (2000)CrossRefGoogle Scholar
  4. 4.
    D. Barbara, S. Jajodia (eds.), Application of Data Mining in Computer Security, vol. 6. Advances in Information Security (Springer, New York, 2002)Google Scholar
  5. 5.
    S. Bhatt, P.K. Manadhata, L. Zomlot, The operational role of security information and event management systems. IEEE Secur. Priv. 12(5), 35–41 (2014)CrossRefGoogle Scholar
  6. 6.
    D. Botta, K Muldner, K Hawkey, K Beznosov, Toward understanding distributed cognition in it security management: the role of cues and norms. Cogn. Tech. Work 13(2), 121–134 (2011)Google Scholar
  7. 7.
    D.S. Chen, R.G. Batson, Y. Dang, Applied Integer Programming (Wiley, Hoboken, 2010)zbMATHGoogle Scholar
  8. 8.
    CIO, DON Cyber Crime Handbook. Department of Navy, Washington, DC (2008)Google Scholar
  9. 9.
    A. D’Amico, K. Whitley, The real work of computer network defense analysts: the analysis roles and processes that transform network data into security situation awareness, in Proceedings of the Workshop on Visualization for Computer Security, pp. 19–37 (2008)Google Scholar
  10. 10.
    D.E. Denning, An intrusion-detection model, in Proceedings of IEEE Symposium on Security and Privacy, Oakland, CA, pp. 118–131 (1986)Google Scholar
  11. 11.
    D.E. Denning, An intrusion-detection model. IEEE Trans. Softw. Eng. 13(2), 222–232 (1987)CrossRefGoogle Scholar
  12. 12.
    R. Di Pietro, L.V. Mancini (eds.), Intrusion Detection Systems, vol. 38. Advances in Information Security (Springer, New York, 2008)Google Scholar
  13. 13.
    H. Du, S.J. Yang, Temporal and spatial analyses for large-scale cyber attacks, in Handbook of Computational Approaches to Counterterrorism, ed. by V.S. Subrahmanian (Springer, New York, 2013), pp. 559–576CrossRefGoogle Scholar
  14. 14.
    R.F. Erbacher, S.E. Hutchinson, Extending case-based reasoning to network alert reporting, in 2012 ASE International Conference on Cyber Security, pp. 187–194 (2012)Google Scholar
  15. 15.
    S.M. Furnell, N. Clarke, R. Werlinger, K. Muldner, K. Hawkey, K. Beznosov, Preparation, detection, and analysis: the diagnostic work of it security incident response. Inf. Manag. Comput. Secur. 18(1), 26–42 (2010)CrossRefGoogle Scholar
  16. 16.
    R. Ganesan, S. Jajodia, A. Shah, H. Cam, Dynamic scheduling of cybersecurity analysts for minimizing risk using reinforcement learning. ACM Trans. Intell. Syst. Technol. 8(1), 4:1–4:21 (2016).
  17. 17.
    R. Ganesan, S. Jajodia, H. Cam, Optimal scheduling of cybersecurity analyst for minimizing risk. ACM Trans. Intell. Syst. Technol. 8(4), 52:1–52:33 (2017).
  18. 18.
    D. Gross, J. Shortle, J. Thompson, C. Harris, Fundamentals of Queuing Theory (Wiley-Interscience, New York, 2008)CrossRefzbMATHGoogle Scholar
  19. 19.
    D. Lesaint, C. Voudouris, N. Azarmi, I. Alletson, B. Laithwaite, Field workforce scheduling. BT Technol. J. 21(4), 23–26 (2003)CrossRefGoogle Scholar
  20. 20.
    G.L. Nemhauser, L.A. Wolsey, Integer and Combinatorial Optimization (Wiley-Interscience, New York, 1999)zbMATHGoogle Scholar
  21. 21.
    Y. Nobert, J. Roy, Freight handling personnel scheduling at air cargo terminals. Transp. Sci. 32(3), 295–301 (1998)CrossRefzbMATHGoogle Scholar
  22. 22.
    S. Northcutt, J. Novak, Network Intrusion Detection, 3rd edn. (New Riders Publishing, Thousand Oaks, CA, 2002)Google Scholar
  23. 23.
    C.D. Paternina-Arboleda, T.K. Das, A multi-agent reinforcement learning approach to obtaining dynamic control policies for stochastic lot scheduling problem. Simul. Model. Pract. Theory 13(5), 389–406 (2005)CrossRefGoogle Scholar
  24. 24.
    V. Paxson, Bro: a system for detecting network intruders in real-time. Comput. Netw. 31(23–24), 2435–2463 (1999)CrossRefGoogle Scholar
  25. 25.
    M. Pinedo, Planning and Scheduling in Manufacturing and Services (Springer, New York, 2009)CrossRefzbMATHGoogle Scholar
  26. 26.
    J. Reis, N. Mamede, Multi-Agent Dynamic Scheduling and Re-Scheduling with Global Temporal Constraints (Kluwer Academic Publishers, Boston, 2002)Google Scholar
  27. 27.
    R. Sadoddin, A. Ghorbani, Alert correlation survey: framework and techniques, in Proceedings of the ACM International Conference on Privacy, Security and Trust (ACM, New York, 2006), pp. 1–10Google Scholar
  28. 28.
    R. Sommer, V. Paxson, Outside the closed world: on using machine learning for network intrusion detection, in Proceedings of IEEE Symposium on Security and Privacy, pp. 305–316 (2010)Google Scholar
  29. 29.
    V.S. Subrahmanian, M. Ovelgonne, T. Dumitras, A. Prakash, The Global Cyber-Vulnerability Report (Springer, Cham, 2015)CrossRefGoogle Scholar
  30. 30.
    S.C. Sundaramurthy, J. McHugh, X. Ou, M. Wesch, A.G. Bardas, S.R. Rajagopalan, Turning contradictions into innovations or: how we learned to stop whining and improve security operations, in Twelfth Symposium on Usable Privacy and Security (SOUPS 2016) (2016)Google Scholar
  31. 31.
    F. Valeur, G. Vigna, C. Kruegel, R.A. Kemmerer, A comprehensive approach to intrusion detection alert correlation. IEEE Trans. Dependable Secure Comput. 1(3), 146–169 (2004)CrossRefGoogle Scholar
  32. 32.
    W. Winston, Operations Research (Cengage Learning, New York, 2003)Google Scholar
  33. 33.
    S.J. Zaccaro, R.S. Dalal, L.E. Tetrick, J.A. Steinke, Psychosocial Dynamics of Cyber Security (Routledge, New York, 2016)Google Scholar
  34. 34.
    F. Zhou, J. Wang, J. Wang, J. Jonrinaldi, A dynamic rescheduling model with multi-agent system and its solution method. J. Mech. Eng. 58(2), 81–92 (2012)CrossRefGoogle Scholar
  35. 35.
    C. Zimmerman, The Strategies of a World-Class Cybersecurity Operations Center (The MITRE Corporation, McLean, VA, 2014)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Center for Secure Information SystemsGeorge Mason UniversityFairfaxUSA
  2. 2.Army Research LaboratoryAdelphiUSA

Personalised recommendations