International Journal of Information Security

, Volume 17, Issue 2, pp 121–134 | Cite as

A methodology to measure and monitor level of operational effectiveness of a CSOC

  • Ankit Shah
  • Rajesh Ganesan
  • Sushil JajodiaEmail author
  • Hasan Cam
Regular Contribution


In a cybersecurity operations center (CSOC), under normal operating conditions in a day, sufficient numbers of analysts are available to analyze the amount of alert workload generated by intrusion detection systems (IDSs). For the purpose of this paper, this means that the cybersecurity analysts can fully investigate each and every alert that is generated by the IDSs in a reasonable amount of time. However, there are a number of disruptive factors that can adversely impact the normal operating conditions such as (1) higher alert generation rates from a few IDSs, (2) new alert patterns that decreases the throughput of the alert analysis process, and (3) analyst absenteeism. The impact of all the above factors is that the alerts wait for a long duration before being analyzed, which impacts the readiness of the CSOC. It is imperative that the readiness of the CSOC be quantified, which in this paper is defined as the level of operational effectiveness (LOE) of a CSOC. LOE can be quantified and monitored by knowing the exact deviation of the CSOC conditions from normal and how long it takes for the condition to return to normal. In this paper, we quantify LOE by defining a new metric called total time for alert investigation (TTA), which is the sum of the waiting time in the queue and the analyst investigation time of an alert after its arrival in the CSOC database. A dynamic TTA monitoring framework is developed in which a nominal average TTA per hour (avgTTA/hr) is established as the baseline for normal operating condition using individual TTA of alerts that were investigated in that hour. At the baseline value of avgTTA/hr, LOE is considered to be ideal. Also, an upper-bound (threshold) value for avgTTA/hr is established, below which the LOE is considered to be optimal. Several case studies illustrate the impact of the above disruptive factors on the dynamic behavior of avgTTA/hr, which provide useful insights about the current LOE of the system. Also, the effect of actions taken to return the CSOC to its normal operating condition is studied by varying both the amount and the time of action, which in turn impacts the dynamic behavior of avgTTA/hr. Results indicate that by using the insights learnt from measuring, monitoring, and controlling the dynamic behavior of avgTTA/hr, a manager can quantify and color-code the LOE of the CSOC. Furthermore, the above insights allow for a deeper understanding of acceptable downtime for the IDS, acceptable levels for absenteeism, and the recovery time and effort needed to return the CSOC to its ideal LOE.


Intrusion detection Cybersecurity operations center Level of operational effectiveness Total time for alert investigation Situational awareness of CSOC 



The authors would like to thank Dr. Cliff Wang of the Army Research Office for the many discussions which served as the inspiration for this research.


  1. 1.
    CIO, DON Cyber Crime Handbook, Dept. of Navy, Washington, DC (2008)Google Scholar
  2. 2.
    Zimmerman, C.: The Strategies of a World-Class Cybersecurity Operations Center. The MITRE Corporation, McLean (2014)Google Scholar
  3. 3.
    Anderson, J.P.: Computer security threat monitoring and surveillance. Tech. rep., James P. Anderson Co., Fort Washington, PA (1980)Google Scholar
  4. 4.
    Denning, D.E.: An intrusion-detection model. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 118–131. Oakland, CA (1986)Google Scholar
  5. 5.
    Denning, D.E.: An intrusion-detection model. IEEE Trans. Softw. Eng. 13(2), 222 (1987)Google Scholar
  6. 6.
    Northcutt, S., Novak, J.: Network Intrusion Detection, 3rd edn. New Riders Publishing, Thousand Oaks (2002)Google Scholar
  7. 7.
    Di Pietro, R., Mancini, L.V. (eds.): Intrusion Detection Systems, Advances in Information Security, vol. 38. Springer, Berlin (2008)Google Scholar
  8. 8.
    Subrahmanian, V.S., Ovelgonne, M., Dumitras, T., Prakash, B.A.: The Global Cyber-Vulnerability Report. Springer, Switzerland (2015)Google Scholar
  9. 9.
    Sommer, R., Paxson, V.: Outside the closed world: on using machine learning for network intrusion detection. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 305–316 (2010)Google Scholar
  10. 10.
    Barbara, D., Jajodia, S. (eds.): Application of Data Mining in Computer Security, Advances in Information Security, vol. 6. Springer, Berlin (2002)zbMATHGoogle Scholar
  11. 11.
    Helm, J.E., AhmadBeygi, S., Van Oyen, M.P.: Design and analysis of hospital admission control for operational effectiveness. Prod. Oper. Manag. 20(3), 359 (2011)CrossRefGoogle Scholar
  12. 12.
    Chen, Z., King, W., Pearcey, R., Kerba, M., Mackillop, W.J.: The relationship between waiting time for radiotherapy and clinical outcomes: a systematic review of the literature. Radiother. Oncol. 87(1), 3 (2008)CrossRefGoogle Scholar
  13. 13.
    Guerriero, F., Guido, R.: Operational research in the management of the operating theatre: a survey. Health Care Manag. Sci. 14(1), 89 (2011)CrossRefGoogle Scholar
  14. 14.
    Vansteenwegen, P., Van Oudheusden, D.: Decreasing the passenger waiting time for an intercity rail network. Transp. Res. Part B: Methodol. 41(4), 478 (2007)CrossRefGoogle Scholar
  15. 15.
    Kelly, C.: A framework for improving operational effectiveness and cost efficiency in emergency planning and response. Disaster Prev. Manag. Int. J. 4(3), 25 (1995)CrossRefGoogle Scholar
  16. 16.
    Robbins, T.R., Medeiros, D.J., Dum, P.: Evaluating arrival rate uncertainty in call centers. In: Proceedings of the 2006 Winter Simulation Conference, pp. 2180–2187. IEEE (2006)Google Scholar
  17. 17.
    Jack, E.P., Bedics, T.A., McCary, C.E.: Operational challenges in the call center industry: a case study and resource-based framework. Manag. Serv. Qual. Int. J. 16(5), 477 (2006)CrossRefGoogle Scholar
  18. 18.
    Tijms, H.: New and old results for the M/D/c queue. AEU Int. J. Electron. Commun. 60(2), 125 (2006)CrossRefGoogle Scholar
  19. 19.
    Marianov, V., Serra, D.: Location models for airline hubs behaving as M/D/c queues. Comput. Oper. Res. 30(7), 983 (2003)CrossRefzbMATHGoogle Scholar
  20. 20.
    Julisch, K., Dacier, M.: Mining intrusion detection alarms for actionable knowledge. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 366–375 (2002)Google Scholar
  21. 21.
    Goodall, J.R., Lutters, W.G., Komlodi, A.: I know my network: collaboration and expertise in intrusion detection. In: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, pp. 342–345 (2004)Google Scholar
  22. 22.
    Ganesan, R., Jajodia, S., Cam, H.: Optimal scheduling of cybersecurity analyst for minimizing risk. ACM Trans. Intell. Syst. Technol. 8(4), 52:1–52:33 (2017). doi: 10.1145/2914795
  23. 23.
    Ganesan, R., Jajodia, S., Shah, A., Cam, H.: Dynamic scheduling of cybersecurity analysts for minimizing risk using reinforcement learning. ACM Trans. Intell. Syst. Technol. 8(1), 4:1 (2016). doi: 10.1145/2882969 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  • Ankit Shah
    • 1
  • Rajesh Ganesan
    • 1
  • Sushil Jajodia
    • 1
    Email author
  • Hasan Cam
    • 2
  1. 1.Center for Secure Information SystemsGeorge Mason UniversityFairfaxUSA
  2. 2.Army Research LaboratoryAdelphiUSA

Personalised recommendations