Abstract
This paper discusses the use of cyber security exercises and competitions to produce data valuable for security research. Cyber security exercises and competitions are primarily arranged to train participants and/or to offer competence contests for those with a profound interest in security. This paper discusses how exercises and competitions can be used as a basis for experimentation in the security field. The conjecture is that (1) they make it possible to control a number of variables of relevance to security and (2) the results can be used to study several topics in the security field in a meaningful way. Among other things, they can be used to validate security metrics and to assess the impact of different protective measures on the security of a system.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Verendel, V.: Quantified security is a weak hypothesis: a critical survey of results and assumptions. In: New Security Paradigms Workshop, pp. 37–50 (2009)
Geer Jr., D., Hoo, K.S., Jaquith, A.: Information security: why the future belongs to the quants. IEEE Security & Privacy 1, 24–32 (2003)
Kotulic, A., Clark, J.G.: Why there aren’t more information security research studies. Information & Management 41, 597–607 (2004)
Gal-Or, E., Ghose, A.: The Economic Incentives for Sharing Security Information. Information Systems Research 16, 186–208 (2005)
Gordon, L.: Sharing information on computer systems security: An economic analysis. Journal of Accounting and Public Policy 22, 461–485 (2003)
Wilander, J., Nikiforakis, N., Younan, Y., Kamkar, M., Joosen, W.: RIPE: Runtime Intrusion Prevention Evaluator. In: Proceedings of the 27th Annual Computer Security Applications Conference, ACSAC, pp. 41–50 (2011)
Shacham, H., Page, M., Pfaff, B., Goh, E.J., Modadugu, N., Boneh, D.: On the effectiveness of address-space randomization. In: Proceedings of the 11th ACM Conference on Computer and Communication Security, pp. 298–307 (2004)
Khattab, S.M., Sangpachatanaruk, C., Melhem, R., Znati, T.: Proactive server roaming for mitigating denial-of-service attacks. In: Proceedings of International Conference on Information Technology: Research and Education, ITRE 2003, pp. 286–290. IEEE (2003)
Ktata, F.B., Kadhi, N.E., Ghédira, K.: Agent IDS based on Misuse Approach. Journal of Software 4, 495–507 (2009)
Conti, G., Babbitt, T., Nelson, J.: Hacking Competitions and Their Untapped Potential for Security Education. IEEE Security & Privacy, 56–59 (2011)
Fanelli, R.L., O’Connor, T.J.: Experiences with practice-focused undergraduate security education. In: Proceedings of the 3rd Workshop on Cyber Security, Washington, DC, United states (2010)
Werther, J., Zhivich, M., Leek, T.: Experiences in cyber security education: The mit lincoln laboratory capture-the-flag exercise. In: The 4th Workshop on Cyber Secuirty Experimentation and Test, San Francisco, CA, United states (2011)
Vigna, G.: The UCSB iCTF, http://ictf.cs.ucsb.edu/
Polytechnic Institute of NYU: CSAW - CyberSecurity Competition, http://www.poly.edu/csaw2011
Cyber Security Challenge: Cyber Security Challange, https://cybersecuritychallenge.org.uk/
National Collegiate Cyber Defense Competition: Welcom to the National Collegiate Cyber Defense Competition, http://www.nationalccdc.org/
Patriciu, V.V., Furtuna, A.C.: Guide for designing cyber security exercises. In: Proceedings of the 8th WSEAS International Conference on E-Activities and Information Security and Privacy, pp. 172–177. World Scientific and Engineering Academy and Society, WSEAS (2009)
Wagner, P.J., Wudi, J.M.: Designing and implementing a cyberwar laboratory exercise for a computer security course. In: Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education - SIGCSE 2004, p. 402 (2004)
Schepens, W.J., Ragsdale, D.J., Surdu, J.R., Schafer, J.: The Cyber Defense Exercise: An evaluation of the effectiveness of information assurance education. The Journal of Information Security 1 (2002)
Conklin, A.: Cyber Defense Competitions and Information Security Education: An Active Learning Solution for a Capstone Course. In: Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS 2006), p. 220b. IEEE (2006)
Hoffman, L.J., Rosenberg, T., Dodge, R., Ragsdale, D.: Exploring a National Cybersecurity Exercise for Universities. IEEE Security and Privacy Magazine, 27–33 (2005)
Childers, N., Boe, B., Cavallaro, L., Cavedon, L., Cova, M., Egele, M., Vigna, G.: Organizing Large Scale Hacking Competitions. In: Kreibich, C., Jahnke, M. (eds.) DIMVA 2010. LNCS, vol. 6201, pp. 132–152. Springer, Heidelberg (2010)
Schepens, W.J., James, J.R.: Architecture of a cyber defense competition. In: IEEE International Conference on Systems, Man and Cybernetics, pp. 4300–4305. IEEE (2003)
Keppel, G., Wickens, T.D.: Design and analysis: a researcher’s handbook. Pearson Education, Upper Saddle River (2004)
McQueen, M.A., Boyer, W.F., Flynn, M.A., Beitel, G.A.: Time-to-Compromise Model for Cyber Risk Reduction Estimation. In: Gollmann, D., Massacci, F., Yautsiukhin, A. (eds.) Quality of Protection, pp. 49–64. Springer US, Boston (2006)
Jonsson, E., Olovsson, T.: A quantitative model of the security intrusion process based on attacker behavior. IEEE Transactions on Software Engineering 23, 235–245 (1997)
Schudel, G., Wood, B., Parks, R.: Modeling behavior of the cyber-terrorist. In: Proceeding of Workshop on RAND National Security Research Division, pp. 45–59 (2000)
Branlat, M., Morison, A.: Challenges in managing uncertainty during cyber events: Lessons from the staged-world study of a large-scale adversarial cyber security exercise. In: Human Systems Integration Symposium (2011)
Olovsson, T., Jonsson, E., Brocklehurst, S., Littlewood, B.: Data collection for security fault forecasting: Pilot experiment, Dept. of Computer Eng., Chalmers Univ. of Technology, and ESPRIT/BRA Project no. 6362 (PDCS2), Toulouse (1993)
Levin, D.: Lessons learned in using live red teams in IA experiments. In: Proceedings of DARPA Information Survivability Conference and Exposition, pp. 110–119. IEEE (2003)
Kewley, D.L., Bouchard, J.F.: DARPA Information Assurance Program dynamic defense experiment summary. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 31, 331–336 (2001)
Guard, L., Crossland, M., Paprzycki, M., Thomas, J.: Developing an empirical study of how qualified subjects might be selected for IT system security penetration testing. Citeseer 2, 413–424 (2004)
Dodge, R.C., Carver, C., Ferguson, A.J.: Phishing for user security awareness. Computers & Security 26, 73–80 (2007)
Kraemer, S., Carayon, P., Duggan, R.: Red team performance for improved computer security. In: Human Factors and Ergonomics Society Annual Meeting Proceedings. Human Factors and Ergonomics Society, pp. 1605–1609 (2004)
Mirkovic, J., Reiher, P., Papadopoulos, C., Hussain, A., Shepard, M., Berg, M., Jung, R.: Testing a Collaborative DDoS Defense In a Red Team/Blue Team Exercise. IEEE Transactions on Computers 57, 1098–1112 (2008)
Kewley, D.L., Lowry, J.: Observations on the effects of defense in depth on adversary behavior in cyber warfare. In: Proceedings of the IEEE SMC Information Assurance Workshop, pp. 1–8 (2001)
Mitropoulos, S., Patsos, D., Douligeris, C.: On Incident Handling and Response: A state-of-the-art approach. Computers & Security 25, 351–370 (2006)
Werlinger, R., Muldner, K., Hawkey, K., Beznosov, K.: Preparation, detection, and analysis: the diagnostic work of IT security incident response. Information Management & Computer Security 18, 26–42 (2010)
Meyers, M.: Computer forensics: the need for standardization and certification. International Journal of Digital Evidence 3, 1–11 (2004)
Sommestad, T., Hunstad, A.: Intrusion detection and the role of the system administrator. In: Proceedings of International Symposium on Human Aspects of Information Security & Assurance, Crete, Greece (2012)
Holm, H., Sommestad, T., Franke, U., Ekstedt, M.: Success rate of remote code execution attacks – expert assessments and observations. Journal of Universal Computer Science 18, 732–749 (2012)
Holm, H., Ekstedt, M., Andersson, D.: Empirical analysis of system-level vulnerability metrics through actual attacks. IEEE Transactions on Dependable and Secure Computing (accepted, 2012)
Egele, M., Caillat, B., Stringhini, G.: Hit’em where it hurts: a live security exercise on cyber situational awareness. Computer Security (2011)
Schudel, G., Wood, B.: Adversary work factor as a metric for information assurance. In: Proceedings of the 2000 Workshop on New Security Paradigms, pp. 23–30. ACM (2001)
Levin, D.: Lessons learned in using live red teams in IA experiments. In: Proceedings DARPA Information Survivability Conference and Exposition, pp. 110–119 (2003)
Ryder, D., Levin, D., Lowry, J.: Defense in depth: A focus on protecting the endpoint clients from network attack. In: Proceedings of the IEEE SMC Information Assurance Workshop (2002)
Paulauskas, N., Garsva, E.: Attacker skill level distribution estimation in the system mean time-to-compromise. In: 1st International Conference on Information Technology, IT 2008, pp. 1–4. IEEE (2008)
Leversage, D., Byres, E.: Comparing Electronic Battlefields: Using Mean Time-To-Compromise as a Comparative Security Metric. Computer Network Security 1, 213–227 (2007)
McQueen, M., Boyer, W., Flynn, M., Beitel, G.: Time-to-compromise model for cyber risk reduction estimation. Quality of Protection (2006)
McHugh, J.: Quality of protection: Measuring the unmeasurable? In: Proceedings of the 2nd ACM Workshop on Quality of Protection, QoP 2006, Co-located with the 13th ACM Conference on Computer and Communications Security, CCS 2006, Alexandria, VA, pp. 1–2 (2006)
Sommestad, T., Holm, H., Ekstedt, M.: Effort estimates for vulnerability discovery projects. In: HICSS 2012: Proceedings of the 45th Hawaii International Conference on System Sciences, Maui, HI, USA (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Sommestad, T., Hallberg, J. (2012). Cyber Security Exercises and Competitions as a Platform for Cyber Security Experiments. In: Jøsang, A., Carlsson, B. (eds) Secure IT Systems. NordSec 2012. Lecture Notes in Computer Science, vol 7617. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34210-3_4
Download citation
DOI: https://doi.org/10.1007/978-3-642-34210-3_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34209-7
Online ISBN: 978-3-642-34210-3
eBook Packages: Computer ScienceComputer Science (R0)