Beware the Soothsayer: From Attack Prediction Accuracy to Predictive Reliability in Security Games

  • Benjamin Ford
  • Thanh Nguyen
  • Milind Tambe
  • Nicole Sintov
  • Francesco Delle Fave
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9406)


Interdicting the flow of illegal goods (such as drugs and ivory) is a major security concern for many countries. The massive scale of these networks, however, forces defenders to make judicious use of their limited resources. While existing solutions model this problem as a Network Security Game (NSG), they do not consider humans’ bounded rationality. Previous human behavior modeling works in Security Games, however, make use of large training datasets that are unrealistic in real-world situations; the ability to effectively test many models is constrained by the time-consuming and complex nature of field deployments. In addition, there is an implicit assumption in these works that a model’s prediction accuracy strongly correlates with the performance of its corresponding defender strategy (referred to as predictive reliability). If the assumption of predictive reliability does not hold, then this could lead to substantial losses for the defender. In the following paper, we (1) first demonstrate that predictive reliability is indeed strong for previous Stackelberg Security Game experiments. We also run our own set of human subject experiments in such a way that models are restricted to learning on dataset sizes representative of real-world constraints. In the analysis on that data, we demonstrate that (2) predictive reliability is extremely weak for NSGs. Following that discovery, however, we identify (3) key factors that influence predictive reliability results: the training set’s exposed attack surface and graph structure.


Prediction Accuracy Source Node Target Node Predictive Reliability Maximum Absolute Error 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This research was supported by MURI Grant W911NF-11-1-0332 and by CREATE under grant number 2010-ST-061-RE0001.


  1. 1.
    Abbasi, Y.D., Short, M., Sinha, A., Sintov, N., Zhang, C., Tambe, M.: Human adversaries in opportunistic crime security games: evaluating competing bounded rationality models. In: 3rd Conference on Advances in Cognitive Systems (2015)Google Scholar
  2. 2.
    Bell, M.G.H., Kanturska, U., Schmöcker, J.D., Fonzone, A.: Attacker-defender models and road network vulnerability. Philos. Trans. Roy. Soc. A: Math. Phys. Eng. Sci. 366(1872), 1893–1906 (2008)CrossRefzbMATHGoogle Scholar
  3. 3.
    Camerer, C.: Behavioral Game Theory: Experiments in Strategic Interaction. Princeton University Press, Princeton (2003)Google Scholar
  4. 4.
    Correa, J.R., Harks, T., Kreuzen, V.J.C., Matuschke, J.: Fare evasion in transit networks. In: CoRR (2014)Google Scholar
  5. 5.
    Cui, J., John, R.S.: Empirical comparisons of descriptive multi-objective adversary models in stackelberg security games. In: Poovendran, R., Saad, W. (eds.) GameSec 2014. LNCS, vol. 8840, pp. 309–318. Springer, Heidelberg (2014) Google Scholar
  6. 6.
    Eppstein, D., Goodrich, M.T.: Studying (non-planar) road networks through an algorithmic lens. In: Proceedings of the 16th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, p. 16. ACM (2008)Google Scholar
  7. 7.
    Fave, F.M.D., Jiang, A.X., Yin, Z., Zhang, C., Tambe, M., Kraus, S., Sullivan, J.: Game-theoretic security patrolling with dynamic execution uncertainty and a case study on a real transit system. J. Artif. Intell. Res. 50, 321–367 (2014)zbMATHGoogle Scholar
  8. 8.
    Gutfraind, A., Hagberg, A., Pan, F.: Optimal interdiction of unreactive markovian evaders. In: Hooker, J.N., van Hoeve, W.-J. (eds.) CPAIOR 2009. LNCS, vol. 5547, pp. 102–116. Springer, Heidelberg (2009) CrossRefGoogle Scholar
  9. 9.
    Jain, M., Conitzer, V., Tambe, M.: Security scheduling for real-world networks. In: AAMAS (2013)Google Scholar
  10. 10.
    Jain, M., Korzhyk, D., Vanek, O., Conitzer, V., Pechoucek, M., Tambe, M.: A double oracle algorithm for zero-sum security games on graphs. In: AAMAS (2011)Google Scholar
  11. 11.
    Kar, D., Fang, F., Fave, F.D., Sintov, N., Tambe, M.: “A game of thrones”: when human behavior models compete in repeated stackelberg security games. In: AAMAS (2015)Google Scholar
  12. 12.
    Manadhata, P., Wing, J.M.: Measuring a system’s attack surface. Technical report, DTIC Document (2004)Google Scholar
  13. 13.
    Morton, D.P., Pan, F., Saeger, K.J.: Models for nuclear smuggling interdiction. IIE Trans. 39(1), 3–14 (2007)CrossRefGoogle Scholar
  14. 14.
    Nguyen, T.H., Yang, R., Azaria, A., Kraus, S., Tambe, M.: Analyzing the effectiveness of adversary modeling in security games. In: AAAI (2013)Google Scholar
  15. 15.
    Shieh, E., An, B., Yang, R., Tambe, M., Baldwin, C., DiRenzo, J., Maule, B., Meyer, G.: Protect: a deployed game theoretic system to protect the ports of the united states. In: AAMAS (2012)Google Scholar
  16. 16.
    Tambe, M.: Security and Game Theory: Algorithms, Deployed Systems. Lessons Learned. Cambridge University Press, New York (2011) CrossRefGoogle Scholar
  17. 17.
    Tsai, J., Yin, Z., Kwak, J.y., Kempe, D., Kiekintveld, C., Tambe, M.: Urban security: Game-theoretic resource allocation in networked physical domains. In: AAAI (2010)Google Scholar
  18. 18.
    Yang, R., Fang, F., Jiang, A.X., Rajagopal, K., Tambe, M., Maheswaran, R.: Modeling human bounded rationality to improve defender strategies in network security games. In: HAIDM Workshop at AAMAS (2012)Google Scholar
  19. 19.
    Yang, R., Ordonez, F., Tambe, M.: Computing optimal strategy against quantal response in security games. In: AAMAS (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Benjamin Ford
    • 1
  • Thanh Nguyen
    • 1
  • Milind Tambe
    • 1
  • Nicole Sintov
    • 1
  • Francesco Delle Fave
    • 1
  1. 1.University of Southern CaliforniaLos AngelesUSA

Personalised recommendations