Skip to main content
SpringerLink
Log in

Exploiting spatio-temporal patterns using partial-state reinforcement learning in a synthetically augmented environment

  • Regular Paper
  • Published:
Progress in Artificial Intelligence Aims and scope Submit manuscript

Abstract

Responding to or anticipating a sequence of events caused by adversarial human actors, such as crimes, can be a difficult task. Reinforcement learning has not been highly utilized as a method for positioning agents to respond to such events. In our earlier work, which was applied to positioning naval vessel agents to respond to Somali maritime piracy attacks, we developed a method to synthetically augment the information in the events’ environment with digital pheromones and other information augmenters, used the resulting augmenter signatures as states that agents could react to, and applied reinforcement learning to exploit regularities in the timing and location of events to position agents in spatio-temporal proximity of anticipated events. This work extends that methodology with a new learning boosting method wherein learning is improved as partial augmenter signatures are reinforced, which is not possible when learning is based only on the aggregated state. The enhanced methodology is applied to positioning police patrols in response to a sequence of business robberies in Denver, Colorado and its effectiveness is analyzed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Barbosa, S.E., Petty, M.D.: Reinforcement learning in an environment synthetically augmented with digital pheromones. Adv. Artif. Intell. 2014, 1–23 (2014). doi:10.1155/2014/932485

  2. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)

    Google Scholar 

  3. Watkins, C.H., Dayan, P.: Q-learning. Mach. Learn. 8(3), 279–292 (1992)

    MATH  Google Scholar 

  4. da Silva, B.C., Basso, E.W., Bazzan, A.L.C., Engel, P.M.: Dealing with nonstationary environments using context detection. In: Proceedings of the 23rd International Conference on Machine Learning, pp 217–224 (2006)

  5. Gordon, D.M.: Ants at Work: How an Insect Society is Organized. The Free Press, New York (1999)

    Google Scholar 

  6. Dorigo, M., Stützle, T.: Ant Colony Optimization. MIT Press, Cambridge (2004)

    Book  MATH  Google Scholar 

  7. Huang, K.L., Liao, C.J.: Ant colony optimization combined with taboo search for the job shop scheduling problem. Comput. Oper. Res. 35(4), 1030–1046 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  8. Lee, Z.J., Lee, C.Y., Su, S.F.: An immunity-based ant colony optimization algorithm for solving weapon-target assignment problem. Appl. Soft Comput. 2(1), 39–47 (2002)

    Article  MathSciNet  Google Scholar 

  9. Bautista, J., Pereira, J.: Ant algorithms for a time and space constrained assembly line balancing problem. Eur. J. Oper. Res. 177(3), 2016–2032 (2007)

    Article  MATH  Google Scholar 

  10. Gosnell, M., O’Hara, S., Simon, M.: Spatially decomposed searching by heterogeneous unmanned systems. In: Proceedings of the International Conference on Integration of Knowledge Intensive Multi-Agent Systems (2007)

  11. Fu, J.G.M., Ang, M.H.: Probabilistic ants (PAnts) in multi-agent patrolling. In: Proceedings of the International Conference on Advanced Intelligent Mechatronics, pp. 1371–1376 (2009)

  12. Chu, H., Glad, A., Simonin, O., Sempe, F., Drogoul, A., Charpillet, F.: Swarm approaches for the patrolling problem, information propagation vs. pheromone evaporation. In: ICTAI’07 IEEE International Conference on Tools with Artificial Intelligence, pp 442–449 (2007)

  13. Sauter, J.A., Matthews, R., Parunak, H.V.D., Brueckner, S.: Performance of digital pheromones for swarming vehicle control. In: Proceedings of the Conference on Autonomous Agents and Multiagent Systems, pp. 903–910 (2005)

  14. Monekosso, N., Remagnino, P.: An analysis of the pheromone Q-learning algorithm. In: Proceedings of the Eighth Ibero-American Conference on Artificial Intelligence, pp 224–232 (2002)

  15. Furtado, V., Melo, A., Coelho, A., Menezes, R., Perrone, R.: A bio-inspired crime simulation model. Decis. Support Syst. 48(1), 282–292 (2009)

    Article  Google Scholar 

  16. Bowers, K.J., Johnson, S.D., Pease, K.: Prospective hot-spotting the future of crime mapping? Br. J. Criminol. 44(5), 641–658 (2004)

    Article  Google Scholar 

  17. Li, L., Jiang, Z., Duan, N., Dong, W., Hu, K., Sun, W.: Police patrol service optimization based on the spatial pattern of hotspots. In: Service Operations, Logistics, and Informatics (SOLI), 2011 IEEE International Conference, pp. 45–50 (2011)

  18. Jones, P.A., Brantingham, P.J., Chayes, L.R.: Statistical models of criminal behavior: the effects of law enforcement actions. Math. Models Methods Appl. Sci. 20(supp01), 1397–1423 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  19. Mohler, G.O., Short, M.B., Brantingham, P.J., Schoenberg, F.P., Tita, G.E.: Self-exciting point process modeling of crime. J. Am. Stat. Assoc. 106(493), 100–108 (2011)

  20. Denver Open Data Catalog, Crime Data. http://data.denvergov.org/dataset/city-and-county-of-denver-crime (2014). Accessed Feb 2014

  21. Denver Police Department. In: Wikipedia, The Free Encyclopedia. Retrieved 13:47, 18 February 2014. http://en.wikipedia.org/w/index.php?title=Denver_Police_Department&oldid=586785112 (2013). Accessed 19 Dec 2013

  22. Denver Police Department. http://www.denvergov.org/police (2014). Accessed Feb 2014

  23. Johnson, S.D., Bernasco, W., Bowers, K.J., Elffers, H., Ratcliffe, J., Rengert, G., Townsley, M.: Space-time patterns of risk: a cross national assessment of residential burglary victimization. J. Quant. Criminol. 23(3), 201–219 (2007)

    Article  Google Scholar 

  24. Bolstad, W.M.: Introduction to Bayesian Statistics. Wiley, Hoboken, New Jersey (2007)

  25. Stone, J.V.: Bayes’ Rule: A Tutorial Introduction to Bayesian Analysis (2013)

  26. Denver Street System. In: Wikipedia, The Free Encyclopedia. Retrieved 13:49, February 18, 2014. http://en.wikipedia.org/w/index.php?title=Street_system_of_Denver&oldid=594739065 (2014). Accessed 9 Feb 2014

Download references

Author information

Authors and Affiliations

  1. University of Alabama in Huntsville, 301 Sparkman Drive, Huntsville, AL, 35899, USA

    Salvador E. Barbosa & Mikel D. Petty

Authors
  1. Salvador E. Barbosa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mikel D. Petty
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Salvador E. Barbosa.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Barbosa, S.E., Petty, M.D. Exploiting spatio-temporal patterns using partial-state reinforcement learning in a synthetically augmented environment. Prog Artif Intell 3, 55–71 (2015). https://doi.org/10.1007/s13748-014-0057-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13748-014-0057-2

Keywords

Search

Navigation