Skip to main content
SpringerLink
Log in

Poisoning Attack for Inter-agent Transfer Learning

  • Conference paper
  • First Online:
Security and Privacy in Communication Networks (SecureComm 2021)

  • 997 Accesses

Abstract

In reinforcement learning, high sample complexity is a big challenge to deal with. Inter-agent transfer learning is one solution to this challenge that can leverage the experience of other more competent agents. In this paradigm, a student can make a query to the teacher and the teacher will give some action advice given the current state. However, most previous works ignored the instruction reliability problem. In this work, we investigate the instruction reliability issue based on the one-to-one teaching framework and formulate the poisoning attack as an optimization problem. By solving the optimization problem, the attacker can significantly influence the performance of the student in three different query models. Evaluation highlights that we need to consider the instruction reliability when using teacher-student frameworks in reinforcement learning.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Amir, O., Kamar, E., Kolobov, A., Grosz, B.J.: Interactive teaching strategies for agent training. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pp. 804–811 (2016)

    Google Scholar 

  2. Barekatain, M., Yonetani, R., Hamaya, M.: Multipolar: multi-source policy aggregation for transfer reinforcement learning between diverse environmental dynamics. arXiv preprint arXiv:1909.13111 (2019)

  3. Barreno, M., Nelson, B., Sears, R., Joseph, A.D., Tygar, J.D.: Can machine learning be secure? In: Proceedings of the 2006 ACM Symposium on Information, Computer and Communications Security, pp. 16–25 (2006)

    Google Scholar 

  4. Biggio, B., Nelson, B., Laskov, P.: Poisoning attacks against support vector machines. In: Proceedings of the 29th International Conference on International Conference on Machine Learning, pp. 1467–1474 (2012)

    Google Scholar 

  5. Brys, T., Harutyunyan, A., Suay, H.B., Chernova, S., Taylor, M.E., Nowé, A.: Reinforcement learning from demonstration through shaping. In: Twenty-fourth International Joint Conference on Artificial Intelligence (2015)

    Google Scholar 

  6. Cao, X., Jia, J., Gong, N.Z.: Data poisoning attacks to local differential privacy protocols. In: 30th USENIX Security Symposium (USENIX Security 2021) (2021)

    Google Scholar 

  7. Clouse, J.A., Utgoff, P.E.: A teaching method for reinforcement learning. In: Machine Learning Proceedings 1992, pp. 92–101. Elsevier (1992)

    Google Scholar 

  8. Clouse, J.A.: On integrating apprentice learning and reinforcement learning. University of Massachusetts Amherst (1996)

    Google Scholar 

  9. Da Silva, F.L., Costa, A.H.R.: A survey on transfer learning for multiagent reinforcement learning systems. J. Artif. Intell. Res. 64, 645–703 (2019)

    Article  MathSciNet  Google Scholar 

  10. Da Silva, F.L., Hernandez-Leal, P., Kartal, B., Taylor, M.E.: Uncertainty-aware action advising for deep reinforcement learning agents. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 5792–5799 (2020)

    Google Scholar 

  11. Da Silva, F.L., Warnell, G., Costa, A.H.R., Stone, P.: Agents teaching agents: a survey on inter-agent transfer learning. Auton. Agent. Multi-Agent Syst. 34(1), 1–17 (2020)

    Article  Google Scholar 

  12. Everitt, T., Krakovna, V., Orseau, L., Legg, S.: Reinforcement learning with a corrupted reward channel. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 4705–4713 (2017)

    Google Scholar 

  13. Fang, M., Cao, X., Jia, J., Gong, N.: Local model poisoning attacks to byzantine-robust federated learning. In: 29th USENIX Security Symposium (USENIX Security 2020), pp. 1605–1622 (2020)

    Google Scholar 

  14. Fang, M., Yang, G., Gong, N.Z., Liu, J.: Poisoning attacks to graph-based recommender systems. In: Proceedings of the 34th Annual Computer Security Applications Conference, pp. 381–392 (2018)

    Google Scholar 

  15. Fernández, F., Veloso, M.: Probabilistic policy reuse in a reinforcement learning agent. In: Proceedings of the fifth International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 720–727 (2006)

    Google Scholar 

  16. Griffith, S., Subramanian, K., Scholz, J., Isbell, C.L., Thomaz, A.L.: Policy shaping: Integrating human feedback with reinforcement learning. Georgia Institute of Technology (2013)

    Google Scholar 

  17. Jagielski, M., Oprea, A., Biggio, B., Liu, C., Nita-Rotaru, C., Li, B.: Manipulating machine learning: poisoning attacks and countermeasures for regression learning. In: 2018 IEEE Symposium on Security and Privacy (SP), pp. 19–35. IEEE (2018)

    Google Scholar 

  18. Kakade, S.M.: On the sample complexity of reinforcement learning. Ph.D. thesis, UCL (University College London) (2003)

    Google Scholar 

  19. Lattimore, T., Hutter, M., Sunehag, P.: The sample-complexity of general reinforcement learning. In: International Conference on Machine Learning, pp. 28–36. PMLR (2013)

    Google Scholar 

  20. Li, B., Wang, Y., Singh, A., Vorobeychik, Y.: Data poisoning attacks on factorization-based collaborative filtering. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 1893–1901 (2016)

    Google Scholar 

  21. Ma, Y., Zhang, X., Sun, W., Zhu, X.: Policy poisoning in batch reinforcement learning and control. In: Advances in Neural Information Processing Systems (2019)

    Google Scholar 

  22. Omidshafiei, S., et al.: Learning to teach in cooperative multiagent reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 6128–6136 (2019)

    Google Scholar 

  23. Rakhsha, A., Radanovic, G., Devidze, R., Zhu, X., Singla, A.: Policy teaching via environment poisoning: training-time adversarial attacks against reinforcement learning. In: International Conference on Machine Learning, pp. 7974–7984. PMLR (2020)

    Google Scholar 

  24. Shafahi, A., et al.: Poison frogs! Targeted clean-label poisoning attacks on neural networks. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6106–6116 (2018)

    Google Scholar 

  25. Sharif, M., Bhagavatula, S., Bauer, L., Reiter, M.K.: Accessorize to a crime: real and stealthy attacks on state-of-the-art face recognition. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1528–1540 (2016)

    Google Scholar 

  26. Suciu, O., Marginean, R., Kaya, Y., Daume III, H., Dumitras, T.: When does machine learning \(fail\)? Generalized transferability for evasion and poisoning attacks. In: 27th USENIX Security Symposium (USENIX Security 2018), pp. 1299–1316 (2018)

    Google Scholar 

  27. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)

    MATH  Google Scholar 

  28. Sutton, R.S., Barto, A.G., et al.: Introduction to Reinforcement Learning, vol. 135. MIT Press, Cambridge (1998)

    MATH  Google Scholar 

  29. Taylor, A., Duparic, I., Galván-López, E., Clarke, S., Cahill, V.: Transfer learning in multi-agent systems through parallel transfer (2013)

    Google Scholar 

  30. Taylor, M.E., Stone, P.: Transfer learning for reinforcement learning domains: a survey. J. Mach. Learn. Res. 10(7), 1633–1685 (2009)

    Google Scholar 

  31. Taylor, M.E., Stone, P., Liu, Y.: Transfer learning via inter-task mappings for temporal difference learning. J. Mach. Learn. Res. 8(9), 2125–2167 (2007)

    Google Scholar 

  32. Torrey, L., Taylor, M.: Teaching on a budget: agents advising agents in reinforcement learning. In: Proceedings of the 2013 International Conference on Autonomous Agents and Multi-agent Systems, pp. 1053–1060 (2013)

    Google Scholar 

  33. Wilson, A., Fern, A., Ray, S., Tadepalli, P.: Multi-task reinforcement learning: a hierarchical Bayesian approach. In: Proceedings of the 24th International Conference on Machine Learning, pp. 1015–1022 (2007)

    Google Scholar 

  34. Yang, G., Gong, N.Z., Cai, Y.: Fake co-visitation injection attacks to recommender systems. In: NDSS (2017)

    Google Scholar 

  35. Zimmer, M., Viappiani, P., Weng, P.: Teacher-student framework: a reinforcement learning approach. In: AAMAS Workshop Autonomous Robots and Multirobot Systems (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Purdue University, West Lafayette, IN, 47906, USA

    Zelei Cheng & Zuotian Li

Authors
  1. Zelei Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zuotian Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zelei Cheng .

Editor information

Editors and Affiliations

  1. Télécom SudParis, Institut Polytechnique de Paris, Palaiseau, France

    Joaquin Garcia-Alfaro

  2. University of Kent Canterbury, Canterbury, Kent, UK

    Shujun Li

  3. University of Washington, Seattle, WA, USA

    Radha Poovendran

  4. Télécom SudParis, Institut Polytechnique de Paris, Palaiseau, France

    Hervé Debar

  5. Google Inc., New York, NY, USA

    Moti Yung

Rights and permissions

Reprints and permissions

Copyright information

© 2021 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cheng, Z., Li, Z. (2021). Poisoning Attack for Inter-agent Transfer Learning. In: Garcia-Alfaro, J., Li, S., Poovendran, R., Debar, H., Yung, M. (eds) Security and Privacy in Communication Networks. SecureComm 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 399. Springer, Cham. https://doi.org/10.1007/978-3-030-90022-9_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-90022-9_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-90021-2

  • Online ISBN: 978-3-030-90022-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation