Advertisement

Learning to Control Random Boolean Networks: A Deep Reinforcement Learning Approach

  • Georgios PapagiannisEmail author
  • Sotiris Moschoyiannis
Conference paper
Part of the Studies in Computational Intelligence book series (SCI, volume 881)

Abstract

In this paper we describe the application of a Deep Reinforcement Learning agent to the problem of control of Gene Regulatory Networks (GRNs). The proposed approach is applied to Random Boolean Networks (RBNs) which have extensively been used as a computational model for GRNs. The ability to control GRNs is central to therapeutic interventions for diseases such as cancer. That is, learning to make such interventions as to direct the GRN from some initial state towards a desired attractor, by allowing at most one intervention per time step. Our agent interacts directly with the environment; being an RBN, without any knowledge of the underlying dynamics, structure or connectivity of the network. We have implemented a Deep Q Network with Double Q Learning that is trained by sampling experiences from the environment using Prioritized Experience Replay. We show that the proposed novel approach develops a policy that successfully learns how to control RBNs significantly larger than previous learning implementations. We also discuss why learning to control an RBN with zero knowledge of its underlying dynamics is important and argue that the agent is encouraged to discover and perform optimal control interventions in regard to cost and number of interventions.

Keywords

Controllability Complex networks Q learning Prioritized Experience Replay Interventions 

References

  1. 1.
    Akutsu, T., Hayashida, M., Ching, W., Ng, M.: Control of Boolean networks: hardness results and algorithms for tree structured networks. J. Theoret. Biol. 244(4), 670–679 (2007).  https://doi.org/10.1016/j.jtbi.2006.09.023MathSciNetCrossRefGoogle Scholar
  2. 2.
    Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (2010)zbMATHGoogle Scholar
  3. 3.
    Chen, S., Hong, Y.: Control of random Boolean networks via average sensitivity of Boolean functions. Chin. Phys. B 20(3), 036401 (2011).  https://doi.org/10.1088/1674-1056/20/3/036401CrossRefGoogle Scholar
  4. 4.
    Fornasini, E., Valcher, M.: Optimal control of Boolean control networks. IEEE Trans. Autom. Control 59(5), 1258–1270 (2014).  https://doi.org/10.1109/TAC.2013.2294821MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Gao, J., Liu, Y., D’Souza, R., Barabási, A.: Target control of complex networks. Nat. Commun. 5(5415) (2014).  https://doi.org/10.1038/ncomms6415
  6. 6.
    Gates, A., Rocha, L.: Control of complex networks requires both structure and dynamics. Sci. Rep. 6, 24,456 (2016).  https://doi.org/10.1038/srep24456CrossRefGoogle Scholar
  7. 7.
    Hasselt, H.: Double Q-learning. In: Advances in Neural Information Processing Systems 23, pp. 2613–2621. Curran Associates, Inc. (2010)Google Scholar
  8. 8.
    Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double Q-learning. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp. 2094–2100. AAAI Press (2016)Google Scholar
  9. 9.
    Huang, S.: Gene expression profiling, genetic networks, and cellular states: an integrating concept for tumorigenesis and drug discovery. J. Mol. Med. 77(6), 469–480 (1999).  https://doi.org/10.1007/s001099900CrossRefGoogle Scholar
  10. 10.
    Huang, S., Ingber, D.: Shape-dependent control of cell growth, differentiation, and apoptosis: switching between attractors in cell regulatory networks. Exp. Cell Res. 261(1), 91–103 (2000).  https://doi.org/10.1006/excr.2000.5044CrossRefGoogle Scholar
  11. 11.
    Kauffman, S.: Metabolic stability and epigenesis in randomly constructed genetic nets. J. Theoret. Biol. 22(3), 437–467 (1969).  https://doi.org/10.1016/0022-5193(69)90015-0MathSciNetCrossRefGoogle Scholar
  12. 12.
    Karlsen, M.R., Moschoyiannis, S.: Evolution of control with learning classifier systems. Appl. Netw. Sci. 3(1), 30 (2018).  https://doi.org/10.1007/s41109-018-0088-xCrossRefGoogle Scholar
  13. 13.
    Karlsen, M.R., Moschoyiannis, S.K.: Optimal control rules for random boolean networks. In: International Workshop on Complex Networks and their Applications, pp. 828–840. Springer (2018)Google Scholar
  14. 14.
    Lin, L.: Self-improving reactive agents based on reinforcement learning, planning and teaching. Mach. Learn. 8, 293–321 (1992).  https://doi.org/10.1007/bf00992699CrossRefGoogle Scholar
  15. 15.
    Liu, Y., Slotine, J., Barabási, A.: Controllability of complex networks. Nature 473, 167–173 (2011).  https://doi.org/10.1038/nature10011CrossRefGoogle Scholar
  16. 16.
    Luque, B., Solé, R.: Controlling chaos in random Boolean networks. Europhys/ Lett. (EPL) 37(9), 597–602 (1997).  https://doi.org/10.1209/epl/i1997-00196-9CrossRefGoogle Scholar
  17. 17.
    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A., Veness, J., Bellemare, M., Graves, A., Riedmiller, M., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).  https://doi.org/10.1038/nature14236CrossRefGoogle Scholar
  18. 18.
    Moschoyiannis, S., Elia, N., Penn, A., Lloyd, D.J.B., Knight, C.: A web-based tool for identifying strategic intervention points in complex systems. In: Proceedings Games for the Synthesis of Complex Systems (CASSTING’16 @ ETAPS 2016), EPTCS, vol. 220, pp. 39–52 (2016)CrossRefGoogle Scholar
  19. 19.
    Paul, S., Su, C., Pang, J., Mizera, A.: A decomposition-based approach towards the control of Boolean networks. In: Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, pp. 11–20. ACM (2018)Google Scholar
  20. 20.
    Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay (2015). https://arxiv.org/abs/1511.05952. Accessed 14 Sep 2019
  21. 21.
    Somogyi, R., Sniegoski, C.: Modeling the complexity of genetic networks: understanding multigenic and pleiotropic regulation. Complexity 1(6), 45–63 (1996).  https://doi.org/10.1002/cplx.6130010612MathSciNetCrossRefGoogle Scholar
  22. 22.
    Sutton, R., Barto, A.: Reinforcement Learning. The MIT Press, Cambridge (2018)zbMATHGoogle Scholar
  23. 23.
    Szallasi, Z., Liang, S.: Modeling the normal and neoplastic cell cycle with realistic Boolean genetic networks: their application for understanding carcinogenesis and assessing therapeutic strategies. In: Pacific Symposium on Biocomputing, vol. 3, pp. 66–76 (1998)Google Scholar
  24. 24.
    Watkins, C., Dayan, P.: Mach. Learn. 8, 279–292 (1992).  https://doi.org/10.1023/a:1022676722315CrossRefGoogle Scholar
  25. 25.
    Watkins, C., Dayan, P.: Q-learning. Mach. Learn. 8, 279–292 (1992).  https://doi.org/10.1007/bf00992698CrossRefzbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Computer Science, Faculty of Engineering and Physical SciencesUniversity of SurreyGuildfordUK

Personalised recommendations