Sequential Extensions of Causal and Evidential Decision Theory

  • Tom EverittEmail author
  • Jan Leike
  • Marcus Hutter
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9346)


Moving beyond the dualistic view in AI where agent and environment are separated incurs new challenges for decision making, as calculation of expected utility is no longer straightforward. The non-dualistic decision theory literature is split between causal decision theory and evidential decision theory. We extend these decision algorithms to the sequential setting where the agent alternates between taking actions and observing their consequences. We find that evidential decision theory has two natural extensions while causal decision theory only has one.


Evidential decision theory Causal decision theory Planning Causal graphical models Dualism Physicalism 



This work was in part supported by ARC grant DP120100950. It started at a MIRIxCanberra workshop sponsored by the Machine Intelligence Research Institute. Mayank Daswani and Daniel Filan contributed in the early stages of this paper and we thank them for interesting discussions and helpful suggestions. We also thank Nate Soares for useful feedback.


  1. [Ahm14]
    Ahmed, A.: Evidence, Decision and Causality. Cambridge University Press, Cambridge (2014)CrossRefGoogle Scholar
  2. [Alt13]
    Altair, A.: A comparison of decision algorithms on Newcomblike problems. Technical report, Machine Intelligence Research Institute (2013).
  3. [Bos14]
    Bostrom, N.: Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Oxford (2014)Google Scholar
  4. [Bri14]
    Briggs, R.: Normative theories of rational choice: expected utility. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy. Fall 2014 edition (2014)Google Scholar
  5. [Ega07]
    Egan, A.: Some counterexamples to causal decision theory. Philos. Rev. 116(1), 93–114 (2007)CrossRefGoogle Scholar
  6. [ELH15]
    Everitt, T., Leike, J., Hutter, M.: Sequential extensions of causal and evidential decision theory. Technical report, Australian National University (2015).
  7. [GH78]
    Gibbard, A., Harper, W.L.: Counterfactuals and two kinds of expected utility. In: Harper, W.L., Stalnaker, R., Pearce, G. (eds.) Foundations and Applications of Decision Theory. The University of Western Ontario Series in Philosophy of Science, vol. 15, pp. 125–162. Springer, Heidelberg (1978)Google Scholar
  8. [Jef83]
    Jeffrey, R.C.: The Logic of Decision, 2nd edn. University of Chicago Press, Chicago (1983)Google Scholar
  9. [Joy99]
    Joyce, J.M.: The Foundations of Causal Decision Theory. Cambridge University Press, Cambridge (1999)CrossRefzbMATHGoogle Scholar
  10. [Lew81]
    Lewis, D.: Causal decision theory. Australas. J. Philos. 59(1), 5–30 (1981)CrossRefGoogle Scholar
  11. [LFY+14]
    LaVictoire, P., Fallenstein, B., Yudkowsky, E., Barasz, M., Christiano, P., Herreshoff, M.: Program equilibrium in the prisoner’s dilemma via Lö’s theorem. In: AAAI Workshop on Multiagent Interaction without Prior Coordination (2014)Google Scholar
  12. [LH14]
    Lattimore, T., Hutter, M.: General time consistent discounting. Theor. Comput. Sci. 519, 140–154 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  13. [Noz69]
    Nozick, R.: Newcomb’s problem and two principles of choice. In: Rescher, N. (ed.) Essays in Honor of Carl G. Hempel. SL, vol. 24, pp. 114–146. Springer, Heidelberg (1969)CrossRefGoogle Scholar
  14. [OR12]
    Orseau, L., Ring, M.: Space-time embedded intelligence. In: Bach, J., Goertzel, B., Iklé, M. (eds.) AGI 2012. LNCS, vol. 7716, pp. 209–218. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  15. [Pea09]
    Pearl, J.: Causality, 2nd edn. Cambridge University Press, New York (2009)CrossRefzbMATHGoogle Scholar
  16. [RDT+15]
    Russell, S., Dewey, D., Tegmark, M., Kramar, J., Mallah, R.: Research priorities for robust and beneficial artificial intelligence. Technical report, Future of Life Institute (2015).
  17. [RN10]
    Russell, S.J., Norvig, P.: Artificial Intelligence. A Modern Approach, 3rd edn. Prentice Hall, Saddle River (2010) Google Scholar
  18. [Sav72]
    Savage, L.J.: The Foundations of Statistics. Dover Publications, New York (1972) zbMATHGoogle Scholar
  19. [SF14a]
    Soares, N., Fallenstein, B.: Aligning superintelligence with human interests: A technical research agenda. Technical report, Machine Intelligence Research Institute (2014).
  20. [SF14b]
    Soares, N., Fallenstein, B.: Toward idealized decision theory. Technical report, Machine Intelligence Research Institute (2014).
  21. [SF15]
    Soares, N., Fallenstein, B.: Two attempts to formalize counterpossible reasoning in deterministic settings. In: Bieger, J., Goertzel, B., Potapov, A. (eds.) Artificial General Intelligence. LNCS, vol. 9205. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  22. [Sky82]
    Skyrms, B.: Causal decision theory. J. Philos. 79(11), 695–711 (1982)CrossRefGoogle Scholar
  23. [Wei12]
    Weirich, P.: Causal decision theory. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy. Winter 2012 edition (2012)Google Scholar
  24. [Yud08]
    Yudkowsky, E.: Artificial intelligence as a positive and negative factor in global risk. In: Bostrom, N., M ćirković, M. (eds.) Global Catastrophic Risks, pp. 308–345. Oxford University Press (2008)Google Scholar
  25. [Yud10]
    Yudkowsky, E.: Timeless decision theory. Technical report, Machine Intelligence Research Institute (2010).

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Australian National UniversityCanberraAustralia

Personalised recommendations