Bounded Rational Decision-Making with Adaptive Neural Network Priors

Intelligent agents are usually faced with the task of optimizing some utility function \(\mathbf {U}\) that is a priori unknown and can only be evaluated sample-wise. We do not restrict ourselves on the form of this function, thus in principle it could be a classification or regression loss, a reward function in a reinforcement learning environment or any other utility function. The framework of information-theoretic bounded rationality [16, 17] and related information-theoretic models [3, 14, 20, 21, 23] provide a formal framework to model agents that behave in a computationally restricted manner by modeling resource constraints through information-theoretic constraints. Such limitations also lead to the emergence of hierarchies and abstractions [5], which can be exploited to reduce computational and search effort. Recently, the main principles have been successfully applied to spiking and artificial neural networks, in particular feedforward-neural network learning problems, where the information-theoretic constraint was mainly employed as some kind of regularization [7, 11, 12, 18]. In this work we introduce bounded rational decision-making with adaptive generative neural network priors. We investigate the interaction between anytime sample-based decision-making processes and concurrent improvement of prior policies through learning, where the prior policies are parameterized as Variational Autoencoders [10]—a recently proposed generative neural network model.

The paper is structured as follows. In Sect. 2 we discuss the basic concepts of information-theoretic bounded rationality, sampled-based interpretations of bounded rationality in the context of Markov Chain Monte Carlo (MCMC), and the basic concepts of Variational Autoencoders. In Sect. 3 we present the proposed decision-making model by combining sample-based decision-making with concurrent learning of priors parameterized by Variational Autoencoders. In Sect. 4 we evaluate the model with toy examples. In Sect. 5 we discuss our results.

In this section we combine MCMC anytime decision-processes with adaptive autoencoder priors. In the case of a single world state, the combination is straightforward in that each decision selected by the MCMC process is fed as an observable input to an autoencoder. The updated autoencoder is then used as an improved prior to initialize the next MCMC decision. In case of multiple world states, there are two straightforward scenarios. In the first scenario there are as many priors as world states and each of them is updated independently. For each world state we obtain exactly the same solution as in the single world state case. In the second scenario there is only a single prior over actions for all world states. In this case the autoencoder is trained with the decisions by all MCMC chains such that the autoencoder should converge to the optimal rate distortion prior. A third, more interesting scenario occurs when we allow multiple priors, but less than world states—compare Fig. 1. This is especially plausible when dealing with continuous world states, but also in the case of large discrete spaces.

3.2 Model Architecture

Equation (10) describe abstractly how a two-step decision process with bounded rational decision-makers should be optimally partitioned. In this section we propose a sample-based model of a bounded rational decision process that approximately corresponds to Eq. (10) such that the performance of the decision process can be compared against its normative baseline. To translate Eq. (10) into a stochastic process we proceed in three steps. First, we implement the priors p(a|x) as Variational Autoencoders. Second, we formulate an MCMC chain that is initialized with a sample from the prior and generates a decision \(a\sim p(a|x,w)\). Third, we design an MCMC chain that functions as a selector between the different priors.

Autoencoder Priors. Each prior p(a|x) in Eq. (10) is represented by a VAE that learns to generate action samples that mimic the samples given by the MCMC chains—compare Fig. 2. The functions \(\mu _\theta (z)\), \(\mu _\eta (a)\) and \(\varSigma _\eta (a)\) are implemented as feed-forward neural networks with one hidden layer. The units in the hidden layer were all chosen with sigmoid activation function, the output units in the case of the \(\mu \)-functions were also chosen as sigmoids and for the \(\varSigma \)-function as ReLU. During training the weights \(\eta \) and \(\theta \) are adapted to optimize the expected log-likelihood of the action samples that are given by the decisions made by the MCMC chains for all world states that have been assigned to the prior p(a|x). Due to the Gaussian shape of the decoder distribution, optimizing the log-likelihood corresponds to minimizing quadratic loss of the reconstruction error. After training, the network can generate sample actions itself by feeding the decoder network with samples from \(\mathcal {N}(z|0,\mathbb {I})\).

MCMC Decision-Making. To implement the bounded rational decision-maker p(a|w, x) we obtain an action sample \(a\sim p(a|x)\) from the autoencoder prior to initialize an MCMC chain that optimizes the target utility \(\mathbf {U}(w,a)\) for the given world state. We run the MCMC chain for \(N_{\max }\) steps. In each step we generate a proposal from a Gaussian distribution with \(g(a'|a)=\mathcal {N}(a'\vert a,\sigma ^2)\) and accept with probability

$$\begin{aligned} A(a'|a) = \min \big \{1, \exp ({\gamma (\mathbf {U}(w, a') - \mathbf {U}(w,a))})\big \}. \end{aligned}$$

(11)

Over the course of \(N_{\text {max}}\) time steps, the precision \(\gamma \) is adjusted following an annealing schedule conditioned on the maximum number of steps \(N_{\text {max}}\). We use an inverse Boltzmann annealing schedule, i.e. \(\gamma ^{(k)} = \gamma ^{0} + \alpha \log (1 + k)\), where \(\alpha \) is a tuning parameter. The rationale behind this is that we assume the sampling process to be coarse grained in the beginning and is getting finer during the search.

Open image in new window

Prior Selection. To implement the bounded rational prior selection \(p(x\vert w)\) through an MCMC process, we first sample an x from the prior p(x) and start an MCMC chain that (approximately) optimizes \(\varDelta {{\mathrm{\text {F}_{\text {par}}}}}(w,x)\) for a given world state w sampled from \(\rho (w)\). The prior p(x) is represented by a multinomial and updated by the frequencies of the selected prior indices x. The number of steps in the prior selection MCMC chain was kept constant at a value of \(N_{\mathrm {max}}^{\text {sel}}\) and similarly the precision \(\gamma ^{\text {sel}}\) was annealed over the course of \(N_{\mathrm {max}}^{\text {sel}}\) time steps. The target \(\varDelta {{\mathrm{\text {F}_{\text {par}}}}}(w,x)\) comprises a trade-off between expected utility and information resources. However, it cannot be directly evaluated and would require the computation of \({{\mathrm{\text {D}_\text {KL}}}}(p(a|x,w)\Vert p(a|x))\). Here we use number of steps in the downstream MCMC process as a resource measure. As the number of downstream steps was constant, the model selector’s choice only depended on the average utility achieved by each decision-maker, which results in the acceptance rule

$$\begin{aligned} A(x'|x) = \min \left\{ 1, \exp ({\gamma ^{\text {sel}}({{\mathrm{\mathbb {E}}}}_{p(a|w,x)}[\mathbf {U}(w,a)] - {{\mathrm{\mathbb {E}}}}_{p(a|w,x')}[\mathbf {U}(w,a)]}))\right\} . \end{aligned}$$

As the priors are discrete choices the proposal distribution \(q(x_{\text {p}}\vert x_\text {p})\) samples globally with \(p(x) = \frac{1}{\vert X \vert }\) for all x.

Open image in new window

To demonstrate our approach we evaluate two scenarios. First, a simple agent, which is equipped with a single prior policy \(p_\eta (a)\), as introduced in Sect. 2. In case of a single agent there is no need for a prior selection stage. Second, we evaluated a multi-prior decision-making system and compared the results to the single prior agent. For the mutli-prior agent, we split a fixed number of MCMC steps between the prior selection and the action selection. The task we designed consists of six world states where each world state has a Gaussian utility function in the interval [0, 1] with a unique optimum. In both settings, we equipped the Variational Autoencoders with one hidden layer consisting of 16 units with ReLU activations. We implemented the experiments using Keras [2]. We show the results in Fig. 3.

Our results indicate that using MCMC evaluation steps as a surrogate for information processing costs can be interpreted as bounded rational decision-making. In Fig. 3 we show the efficiency of several agents with different processing constraints. To compare our results to the theoretical baseline, we discretized the action space into 100 equidistant slices and solved the problem using the algorithm proposed in [5] to implement Eq. (10). Furthermore our results indicate that the multi-prior system generally outperforms the single-prior system in terms of utility.

To illustrate the differences in efficiency between the single prior agent and the multi-prior agents, we plotted in Fig. 4 utility gained through the second MCMC optimization. For multi-prior agents this is caused by specialized priors which provide initializations to the MCMC chains that are close to the optimal action. In this particular case, \(\varDelta \mathbf {U}\) does not become zero because we allow only three priors to cover six world states, thus leading to abstraction, i.e. specializing on actions that fit well for the assigned world states. In single-prior agents, the prior is adapting to all world states, thus providing, on average, an initial action that is suboptimal for the requested world state.

Open image in new window

In this study we implemented bounded rational decision makers with adaptive priors. We achieved this with Variational Autoencoder priors. The bounded rational decision-making process was implemented by MCMC optimization to find the optimal posterior strategy, thus giving a computationally simple way of generating samples. As the number of steps in the optimization process was constrained, we could quantify the information processing capabilities of the resulting decision-makers using relative Shannon entropy. Our analysis may have interesting implications, as it provides a normative framework for this kind of combined optimization of adaptive priors and decision-making processes. Prior to our work there have been several attempts to apply the framework of information-theoretic bounded rationality to machine learning tasks [7, 11, 12, 18]. The novelty of our approach is that we design adaptive priors for both the single-step case and the multi-agent case and we demonstrate how to transform information-theoretic constraints into computational constraints in the form of MCMC steps.

Recently, the combination of Monte Carlo optimization and neural networks has gained increasing popularity. These approaches include both using MCMC processes to find optimal weights in ANNs [1, 4] and using ANNs as parametrized proposal distributions in MCMC processes [8, 13]. While our approach is more similar to the latter, the important difference is that in such adaptive MCMC approaches there is only a single MCMC chain with a single (adaptive) proposal to optimize a single task, whereas in our case there are multiple adaptive priors to initialize multiple chains with otherwise fixed proposal, which can be used to learn multiple tasks simultaneously. In that sense our work is more related to mixture-of-experts methods and divide-and-conquer paradigms [6, 9, 24], where we employ a selection policy rather than a blending policy, as we design our model specifically to encourage specialization. In mixture-of-experts models, there are multiple decision-makers that correspond to multiple priors in our case, but experts are typically not modeled as anytime optimization processes. The possibly most popular combination of neural network learning with Monte Carlo methods was achieved by AlphaGo [19], which beat the leading Go champion by optimizing the strategies provided by value networks and policy networks with Monte Carlo Tree Search, leading to a major breakthrough in reinforcement learning. An important difference here is that the neural network is used to directly approximate the posterior and MCMC is used to improve performance by concentrating on the most promising moves during learning, whereas in our case ANNs are used to represent the prior. Moreover, in our work we assumed the utility function (i.e. the value network) to be given. For future work it would be interesting to investigate how to incorporate learning the utility function into our model to investigate more complex scenarios such as in reinforcement learning.