Subjective Probability as Sampling Propensity

Abstract

Subjective probability plays an increasingly important role in many fields concerned with human cognition and behavior. Yet there have been significant criticisms of the idea that probabilities could actually be represented in the mind. This paper presents and elaborates a view of subjective probability as a kind of sampling propensity associated with internally represented generative models. The resulting view answers to some of the most well known criticisms of subjective probability, and is also supported by empirical work in neuroscience and behavioral psychology. The repercussions of the view for how we conceive of many ordinary instances of subjective probability, and how it relates to more traditional conceptions of subjective probability, are discussed in some detail.

Appendices

Appendix A: Boltzmann Machine as Gibbs Sampler

In this Appendix we explain the sense in which the activation rule for the Boltzmann Machine carries out Gibbs sampling on an underlying Markov random field. This fact is folklore in cognitive science.

Gibbs sampling is an instance of the Metropolis-Hastings algorithm, which in turn is a typle of Markov chain Monte Carlo inference (MacKay 2003). Suppose we have a multivariate distribution P(X ₁,…,X _n ) and we want to draw samples from it. The Gibbs sampling algorithm is as follows:

• 1. Specify some initial values for all the random variables $\mathbf {y}^{(0)} = \left (y_{1}^{(0)},\dots ,y_{n}^{(0)}\right )$.

• 2. Given y ^(r), randomly choose a number i∈{0,…,n}, and let y ^(r+1) be exactly like y ^(r), except that $y_{i}^{(r+1)}$ is redrawn from the conditional distribution P(X _i | y _−i ).

• 3. At some stage R, return some subset of {y ⁽⁰⁾,…,y ^(R)} as samples.

Note that the sequence of value vectors y ⁽⁰⁾,…,y ^(r),… forms a Markov chain because the next sample y ^(r+1) only depends on the previous sample y ^(r). Let q(y→y ^′) be the probability of moving from y at a given stage r to y ^′ at stage r+1 (which is the same for all r, and equal to 0 when y and y ^′ differ by more than one coordinate.). And let π ^r(y) be the probability of being in state y at stage r. Then we clearly have:

$$\pi^{r+1}(\mathbf{y}^{\prime}) = \sum\limits_{\mathbf{y}} \pi^{r}(\mathbf{y}) q(\mathbf{y}\rightarrow\mathbf{y}^{\prime})\;.$$

By general facts about Markov chains, it can be shown that this process reaches a unique stationary distribution, i.e., π ^r = π ^r+1 for all r>s for some s. This is the unique distribution π ^∗ for which the following holds:

$$\pi^{*}(\mathbf{y}^{\prime}) = \sum\limits_{\mathbf{y}} \pi^{*}(\mathbf{y}) q(\mathbf{y}\rightarrow\mathbf{y}^{\prime})\;.$$

Since this equation also holds for P, that shows the Markov chain converges to P = π ^∗.

Recall the Boltzmann Machine is defined by a set of nodes and weights between those nodes.³⁸ If we think of the nodes as binary random variables (X ₁,…,X _n ), taking on values {0,1}, then the weight matrix W gives us a natural distribution P(X ₁,…,X _n ) on state space {0,1}ⁿ as follows. First, define an energy function E on the state space:

$$E(\mathbf{y}) = -\frac{1}{2} \sum\limits_{i,j} W_{i,j} y_{i}y_{j}\;.$$

Then the resulting Boltzmann distribution is given by:

$$P(\mathbf{y}) = \frac{e^{-E(\mathbf{y})}}{\sum\limits_{\mathbf{y}^{\prime}}e^{-E(\mathbf{y}^{\prime})}}\;.$$

Note that the denominator becomes a very large sum very quickly. Recall from our earlier discussion that the size of the state space is 32 with 5 nodes, but over a trillion with 40 nodes. Suppose we apply the Gibbs sampling algorithm to this distribution. Then at a given stage, we randomly choose a node to update, and the new value is determined by the conditional probability as above. This step is equivalent to applying the activation function for the Boltzmann Machine (where in the following, y ^′ is just like y, except that $y_{i}^{\prime } = 1$ and y _i = 0):

$$\begin{array}{@{}rcl@{}} P(X_{i}=1\; |\; \{y_{j}\}_{j\neq i}) & = & P(X_{i} = 1\;|\;\mathbf{y}\text{ or }\mathbf{y}^{\prime}) \\ & = & \frac{P(\mathbf{y}^{\prime})}{P(\mathbf{y}\text{ or }\mathbf{y}^{\prime})} \\ & = & \frac{e^{-E(\mathbf{y}^{\prime})}}{e^{-E(\mathbf{y})}+e^{-E(\mathbf{y}^{\prime})}} \\ & = & \frac{1}{1+e^{E(\mathbf{y}^{\prime}) -E(\mathbf{y})}} \\ & = & \frac{1}{1+ e^{-\sum\limits_{j} W_{i,j}y_{j}}}\\ & = & \frac{1}{1+ e^{-net_{i}}}\;. \end{array} $$

Thus, the above argument shows that the Boltzmann Machine (eventually) samples from the associated Boltzmann distribution. Incidentally, we can also see now why n e t _i is equivalent to the log odds ratio under the Boltzmann distribution (recall the discussion in Section 8 of the Yang and Shadlen 2007 experiment):

$$\begin{array}{@{}rcl@{}} \frac{P(\mathbf{y}^{\prime})}{P(\mathbf{y})} & = & \frac{e^{-E(\mathbf{y}^{\prime})}}{e^{-E(\mathbf{y})}} \\ & = & e^{E(\mathbf{y}) - E(\mathbf{y}^{\prime})} \\ & = & e^{\sum\limits_{j} W_{i,j}y_{j}} \\ & = & e^{net_{i}}\;. \end{array} $$

And thus,

$$\begin{array}{@{}rcl@{}} \log\;\frac{P(\mathbf{y}^{\prime})}{P(\mathbf{y})} & = & \log \left( e^{net_{i}}\right) \\ & = & net_{i}\;.\end{array} $$

Appendix B: Softmax versus Sampling Rule

In this Appendix, we offer some observations about the relation between the softmax (or generalized Luce-Shepard) choice rule and the sampling-based decision rules discussed in this chapter. Suppose we associate with a subject a probability function P(⋅) on sample space $\mathcal {H} = \{H_{1},\dots ,H_{n}\}$ and a utility u over actions $\mathcal {A} = \{A_{1},\dots ,A_{m}\}$.

The softmax rule says that the subject will give response A with probability

$$\frac{e^{v(A)/\beta}}{\sum\limits_{A^{\prime}\in\mathcal{A}} e^{v(A^{\prime})/\beta}} , $$

where v is some value function, which in this case we will assume is the log expected utility:

$$v(H) = \log \sum\limits_{H\in\mathcal{H}}P(H)u(A,H) .$$

As a representative example of a sampling based rule, recall Decision Rule B:

Decision Rule B: Suppose we are given a generative model $\mathcal {M}$ with $\mathcal {V}$ as possible return values. To select an action, take R samples, H ⁽¹⁾,…,H ^(R), using $\mathcal {M}$, and let best be the set of actions that receive the largest summed utilities, i.e.,

$$\text{\textsc{best}} = \left\{A_{j}: \sum\limits_{i=1}^{R}u(A_{j},H^{(i)})\text{ is maximal}\right\}.$$

Take action A _j ∈best with probability $\frac {1}{\vert \text {\textsc {best}}\vert }$.

In the specific case of a certain kind of estimation problem—where $\mathcal {H} = \mathcal {A}$ and the utility of an estimate is 1 if correct, 0 otherwise; thus expected utility and probability coincide—it is easy to see that the softmax rule with β = 1 and Decision Rule B with R = 1 (or R = 2) are equivalent. The probability of returning hypothesis H is just P(H), i.e., we have probability matching.

Unfortunately, even restricting to these simple estimation problems, the relation between the two rules as functions of β and R is intractable and varies with the probability distribution P(⋅), as Vul (2010) points out. Thus, beyond β = R = 1 it is hard to study their relationship. As mentioned in the text, both can be used to fit much of the psychological data, though one might suspect the sampling rule is more reasonable on the basis of computational considerations.

Interestingly, for more general classes of decision problems, these two classes of rules can be qualitatively distinguished. The Luce Choice Axiom, from which the Luce choice rule was originally derived (Luce 1959), gives a hint of how we might do this. Where P _S (T) is the probability of choosing an action from $T\subseteq \mathcal {A}$ from among options in $S\subseteq \mathcal {A}$, the choice axiom states that for all R such that T⊆R⊆S:

$$P_{S}(T) \;=\; P_{S}(R) \;P_{R}(T)\;.$$

It is easy to see the softmax rule satisfies the choice axiom for all values of β:

$$\begin{array}{@{}rcl@{}} P^{\text{softmax}}_{S}(T) & = & \frac{\sum\limits_{A\in T}e^{v(A)/\beta}}{\sum\limits_{A\in S}e^{v(A)/\beta}} \\ & = & \frac{\sum\limits_{A\in R}e^{v(A)/\beta}}{\sum\limits_{A\in S}e^{v(A)/\beta}} \;\cdot\;\frac{\sum\limits_{A\in T}e^{v(A)/\beta}}{\sum\limits_{A\in R}e^{v(A)/\beta}} \\ & = & P^{\text{softmax}}_{S}(R) \;P^{\text{softmax}}_{R}(T)\;. \end{array} $$

For estimation problems, Decision Rule B with R = 1 also satisfies this axiom. However, in more general contexts, even for the case of R = 1, it does not. Perhaps the simplest illustration of this is the decision problem in Table 1, where $\mathcal {H} = \{H_{1},H_{2}\}$ and $\mathcal {A} = \{A_{1},A_{2},A_{3}\}$, and 𝜖>0.

Table 1 Distinguishing softmax and sample-based decision rules

We clearly have $P^{\text {\textsc {Rule B}}}_{\{A_{1},A_{2},A_{3}\}}(\{A_{1}\}) = 0$. Yet, as long as P(H ₁),P(H ₂)>0 and 𝜖<1, we have

$$P^{\text{\textsc{Rule B}}}_{\{A_{1},A_{2},A_{3}\}}(\{A_{1},A_{2}\})\;P^{\text{\textsc{Rule B}}}_{\{A_{1},A_{2}\}}(\{A_{1}\})>0\;,$$

showing the violation of the choice axiom. As the softmax rule satisfies the axiom, this gives us an instance where we would expect different behavior, depending on which rule (if either) a subject is using. When presented with such a problem, a softmax-based agent would sometimes choose action A ₁. In fact, the probability of choosing A ₁ can be nearly as high as 1/3, for example, when β = 1, 𝜖 is very small, and H ₁ and H ₂ are equiprobable. However, for any set of samples of any length R, one of A ₂ or A ₃ would always look preferable for a sampling agent. Thus, such an agent would never choose A ₁. It would be interesting to test this difference experimentally. Doing so could offer important evidence about the tenability of the Sampling Hypothesis.