On the distribution of posterior probability in Bayesian inference with a large number of observations

Abstract

Probabilistic generative models work in many applications of image analysis and speech recognition. In general, there is an observation vector $\vec y$ and a state vector $\vec x$, and a joint dependency structure among them. The object of interest is, given $\vec y$, the most likely configuration $\vec x_{MAP}$ and its posterior distribution. In practice, the exact value of the posterior probability of $\vec x_{MAP}$ is hard to obtain, especially when there is a large number of observed variables. Here we analyze the distribution of posterior probabilities of $\vec x_{MAP}$ when there are N = 200–1000 observations. We used a probabilistic model with simple linear dependency structure in which the exact value of the posterior probability of $\vec x_{MAP}$ is obtainable. Computer experiments show that even identical models generate a variety of posterior distributions, which suggest difficulties in understanding the meaning of posterior probability. Finally, by computing $P(\vec x * |\vec y)$’s where $\vec x$*’s are neighbors of $\vec x_{MAP}$, we propose a method of knowing whether the $\vec x_{MAP}$ is reliable even when the posterior probability $P(\vec x_{MAP} |\vec y)$ is small.