Optimal Measurement of Visual Motion Across Spatial and Temporal Scales

Abstract

Sensory systems use limited resources to mediate the perception of a great variety of objects and events. Here a normative framework is presented for exploring how the problem of efficient allocation of resources can be solved in visual perception. Starting with a basic property of every measurement, captured by Gabor’s uncertainty relation about the location and frequency content of signals, prescriptions are developed for optimal allocation of sensors for reliable perception of visual motion. This study reveals that a large-scale characteristic of human vision (the spatiotemporal contrast sensitivity function) is similar to the optimal prescription, and it suggests that some previously puzzling phenomena of visual sensitivity, adaptation, and perceptual organization have simple principled explanations.

Appendices

Appendices

7.1.1 Appendix 1. Additivity of Uncertainty

For the sake of simplicity, the following derivations concern the stimuli that can be modeled by integrable functions $I:{\mathbb{R}} \to {\mathbb{R}}$ of one variable $x$. Generalizations to functions of more than one variable are straightforward. Consider two quantities:

• Stimulus location on $x$, where $x$ can be space or time, and the “location” indicates respectively “where” or “when” the stimulus has occurred.

• Stimulus content on $f_{x}$, where $f_{x}$ can be spatial or temporal frequency of stimulus modulation.

Suppose a sensory system is equipped with many measuring devices (“sensors”), each used to estimate both stimulus location and frequency content from “image” (or “input”) $I(x)$. Assume that the outcome of measurement is a random variable with probability density function $p(x,f)$. Let

$$ \begin{array}{*{20}l} {p_{x} (x)} \hfill & { = \int p(x,f)df,} \hfill \\ {p_{f} (f)} \hfill & { = \int p(x,f)dx} \hfill \\ \end{array} $$

(7.12)

be the (marginal) means of $p(x,f)$ on dimensions $x$ and $f_{x}$ (abbreviated as $f$).

It is sometimes assumed that sensory systems “know” $p(x,f)$, which is not true in general. Generally, one can only know (or guess) some properties of $p(x,f)$, such as its mean and variance. Reducing the chance of gross error due to the incomplete information about $p(x,f)$ is accomplished by a conservative strategy: finding the minima on the function of maximal uncertainty, i.e., using a minimax approach [15, 16].

The minimax approach is implemented in two steps. The first step is to find such $p_{x} (x)$ and $p_{f} (f)$, for which measurement uncertainty is maximal. (The uncertainty is characterized conservatively, in terms of variance alone [2]). The second step is to find the condition(s), at which the function of maximal uncertainty has the smallest value: the minimax point(s).

Maximal uncertainty is evaluated using the well-established definition of entropy [58] (cf. [59, 60]):

$$ H(X,F) = - \int p(x,f){\kern 1pt} \,\log \,p(x,f)\,dx{\kern 1pt} df. $$

According to the independence bound on entropy (Theorem 2.6.6 in [61]),

$$ H(X,F) \le H(X) + H(F) = H^{ * } (X,F), $$

(7.13)

where

$$ \begin{array}{*{20}l} {H(X) = } \hfill & { - \int p_{x} (x)\,\log p_{x} (x)\,dx,} \hfill \\ {H(F) = } \hfill & { - \int p_{f} (f)\,\log p_{f} (f)\,df.} \hfill \\ \end{array} $$

Therefore, the uncertainty of measurement cannot exceed

$$ \begin{array}{*{20}l} {H^{ * } (X,F) = } \hfill & { - \int p_{x} (x)\,\log p_{x} (x)\,dx} \hfill \\ {} \hfill & { - \int p_{f} (f)\,\log p_{f} (f)\,df.} \hfill \\ \end{array} $$

(7.14)

Eq. 7.14 is the “envelope” of maximal measurement uncertainty: a “worst-case” estimate.

By the Boltzmann theorem on maximum-entropy probability distributions [61], the maximal entropy of probability densities with fixed means and variances is attained, when the functions are Gaussian. Then, the maximal entropy is a sum of their variances [61] and

$$ \begin{array}{*{20}l} {p_{x} (x)} \hfill & { = \frac{1}{{\sigma_{x} \sqrt {2\pi } }}e^{{ - x^{2} /2\sigma_{x}^{2} }} ,} \hfill \\ {p_{f} (f)} \hfill & { = \frac{1}{{\sigma_{f} \sqrt {2\pi } }}e^{{ - f^{2} /2\sigma_{f}^{2} }} ,} \hfill \\ \end{array} $$

where $\sigma_{x}$ and $\sigma_{f}$ are the standard deviations. Then maximal entropy is

$$ H = \sigma_{x}^{2} + \sigma_{f}^{2} . $$

(7.15)

That is, when $p(x,f)$ is unknown, and all one knows about marginal distributions $p_{x} (x)$ and $p_{f} (f)$ is their means and variances, the maximal uncertainty of measurement is the sum of variances of the estimates of $x$ and $f$. The following minimax step is to find the conditions of measurement, at which the sum of variances is the smallest.

7.1.2 Appendix 2. Improving Resolution by Multiple Sampling

How does an increased allocation of resources to a specific condition of measurement affect the (spatial or temporal) resolution at that condition? Consider set $\varPsi$ of sampling functions

$$ \psi (s\sigma + \delta ),\;\sigma \in {\mathbb{R}},\;\sigma > 0,\;\delta \in {\mathbb{R}}, $$

where $\sigma$ is a scaling parameter and $\delta$ is a translation parameter. For a broad class of functions $\psi ( \cdot )$, any element of $\varPsi$ can be obtained by addition of weighted and shifted $\psi (s)$. The following argument proves that any function from a sufficiently broad class that includes $\psi (s\sigma + \delta )$ can be represented by a weighted sum of translated replicas of $\psi (s)$.

Let $\psi^{ * } (s)$ be a continuous function that can be expressed as a sum of a converging series of harmonic functions:

$$ \psi^{ * } (s) = \sum\limits_{i} a_{i} \,\cos (\omega_{i} s) + b_{i} \,\sin (\omega_{i} s). $$

For example, Gaussian sampling functions of arbitrary widths can be expressed as a sum of $\cos ( \cdot )$ and $\sin ( \cdot )$. Let us show that, if $|\psi (s)|$ is Riemann-integrable, i.e., if

$$ - \infty < \int\limits_{ - \infty }^{\infty } {\psi (s)|ds < \infty } $$

and its Fourier transform $\widehat{\psi }$ does not vanish for all $\omega \in {\mathbb{R}}$: $\widehat{\psi }(\omega ) \ne 0$ (i.e., its spectrum has no “holes”), then the following expansion of $\psi^{ * }$ is possible:

$$ \psi^{ * } (s) = \sum\limits_{i} c_{i} \psi (s + d_{i} ) + \varepsilon (s), $$

(7.16)

where $\varepsilon (s)$ is a residual that can be arbitrarily small. This goal is attained by proving identities

$$ \begin{array}{*{20}l} {\cos (\omega_{0} s)} \hfill & { = \sum\limits_{i} c_{i,1} \psi (s + d_{i,1} ) + \varepsilon_{1} (s),} \hfill \\ {\sin (\omega_{0} s)} \hfill & { = \sum\limits_{i} c_{i,2} \psi (s + d_{i,2} ) + \varepsilon_{2} (s),} \hfill \\ \end{array} $$

(7.17)

where $c_{i,1}$, $c_{i,2}$ and $d_{i,1}$, $d_{i,2}$ are real numbers, while $\varepsilon_{1} (s)$ and $\varepsilon_{2} (s)$ are arbitrarily small residuals.

First, write the Fourier transform of $\psi (s)$ as

$$ \widehat{\psi }(\omega ) = \int\limits_{ - \infty }^{\infty } {\psi (s)e^{ - i\omega s} ds} $$

and multiply both sides of the above expression by $e^{{i\omega_{0} \upsilon }}$:

$$ e^{{i\omega_{0} \upsilon }} \widehat{\psi }(\omega ) = e^{{i\omega_{0} \upsilon }} \int\limits_{ - \infty }^{\infty } {\psi (s)e^{ - i\omega s} ds = } \int\limits_{ - \infty }^{\infty } {\psi (s)e^{{ - i(\omega s - \omega_{0} \upsilon )}} ds.} $$

(7.18)

Change the integration variable:

$$ x = \omega s - \omega_{0} \upsilon \Rightarrow dx = \omega ds,\;s = \frac{{x + \omega_{0} \upsilon }}{\omega }, $$

such that Eq. 7.18 transforms into

$$ e^{{i\omega_{0} \upsilon }} \widehat{\psi }(\omega ) = \frac{1}{\omega }\int\limits_{ - \infty }^{\infty } {\psi \left( {\frac{{x + \omega_{0} \upsilon }}{\omega }} \right)e^{ - ix} dx.} $$

Notice that $\widehat{\psi }(\omega ) = a(\omega ) + ib(\omega )$. Hence

$$ e^{{i\omega_{0} \upsilon }} \widehat{\psi }(\omega ) = e^{{i\omega_{0} \upsilon }} (a(\omega ) + ib(\omega )) = (\cos (\omega_{0} \upsilon ) + i\,{ \sin }(\omega_{0} \upsilon ))(a(\omega ) + ib(\omega )) $$

and

$$ \begin{array}{*{20}l} {e^{{i\omega_{0} \upsilon }} \widehat{\psi }(\omega )} \hfill & { = (\cos (\omega_{0} \upsilon )a(\omega ) - \sin (\omega_{0} \upsilon )b(\omega )) + i(\cos (\omega_{0} \upsilon )b(\omega ) + \sin (\omega_{0} \upsilon )a(\omega )).} \hfill \\ \end{array} $$

Since $\widehat{\psi }(\omega ) \ne 0$ is assumed for all $\omega \in {\mathbb{R}}$, then $a(\omega ) + ib(\omega ) \ne 0$. In other words, either $a(\omega ) \ne 0$ or $b(\omega ) \ne 0$ should hold. For example, suppose that $a(\omega ) \ne 0.$ Then

$$ Re\left( {e^{{i\omega_{0} \upsilon }} \widehat{\psi }(\omega )} \right) + \frac{b(\omega )}{a(\omega )}Im\left( {e^{{i\omega_{0} \upsilon }} \widehat{\psi }(\omega )} \right) = \cos (\omega_{0} \upsilon )\left( {\frac{{a^{2} (\omega ) + b^{2} (\omega )}}{a(\omega )}} \right). $$

Therefore,

$$ \begin{array}{*{20}l} {\cos (\omega_{0} \upsilon )} \hfill & { = \left( {\frac{a(\omega )}{{a^{2} (\omega ) + b^{2} (\omega )}}} \right)Re\left( {\frac{1}{\omega }\int\limits_{ - \infty }^{\infty } {\psi \left( {\frac{{x + \omega_{0} \upsilon }}{\omega }} \right)e^{ - ix} dx} } \right)} \hfill \\ {} \hfill & { + \left( {\frac{b(\omega )}{{a^{2} (\omega ) + b^{2} (\omega )}}} \right)Im\left( {\frac{1}{\omega }\int\limits_{ - \infty }^{\infty } {\psi \left( {\frac{{x + \omega_{0} \upsilon }}{\omega }} \right)e^{ - ix} dx} } \right).} \hfill \\ \end{array} $$

(7.19)

Because function $\psi (s)$ is Riemann-integrable, the integrals in Eq. 7.19 can be approximated as

$$ Re\left( {\frac{1}{\omega }\int\limits_{ - \infty }^{\infty } {\psi \left( {\frac{{x + \omega_{0} \upsilon }}{\omega }} \right)e^{ - ix} dx} } \right) = \frac{\varDelta }{\omega }\sum\limits_{k = 1}^{N} \psi \left( {\frac{{x_{k} + \omega_{0} \upsilon }}{\omega }} \right)\cos (x_{k} ) + \frac{{\bar{\varepsilon }_{1} (\upsilon ,N)}}{\omega }, $$

(7.20)

$$ Im\left( {\frac{1}{\omega }\int\limits_{ - \infty }^{\infty } {\psi \left( {\frac{{x + \omega_{0} \upsilon }}{\omega }} \right)e^{ - ix} dx} } \right) = \frac{\varDelta }{\omega }\sum\limits_{p = 1}^{N} \psi \left( {\frac{{x_{p} + \omega_{0} \upsilon }}{\omega }} \right)\sin (x_{p} ) + \frac{{\bar{\varepsilon }_{2} (\upsilon ,N)}}{\omega }, $$

(7.21)

where $\;x_{k}$ and $x_{p}$ are some elements of ${\mathbb{R}}$.

From Eqs. 7.19–7.21 it follows that

$$ \cos (\omega_{0} \upsilon ) = \sum\limits_{j = 1}^{2N} c_{j,1} \psi \left( {\frac{{\omega_{0} \upsilon }}{\omega } + d_{j,1} } \right) + \varepsilon_{1} (\upsilon ,N). $$

Given that $\widehat{\psi }(\omega ) \ne 0$ for all $\omega$ and letting $\omega = \omega_{0}$, it follows that

$$ \cos (\omega_{0} \upsilon ) = \sum\limits_{j = 1}^{2N} c_{j,1} \psi \left( {\upsilon + d_{j,1} } \right) + \varepsilon_{1} (\upsilon ,N), $$

(7.22)

where

$$ \frac{\bar{\varepsilon }_{1} (\upsilon ,N)}{\omega_{0}}\frac{{a(\omega_{0} )}}{{a^{2} (\omega_{0} ) + b^{2} (\omega_{0} )}} + \frac{\bar{\varepsilon }_{2} (\upsilon ,N)}{\omega_{0}}\frac{{ b(\omega_{0} )}}{{a^{2} (\omega_{0} ) + b^{2} (\omega_{0} )}} $$

(7.23)

An analogue of Eq. 7.22 for $\sin (\omega_{0} \upsilon )$ follows from $\sin (\omega_{0} \upsilon ) = \cos (\omega_{0} \upsilon + \pi /2)$. This completes the proof of Eq. 7.17 and hence of Eq. 7.16.