Randomness? What Randomness?

Abstract

This is a review of the issue of randomness in quantum mechanics, with special emphasis on its ambiguity; for example, randomness has different antipodal relationships to determinism, computability, and compressibility. Following a (Wittgensteinian) philosophical discussion of randomness in general, I argue that deterministic interpretations of quantum mechanics (like Bohmian mechanics or ’t Hooft’s Cellular Automaton interpretation) are strictly speaking incompatible with the Born rule. I also stress the role of outliers, i.e. measurement outcomes that are not 1-random. Although these occur with low (or even zero) probability, their very existence implies that the no-signaling principle used in proofs of randomness of outcomes of quantum-mechanical measurements (and of the safety of quantum cryptography) should be reinterpreted statistically, like the second law of thermodynamics. In three appendices I discuss the Born rule and its status in both single and repeated experiments, review the notion of 1-randomness (or algorithmic randomness) that in various guises was investigated by Kolmogorov and others and treat Bell’s (Physics 1:195–200, 1964) Theorem and the Free Will Theorem with their implications for randomness.

1 Introduction

Quantum mechanics commands much respect. But an inner voice tells me that it is not the real McCoy. The theory delivers a lot, but it hardly brings us closer to God’s secret. Anyway, I’m sure he does not play dice. (Einstein to Born, 1926).¹

In our scientific expectations we have grown antipodes. You believe in God playing dice and I in perfect laws in the world of things existing as real objects, which I try to grasp in a wildly speculative way. (Einstein to Born 1944).²

Einstein’s idea of ‘perfect laws’ that should in particular be deterministic is also central to ’t Hooft’s view of physics, as exemplified by his intriguing Cellular Automaton Interpretation of Quantum Mechanics (’t Hooft [64]). One aim of this paper is to provide arguments against this view,³ but even if these turn out to be unsuccessful I hope to contribute to the debate about the issue of determinism versus randomness by providing a broad view of the latter.⁴ My point in Sect. 2 is that randomness is a Wittgensteinian family resemblance (Sluga [106]; Baker and Hacker [6], Ch. XI), but a special one that is always defined through its antipode, which may change according to the specific use of the term.

The antipode defining which particular notion of randomness is meant may vary even within quantum mechanics, and here two main candidates arise (Sect. 3): one is determinism, as emphatically meant by Born [16] and most others who claim that randomness is somehow ‘fundamental’ in quantum theory, but the other is compressibility or any of the other equivalent notions defining what is called 1-randomness in mathematics as its antipode (see Appendix B for an explanation of this). The interplay between these different notions of randomness is the topic of Sects. 4–5. In Sect. 5 I argue that one cannot eat one’s cake and have it, in the sense of having a deterministic hidden variable theory underneath quantum mechanics that is strictly compatible with the Born rule. I also propose, more wildly, that Einstein’s prohibition of superluminal signaling should be demoted from an absolute to a statistical law, much as the second law of thermodynamics. My analysis relies on some mathematical background presented in (independent) Appendices A–C on the Born rule, Algorithmic (or 1-) randomness, and the Bell and Free Will Theorems.

2 Randomness as a Family Resemblance

The idea that in order to get clear about the meaning of a general term one had to find the common element in all its applications has shackled philosophical investigation; for it has not only led to no result, but also made the philosopher dismiss as irrelevant the concrete cases, which alone could have helped him to understand the usage of the general term. (Wittgenstein, Blue Book, Sects. 19–20).

I’m saying that these phenomena have no one thing in common in virtue of which we use the same word for all – but there are many kinds of affinity between them. (...) we see a complicated network of similarities overlapping and criss-crossing: similarities in the large and in the small. I can think of no better expression to characterize these similarities than “family resemblances”; for the various resemblances between members of a family – build, features, colour of eyes, gait, temperament, and so on and so forth – overlap and criss-cross in the same way. (Wittgenstein, Philosophical Investigations, Sects. 65–67).

Though he did not mention it himself, randomness seems a prime example of a phenomenon Wittgenstein would call a ‘family resemblance’.⁵ Independently, as noted by historians Lüthy and Palmerino [83] on the basis of examples from antiquity and medieval thought,⁶ the various different meanings of randomness (or chance) can all be identified by their antipode. Combining these ideas, I submit that randomness is not just any old Wittgensteinian family resemblance, but a special one that is always defined negatively:

• To begin with, in Aristotle’s famous example of a man who goes to the market and walks into his debtor, the randomness of the encounter derives from the fact that the man did not go the the market in order to meet his debtor (but instead went there to buy food). Similarly for the man who digs a hole in his garden to plant a tree and finds a treasure. Even the birth of female babies (and certain other ‘chance substances’ for which he literally uses the Greek word for ‘monsters’) was identified by The Philosopher as a failure of purpose in Nature. Thus what actually happened in all these examples was accidental because (as we would say it) it was not intended, or, in Aristotelian parlance, because there was no final cause. By the same token, Aristotle found the atomistic cosmos of Democritus “random” because it was purposeless, ridiculing him for making the cosmic order a product of chance.

• In contrast, half a century later Epicurus found the atomic world not random at all and introduced randomness through the swerve, immortalized by Lucretius:

  When the atoms are traveling straight down through empty space by their own weight, at quite indeterminate times and places they swerve ever so little from their course, just so much that you can call it a change of direction. If it were not for this swerve, everything would fall downwards like raindrops through the abyss of space. No collision would take place and no impact of atom on atom would be created. Thus nature would never have created anything. (Lucretius, De Rerum Natura, Book II).⁷

  This was, so to speak, the first complaint against determinism (the goal of Epicurus/Lucretius was to make room for free will in a world otherwise seen as effectively dead because of the everlasting sequence of cause and effect), and indeed, in the context of our analysis, the key point is that the swerve is random because it is indeterminate, or because the atoms depart from their natural straight course.

• Neither of these classical meanings is at all identical with the dominant usage from medieval times to the early 20th century, which was exemplified by Spinoza, who claimed that not only miracles, but also circumstances that have concurred by chance are reducible to ignorance of the true causes of phenomena, for which ultimately the will of God (‘the sanctuary of ignorance’) is invoked as a placeholder.⁸ Thus Spinozist randomness lies in the absence of full knowledge of the entire causal chain of events.

• In the Leibniz–Clarke correspondence,⁹ the latter, speaking for Newton, meant involuntariness by randomness. Against Leibniz, Clarke (and Newton) denied that at least God could be limited by involuntariness. Leibniz, on the other hand, in some sense ahead of his time (yet in another following Epicurus/Lucretius), used the word ‘random’ to designate the absence of a determining cause—a possibility which he (unlike Epicurus/Lucretius) in turn denied on the basis of his principle of sufficient reason.¹⁰ This is clear from an interesting passage which is not widely known and predates Laplace:

  One sees then that everything proceeds mathematically - that is, infallibly - in the whole wide world, so that if someone could have sufficient insight into the inner parts of things, and in addition could have remembrance and intelligence enough to consider all the circumstances and to take them into account, then he would be a prophet and would see the future in the present as in a mirror. (Leibniz).¹¹

• Arbuthnot, a younger contemporary and follower of Newton, may have been one of the first authors to explicitly address the role of randomness in the deterministic setting of Newtonian physics. In the Preface to his translation of Huygens’s path-breaking book De Ratiociniis in Ludo Aleae on probability theory, he wrote:

  It is impossible for a Die, with such determin’d force and direction, not to fall on such determin’d side, only I don’t know the force and direction which makes it fall on such determin’d side, and therefore I call it Chance, which is nothing but the want of Art. (Arbuthnot 1692).¹²

  And similarly, but highlighting the alleged negativity of the concept even more:

  Chance, in atheistical writings or discourse, is a sound utterly insignificant: It imports no determination to any mode of Existence; nor in deed to Existence itself, more than to non existence; it can neither be defined nor understood; nor can any Proposition concerning it be either affirmed or denied, excepting this one, “That it is a mere word.” (De Moivre 1718).¹³

  So this is entirely in the medieval spirit, where ignorance—this time relative to Newton’s physics as the ticket to full knowledge—is seen as the origin of randomness.

• A century later, and like Arbuthnot and De Moivre again in a book on probability theory (Essai philosophique sur les probabilités, from 1814), Laplace portrayed his demon to make the point that randomness arises in the absence of such an intellect:

  An intelligence which could comprehend all the forces that set nature in motion, and all positions of all items of which nature is composed—an intelligence sufficiently vast to submit these data to analysis—it would embrace in the same formula the movements of the greatest bodies in the universe and those of the lightest atom; for it, nothing would be uncertain and the future, as well as the past, would be present to its eyes. (Laplace 1814).¹⁴

  Note that Leibniz’ prophet appeals to the logical structure of the universe that makes it deterministic, whereas Laplace’s intelligence knows (Newtonian) physics.¹⁵ In any case, it is important to note that Laplacian randomness is defined within a deterministic world,¹⁶ so that its antipode is not indeterminism but full knowledge (and computing power, etc.). Indeed, less well known than the above quotation is the following one, from the very same source:

  All events, even those which on account of their insignificance do not seem to follow the great laws of nature, are a result of it just as necessarily as the revolutions of the sun. (Laplace 1814).¹⁷

• The ignorance interpretation of randomness and chance still lurks behind the probabilities introduced in the 19th century in statistical mechanics, which in my view were therefore wholeheartedly construed in the medieval and early modern sprit.¹⁸

• Hacking [54] explains how the doctrine of necessity began to erode in the 19th century, largely through the use of statistics in population studies and biology.¹⁹ In this respect, the century to some extent culminated in the following words of Peirce:

  I believe I have thus subjected to fair examination all the important reasons for adhering to the theory of universal necessity, and shown their nullity. (...) If my argument remains unrefuted, it will be time, I think, to doubt the absolute truth of the principle of universal law. (Peirce [96], p. 337).

  This partly paved the way for the claim of irreducible randomness in quantum mechanics, although the influence of population studies and biology on intrinsic developments in physics should perhaps not be overestimated. However, the insight that probability and statistics gave rise to their own laws (as opposed to the fear that randomness is pretty much the same as lawlessness), which partly dated back to the previous two centuries (Hacking [55]), surely made quantum theory possible.

• The randomness of variations in heritable traits that—almost simultaneously with the rise of statistical physics—were introduced in Darwin’s theory of evolution by natural selection meant something completely different from Laplace etc., best expressed by the renowned geneticist Theodosius Dobzhansky a century later (cf. Merlin [87]):

  Mutations are random changes because they occur independently of whether they are beneficial or harmful. (Dobzhansky et al. [36], p. 66).

  Indeed, both historically and actually, the antipode to Darwin’s randomness of variations is Lamarck’s goal-orientedness thereof,²⁰ intended to strengthen the species (like the proverbial sons of the blacksmith who according to Lamarck inherit his strong muscles). In particular, it does not matter if the variations are of known or unknown origin, or fall under a deterministic or an indeterministic kind of physics.

• Continuing our detour into biology, the well-known Hardy–Weinberg equilibrium law in population genetics, which gives the relative frequencies of alleles and genotypes in a given (infinite) population, is based on the assumption of random mating. This means that mating takes place between pairs of individuals who have not selected each other on the basis of their genetic traits (i.e. there is no sexual selection).

• Eagle ([41], p. 775–776) proposes that ‘randomness is maximal unpredictability’ (which agrees with criterion 3 at the end of this section), and argues that this is equivalent to a random event being ‘probabilistically independent of the current and past states of the system, given the probabilities supported by the theory.’

• Most people, especially those without scientific training,²¹ asked to mention a random event they have encountered, typically mention what is called a coincidence. This notion goes back at least to Aristotle, but a modern definition is the following:

  A coincidence is a surprising concurrence of events, perceived as meaningfully related, with no apparent causal connection. (Diaconis and Mosteller [34], p. 853).

  The subjective nature of this definition (i.e. ‘surprising’, ‘perceived’, ‘meaningful’, and ‘apparent’) may repel the scientist, but ironically, the aim of this definition is to debunk coincidences. Moreover, there is a striking similarity between analyzing coincidences in daily life and coincidences in the physical setting of bipartite correlation experiments à la EPR–Bohm–Bell: to this end, let me briefly recall how most if not all everyday coincidences can be nullified on the basis of the above definition:²²

  • 1. Against first appearances there was a causal connection, either through a common cause or through direct causation. This often works in daily life, and also in Bohmian mechanics, where direct (superluminal) causation is taken to be the “explanation” of the correlations in the experiments just referred to. However, if superluminal causation is banned, then one’s hand is empty because one interpretation of Bell’s Theorem (cf. Appendix C) excludes common causes (van Fraassen [121]), and hence both kinds of causation are out! See also Sect. 3.

  • 2. The concurrence of events was not at all as surprising as initially thought. This argument is either based on the inability of most people to estimate probabilities correctly (as in the well-known birthday problem), or, if the events were really unlikely, on what Diaconis and Mosteller call the law of truly large numbers:²³

    With a large enough sample, any outrageous thing is likely to happen. (Diaconis and Mosteller [34], p. 859).

  Neither of this helps in ERR–Bohm–Bell, though, leaving one’s hand truly empty.

• The choice sequences introduced by Brouwer in his intuitionistic construction of the real numbers, and then especially the “lawless” ones (which Brouwer never defined precisely) are often associated with notions of “freedom” and “randomness”:

  A choice sequence in Brouwer’s sense is a sequence of natural numbers (to keep it simple), which is not a priori given by a law or recipe for the elements, but which is created step by step (...); the process will go on indefinitely, but it is never finished. (...) Informally, we think of a lawless sequence of natural numbers as a process of choosing values in ${{\mathbb {N}}}$ (...) under the a priori restriction that at any stage of the construction never more that an initial segment has been determined and that no restrictions have been imposed on future choices, [and that] there is a commitment to determine more and more values (so the sequence is infinite). (...) A lawless sequence may be compared to the sequence of casts of a die. There too, at any given stage in the generation of a sequence never more than an initial segment is known. (Troelstra [113], italics added).²⁴

  Indeed, von Mises [128] mentioned choice sequences as an inspiration for his idea of a Kollektiv (van Lambalgen [123]), which in turn paved the way for the theory of algorithmic randomness to be discussed as the next and final example of this section. Nonetheless, a more precise analysis (Moschovakis [88, 89]) concludes as follows:

  Lawless and random are orthogonal concepts. A random sequence of natural numbers should possess certain definable regularity properties (e.g. the percentage of even numbers in its nth segment should approach 50 as n increases),²⁵ while a lawless sequence should possess none. Any regularity property definable in ${\mathcal {L}}$ by a restricted formula can be defeated by a suitable lawlike predictor. (Moschovakis [89], pp. 105–106, italics in original, footnote added).

  A further conceptual mismatch between lawless choice sequences and random sequences of the kind studied in probability theory is that any randomness of the former seems to lie on the side of what is called process randomness, whereas the latter is concerned with product randomness.²⁶ In a lawless choice sequence it is its creation process that is lawless, whereas no finished sequence (i.e. the outcome) can be totally lawless by Baudet’s Conjecture, proved by van der Waerden [120], which implies that every binary sequence x satisfies certain arithmetic laws.

• Finally, serving our aim to compare physical and mathematical notions of randomness, I preview the three equivalent definitions of 1-randomness (see Appendix B and references therein for details) and confirm that also they fit into our general picture of randomness being defined by some antipode. What will be remarkable is that the three apparently different notions of randomness to be discusses now, which at first sight are as good a family resemblance as any, actually turn out to coincide. The objects whose randomness is going to be discussed are binary strings, and our discussion here is so superficial that I will not even distinguish between finite and infinite ones; see Appendix B for the kind of precision that does enable one to do so.

  • 1. A string x is 1-random if its shortest description is x itself, i.e., there exists no lossless compression of x (in the sense of a computer program that outputs x and whose length is shorter than the length of x): thus x is incompressible.

  • 2. A string x is 1-random if it fails all tests for patterns (in a computable class).

  • 3. A string x is 1-random if there exists no successful (computable) gambling strategy on the digits of x; roughly speaking, these digits are unpredictable.

3 Randomness in Quantum Mechanics

Moving towards the main goal of the paper, I now continue our list of examples (i.e. of the principle that randomness is a family resemblance whose different meanings are always defined negatively through their antipodes) in the context of quantum mechanics, which is rich enough by itself to provide its own family of different meanings of randomness (all duly defined negatively), although these may eventually be traceable to the above cases.²⁷ Already the very first (scholarly) exposition of the issue of randomness in quantum mechanics by Born made many of the major points that are still relevant today:

Thus Schrödinger’s quantum mechanics gives a very definite answer to the question of the outcome of a collision; however, this does not involve any causal relationship. One obtains no answer to the question “what is the state after the collision,” but only to the question “how probable is a specific outcome of the collision” (in which the quantum-mechanical law of [conservation of] energy must of course be satisfied). This raises the entire problem of determinism. From the standpoint of our quantum mechanics, there is no quantity that could causally establish the outcome of a collision in each individual case; however, so far we are not aware of any experimental clue to the effect that there are internal properties of atoms that enforce some particular outcome. Should we hope to discover such properties that determine individual outcomes later (perhaps phases of the internal atomic motions)? Or should we believe that the agreement between theory and experiment concerning our inability to give conditions for a causal course of events is some pre-established harmony that is based on the non-existence of such conditions? I myself tend to relinquish determinism in the atomic world. But this is a philosophical question, for which physical arguments alone are not decisive.

(Born [16], p. 866).²⁸

Given the fact that Born was the first to discuss such things in the open literature, it is remarkable how perceptive his words are: he marks the opposition of randomness to determinism, recognizes the possibility of hidden variables (with negative advice though), and understands that the issue is not just a technical one. Bravo! Having said this, in line with the previous section our aim is, of course, to confront the antipode of determinism with other possible antipodes to randomness as it is featured by quantum mechanics.

The introduction of fundamental probabilities in quantum theory is delicate in many ways, among which is the fact that the Schrödinger equation is even more deterministic than Newton’s laws.²⁹ Hence what is meant is randomness of measurements outcomes; since it is not our aim (here) to solve the measurement problem–for which see Landsman [75], Chapter 11–I simply assume that (i) measurement is a well-defined laboratory practice, and (ii) measurements have outcomes. In all that follows, I also accept the statistical predictions of quantum mechanics for these outcomes (which are based on the Born rule reviewed in Appendix A). Even so, the claim of irreducibility of randomness, which is typical for all versions of the Copenhagen Interpretation (and for any mainstream view held by physicists) is almost incomprehensible, since one of the pillars of this interpretation is Bohr’s doctrine of classical concepts, according to which the apparatus must be described classically; randomness of measurement outcomes is then seen as a consequence of the very definition of a measurement. But this suggests that randomness should be reducible to ignorance about the quantum-mechanical degrees of freedom of the apparatus:

these uncertainties (...) are simply a consequence of the fact that we describe the experiment in terms of classical physics. (Heisenberg [58], p. 53).

Ironically, Bell’s Theorem(s), which arose in opposition to the the Copenhagen Interpretation (Cushing [32]), did not only prove Einstein (as the leading opponent of this interpretation) wrong on the issue that arguably mattered most to him (namely locality in the sense defined later in a precise way by Bell), but also proved Bohr and Heisenberg right on the irreducibility of randomness, at least if we grant them randomness per se. Indeed, suppose we equate reducibility of randomness with the existence of a “Laplacian” deterministic hidden variable theory (i.e. use the antipode of determinism), and assume, as the Copenhagenists would be pleased to, the conjunction of the following properties:

• 1. The Born rule and the ensuing statistical predictions of quantum mechanics;

• 2. (Hidden) Locality, i.e. the impossibility of active superluminal communication or causation if one knows the state $\lambda$ of the underlying deterministic theory.³⁰

• 3. Freedom (or free choice), that is, the independence of the choice of measurement settings from the state of the system one measures using these settings, in a broad sense of ‘state’ that includes the prepared state as well as the “hidden” state $\lambda$.³¹

Bell’s [10] Theorem then implies (robustly) that a deterministic hidden variable satisfying these assumptions theory cannot exist, as does the so-called Free Will Theorem (which relies on a non-probabilistic but non-robust special case of the Born rule implying perfect correlation, and also on the Kochen–Specker Theorem, which restricts its validity to quantum systems with three levels or more).³² Thus the Laplacian interpretation of randomness does not apply to quantum mechanics (granting assumptions 1–3), which warrants the Copenhagen claim of irreducible or non-Laplacian or Leibnizian randomness.

Viable deterministic hidden variable theories compatible with the Born rule therefore have to choose between giving up either Hidden Locality or Freedom (see also the conclusion of Appendix C). Given this choice, we may therefore distinguish between theories that:

• give up Hidden Locality, like Bohmian mechanics;

• give up Freedom, like the cellular automata interpretation of ’t Hooft [64].

In both cases the statistical predictions of quantum mechanics are recovered by averaging the hidden variable or state with respect to a probability measure $\mu _{\psi }$ on the space of hidden variables, given some (pure) quantum state $\psi$. The difference is that in Bohmian mechanics the total state (which consists of the hidden configuration plus the “pilot wave” $\psi$) determines the measurement outcomes given the settings, whereas in ’t Hooft’s theory the hidden state (see below) all by itself determines the outcomes as well as the settings.

• In Bohmian mechanics the hidden variable is position q, and $d\mu _{\psi }=|\psi (q)|^2 dx$ is the Born probability for outcome q with respect to the expansion $|\psi \rangle =\int dq\, \psi (q) |q\rangle$.³³

• In ’t Hooft’s theory the hidden state is identified with a basis vector $|n\rangle$ in some Hilbert space H ($n\in {{\mathbb {N}}}$), and once again the measure $\mu _{\psi }(n)=|c_{n}|^2$ is given by the Born probability for outcome n with respect to the expansion $|\psi \rangle =\sum _n c_n|n\rangle$.

In Bohmian mechanics (’t Hooft does not need this!) such averaging also restores Surface Locality, i.e., the impossibility of superluminal communication on the basis of actual measurement outcomes (which is a theorem of quantum theory, though a much more delicate one than is usually thought, as I will argue in Sect. 5), see also Valentini [117, 118]).

4 Probabilistic Preliminaries

O False and treacherous Probability,

Enemy of truth, and friend to wickednesse;

With whose bleare eyes Opinion learnes to see,

Truth’s feeble party here, and barrennesse.

When thou hast thus misled Humanity,

And lost obedience in the pride of wit,

With reason dar’st thou judge the Deity,

And in thy flesh make bold to fashion it.

Vaine thoght, the word of Power a riddle is,

And till the vayles be rent, the flesh newborne,

Reveales no wonders of that inward blisse,

Which but where faith is, every where findes scorne;

Who therfore censures God with fleshly sp’rit,

As well in time may wrap up infinite

Philip Sidney (1554–1586), Cœlica, Sonnet CIV.³⁴

My aim is to give a critical assessment of the situation described in the previous section. My analysis is based on the interplay between the single-case probability measure $\mu$ on an outcome space X, which for the purpose of this paper will be the Born measure $\mu =\mu _a$ on the spectrum $X=\sigma (a)$ of some self-adjoint operator a, and hence is provided by theory (see also Appendix A), and the probabilities defined as long-run frequencies for outcome sequences $x=(x_1, x_2, \ldots )$ of the Bernoulli process defined by $(X,\mu )$, which are given by experiment. To obtain clean mathematical results, I assume experiments can be repeated infinitely often. This is clearly an idealization, which is regulated by Earman’s Principle:

While idealizations are useful and, perhaps, even essential to progress in physics, a sound principle of interpretation would seem to be that no effect can be counted as a genuine physical effect if it disappears when the idealizations are removed. (Earman [44], p. 191).

As shown in Appendix B, finite-size effects amply confirm the picture below, comparable to the way that the law(s) of large numbers have finite-size approximants (such as the Chernoff-Hoeffding bound). In particular, zero/unit probability of infinite sequence comes down to very low/high probability for the corresponding finite strings.³⁵ Consequently, Earman’s Principle (in contrapositive form) holds if we use the canonical probability measure $\mu ^{\infty }$ on the infinite product space $X^{{{\mathbb {N}}}}$ of all infinite sequences $x=(x_1, x_2, \ldots )$, where $x_n\in X$, and $X^{{{\mathbb {N}}}}$ are canonically equipped with the cylindrical $\sigma$-algebra ${{\mathcal {S}}}\subset {{\mathcal {P}}}(X^{{{\mathbb {N}}}})$.³⁶

To see how one may recover or verify $\mu$ from the long-term frequencies governed by the product measure $\mu ^{\infty }$, for any function $f:X\rightarrow {{\mathbb {R}}}$ define $f^{(N)}:X^N\rightarrow {{\mathbb {R}}}$ by

$$\begin{aligned} f^{(N)}(x_1, \ldots , x_N)=\frac{1}{N}(f(x_1)+\cdots + f(x_N)). \end{aligned}$$

(4.1)

Then (by the ergodic theorem), for any $f\in C(X)$, almost surely with respect to $\mu ^{\infty }$,

$$\begin{aligned} \lim _{N\rightarrow \infty }f^{(N)}=\langle f\rangle _{\mu } \equiv \int _{X}d\mu (x)\, f(x) , \end{aligned}$$

(4.2)

times the unit function $1_{X^{{{\mathbb {N}}}}}$ on $X^{{{\mathbb {N}}}}$. This is for continuous functions f, but a limit argument extends it to characteristic functions $1_A:X\rightarrow \underline{2}\equiv \{0,1\}$, where $A\subset X$, so that

$$\begin{aligned} \lim _{N\rightarrow \infty } 1_A^{(N)}= \mu (A), \end{aligned}$$

(4.3)

times $1_{X^{{{\mathbb {N}}}}}$, again $\mu ^{\infty }$-almost surely. If we define the probability of A within some infinite sequence $(x_1, x_2, \ldots )$ as the relative frequency of A (i.e. the limit as $N\rightarrow \infty$ of the number of $x_n$ within $(x_1, \ldots , x_N)$ that lie in A divided by N), then (4.3) states that for almost all sequences in $X^{{{\mathbb {N}}}}$ with respect to the infinite product measure $\mu ^{\infty }$ this probability of A equals its Born probability. This is useful, since the latter is a purely mathematical quantity, whereas the former is experimentally accessible (at least for large N).

For simplicity (but without real loss of generality, cf. footnote 45), in what follows I specialize this setting to a (theoretically) fair coin flip, that is, $X=\underline{2}=\{0,1\}$ and $\mu (0)=\mu (1)=1/2$. Hence $\underline{2}^{{{\mathbb {N}}}}$ is the space of infinite binary sequences, equipped with the probability measure $\mu ^{\infty }$ induced by $\mu$ (as I shall argue in Sect. 5 below, despite tradition this situation cannot in fact arise classically, at least not in a deterministic theory). We have:

Theorem 4.1

Almost every binary sequence $x\in \underline{2}^{{{\mathbb {N}}}}$ is 1-random with respect to $\mu ^{\infty }$.

See e.g. Calude [25], Corollary 6.32. Thus the set E of all sequences that are not 1-random has probability zero, i.e. $\mu ^{\infty }(E)=0$, but this by no means implies that E is “small” in any other sense: set-theoretically, it is as large as its complement, i.e. the set of all 1-random sequences, which makes it bizarre that (barring a few exceptions related to Chaitin’s number $\Omega$) not a single one of these 1-random sequences can actually be proved to be 1-random, cf. Appendix B. Theorem 4.1 has further amazing consequences:

Corollary 4.2

With respect to $\mu ^{\infty }$, almost every infinite outcome sequence x of a fair coin flip is Borel normal,³⁷ incomputable,³⁸ and contains any finite string infinitely often.³⁹

This follows because any 1-random sequence has these properties with certainty, see Calude [25], Sect. 6.4. The relevance of Bernoulli processes for quantum theory comes from Theorem 5.1 in the next section, whose second option almost by definition yields these processes.

5 Critical Analysis and Claims

The relevance of the material in the previous section comes from the following result.

Theorem 5.1

The following procedures for repeated identical measurements are equivalent (in giving the same possible outcome sequences with the same probabilities):

• 1. Quantum mechanics is applied to the whole run (with classically recorded outcomes).

• 2. Quantum mechanics is just applied to single experiments (with classically recorded outcomes), upon which classical probability theory takes over to combine these.

See Appendix A for the proof, which culminates in Eq. (A.35), showing that the Born probability $\mu _a$ for single outcomes induces the Bernoulli process probability $\mu _a^{\infty }$ on the space $\sigma (a)^{{{\mathbb {N}}}}$ of infinite outcome sequences. As mentioned before, I specialize to fair quantum coin flips producing 50-50 Bernoulli processes, of which there are examples galore: think of measuring the third Pauli matrix $\sigma _z=\mathrm {diag}(1,-1)$ in a state like $\psi =(1,1)/\sqrt{2}$. In that case, Theorem 5.1 and Corollary 4.2 obviously have the following implication:

Corollary 5.2

With respect to the product measure $\mu _a^{\infty }$ coming from the Born measure $\mu _a$, almost every infinite outcome sequence x of a fair quantum coin flip is 1-random and therefore Borel normal, incomputable, and contains any finite string infinitely often.

The Born rule therefore implies very strong randomness properties of outcome sequences, albeit merely with probability one (i.e. almost surely) with respect to $\mu _a^{\infty }$. Moreover, Chaitin’s Incompleteness Theorem (see Appendix B) makes it impossible to prove that some given outcome sequence is 1-random even if it is!⁴⁰ Also in the spirit of the general ‘family resemblance’ philosophy of randomness, this puzzling situation makes it natural to compare randomness of infinite measurement outcome sequences as defined by:⁴¹

• 1. 1-randomness (with compressible sequences as its antipode), as suggested by the Born rule and the above analysis of its mathematical implications;⁴²

• 2. indeterminism, as suggested by Born himself, and in his wake also by Bell’s Theorem and the Free Will Theorem (seen as proofs of indeterminism under assumptions 1–3).

To make this comparison precise, we once again need the case distinction between hidden variable theories giving up Hidden Locality like Bohmian mechanics and those giving up Freedom, like ’t Hooft’s theory. See Appendix C for notation. In the usual EPR–Bohm–Bell setting (Bub [21]), let $(\alpha ,\beta )\in O\times O$ be Alice’s and Bob’s outcomes for given settings $(a,b)\in S\times S$, so that $(\alpha ,a)$ are Alice’s and $(\beta ,b)$ are Bob’s (outcomes, settings).

• In Bohmian mechanics, the hidden state $q\in Q$ just pertains to the correlated particles undergoing measurement, whilst the settings (a, b) are supposed to be “freely chosen” for each measurement (and in particular are independent of q).⁴³ The outcome is then given by $(\alpha ,\beta )=q(a,b)$. So if we number the consecutive runs of the experiment by $n\in {{\mathbb {N}}}=\{1,2, \ldots \}$, then everything is determined by functions

  $$\begin{aligned} f_1&: {{\mathbb {N}}}\rightarrow Q;&f_1(n)&=q_n; \end{aligned}$$

  (5.1)
  $$\begin{aligned} f_2&: {{\mathbb {N}}}\rightarrow S\times S;&f_2(n)&=(a_n,b_n), \end{aligned}$$

  (5.2)

  since these also give the outcome sequence by $(\alpha _n,\beta _n)=q_n(a_n,b_n)=f_1(n)(f_2(n)).$

• In ’t Hooft’s theory, the hidden state $x\in X$ of “the world” determines the settings as well as the outcomes, i.e. $(a,b,\alpha ,\beta )=(a(x),b(x),\alpha (x),\beta (x))$. In this case, the entire situation including the outcome sequence is therefore determined by a function

  $$ g: {{\mathbb {N}}}\rightarrow X;\qquad g(n)= x_n.$$

  (5.3)

A key point in the analysis of the functions $f_i$ and g is the requirement that both theories reproduce the statistical predictions of quantum mechanics given through the Born rule relative to some pure state $\psi$. As already noted, this is achieved by requiring that q is averaged with respect to some probability measure $\mu _{\psi }$ on Q, and likewise x is averaged with respect to some probability measure $\mu _{\psi }'$ on X. If the experimental run is to respect this averaging, then in Bohmian mechanics the map $f_1$ must be “typical” for the Born-like measure $\mu _{\psi }$ (cf. Dürr, Goldstein, and Zanghi [40]; Callender [23]; Norsen [93]; Valentini [119]);⁴⁴ see below for the special problems posed also by $f_2$. Analogously to the previous discussion of the Born measure itself and its sampling, any sampling of $\mu _{\psi }$ produces a sequence $(q_1, q_2, \ldots )$ that almost surely should have typical properties with respect to $\mu ^{\infty }_{\psi }$. In particular, such a sequence should typically be 1-random (in a suitable biased sense fixed by $\mu _{\psi }$).⁴⁵ Anything remotely deterministic, like computable samplings, will only contribute sequences that are atypical (i.e. collectively have measure zero) for $\mu _{\psi }$.⁴⁶

Likewise for the sampling of X with respect to $\mu _{\psi }'$ in ’t Hooft’s theory. Thus the requirement that the functions $f_1$ and g randomly sample $\mu _{\psi }$ and $\mu _{\psi }'$ introduces an element of unavoidable randomness into the hidden variable theories in question, which seems whackingly at odds with their deterministic character. Indeed, I only see two possibilities:

• This sampling is provided by the hidden variable theory. In that case, the above argument shows that the theory must contain an irreducibly random ingredient.

• The sampling is not provided by the theory. In that case, the theory fails to determine the outcome of any specific experiment and just provides averages of outcomes.

Either way, although at first sight our hidden variable theories are (Laplacian) deterministic (as is quantum mechanics, see footnote 29), in their accounting for measurement outcomes they are not (again, like quantum theory). What is the source of indeterminism?

• In standard (Copenhagen) quantum mechanics this source lies in the outcomes of experiments given the quantum state, whose associated Born measure is sampled;

• In “deterministic” hidden variable theories it is the assignment of the hidden variable to each measurement no. n, i.e. the sampling of the Born-like measures $\mu _{\psi }$ and $\mu _{\psi }'$.

So at best the source of indeterminism has been shifted. Moreover, in Bohmian mechanics and ’t Hooft’s theory $\mu _{\psi }$ and $\mu _{\psi }'$equal the Born measure, so one wonders what has been gained against Copenhagen quantum mechanics. Therefore, one has to conclude that:

Trulydeterministic hidden variable theories (i.e. those in which well-defined experiments have determined outcomes given the initial state and no appeal has to be made to irreducibly random samplings from this state) compatible with the Born rule do not exist.

In other words, as long as they reproduce all statistical predictions of quantum mechanics, deterministic theories underneath quantum mechanics still need a source of irreducible randomness in each and every experiment. In my view, this defeats their purpose.⁴⁷

In classical coin tossing the role of the hidden state is played by the initial conditions (cf. Diaconis and Skyrms [35], Chapter 1, Appendix 2). The 50-50 chances (allegedly) making the coin fair are obtained by averaging over the initial conditions, i.e., by sampling. By the above argument, this sampling cannot be deterministic, for otherwise the outcome sequences appropriate to a fair coin do not obtain: it must be done in a genuinely random way. This is impossible classically, so that fair classical coins do not exist, as confirmed by the experiments of Diaconis et al. reviewed in Diaconis and Skyrms [35], Chapter 1.

In response to this argument, both the Bohmians and ’t Hooft go for the second option and blame the randomness in question on the initial conditions, whose specification is indeed usually seen as lying outside the range of a deterministic theory.⁴⁸ As explained by both parties (Dürr et al. [40]; ’t Hooft [64]), the randomness in the outcomes of measurement on quantum system, including the Born rule, is a direct consequence of the above randomness in initial conditions. But in a Laplacian deterministic theory one can either predict or retrodict and these procedures should be equivalent; so within the context of a deterministic hidden variable theory of the kinds under discussion, Copenhagenists attributing the origin of randomness to the outcomes of measurement and our hidden variable theorists attributing it to the initial conditions for measurement, should be equivalent. Once again, this makes it impossible to regard the hidden variable theories in question as deterministic underpinnings of (Copenhagen) quantum mechanics.

Bohmians (but not ’t Hooft!) have another problem, namely the function (5.2) that provides the settings. Although $f_2$ is outside their theory, Bohmians should either account for both the “freedom” of choosing these settings and their randomness, or stop citing Bell’s Theorem (whose proof relies on averaging over random settings) in their favour.

Bell [11] tried to kill these two birds with the same stone by saying that the settings had to be ‘at least effectively free for the purpose at hand’, and clarifiying this as follows:

Suppose that the instruments are set at the whim, not of experimental physicists, but of mechanical random generators. (...) Could the input of such mechanical devices be reasonably be regarded as sufficiently free for the purpose at hand? I think so. Consider the extreme case of a “random” generator which is in fact perfectly deterministic in nature and, for simplicity, perfectly isolated. In such a device the complete final state perfectly determines the complete initial state–nothing is forgotten. And yet for many purposes, such a device is precisely a “forgetting machine”. (...) To illustrate the point, suppose that the choice between two possible [settings], corresponding to a and $a'$, depended on the oddness of evenness of the digit in the millionth decimal place of some input variable. Then fixing a or $a'$ indeed fixes something about the input–i.e., whether the millionth digit is odd or even. But this peculiar piece of information is unlikely to be the vital piece for any distinctly different purpose, i.e., it is otherwise rather useless. (...) In this sense the output of such a device is indeed a sufficiently free variable for the purpose at hand. (Bell [11], p. 105).

This seems to mix up the two issues. Though independence of the settings is defensible in a theory like Bohmian mechanics (Esfeld [47]), concerning their randomness Bell apparently ignored von Neumann’s warning against mechanical (i.e. pseudo) random generators:

Any one who considers arithmetical methods of producing random digits is, of course, in a state of sin (von Neumann [129]).

See also Markowsky [85]. Thus Bell’s statement was questionable already at the time of writing, but today we know for sure that mechanical random generators leave a loophole in the EPR–Bohm–Bell experiment: as soon as just one of the functions defining the settings (i.e. either of Alice or of Bob) is computable,⁴⁹ there is a model that is local (in the sense of Bell). See Bendersky et al. [13] or Senno [105], Theorem 2.2.1 This implies that Bohmian mechanics (as well as other deterministic hidden variable theories that leaves the settings free) requires even more randomness than the sampling of the (Born) probability measure $\mu _{\psi }$, which further undermines the claim that it is a deterministic theory.⁵⁰

The analysis given so far focused on the necessity of correctly sampling a probability measure $\mu$: if, so far in the context of hidden variable theories, where $\mu =\mu _{\psi }$, this is not done correctly, quantum-mechanical predictions such as Surface Locality may be threatened. But in general there are measurement outcome sequences that fail to sample $\mu$: atypical events with very low or even zero probability can and do occur. This is even the whole point of the “law of truly large numbers” quoted in Sect. 2! In general, the Hidden Locality (or no-signaling) property of quantum mechanics states that the probability

$$\begin{aligned} P_{\psi }(\alpha \mid a,b):= \sum _\beta P_{\psi }(\alpha , \beta \mid a, b) \end{aligned}$$

(5.4)

is independent of b, where $P_{\psi }(\alpha , \beta \mid a, b)$ is the Born probability that measurement of observables determined by the settings a and b give outcomes $\alpha$ and $\beta$. Indeed, we have

$$\begin{aligned} P_{\psi }(\alpha \mid a,b)=P_{\psi _{|A}}(\alpha \mid a), \end{aligned}$$

(5.5)

where $\psi _{|A}$ is the restriction of the state $\psi$ on $B(H_A\otimes H_B)$ to Alice’s part $B(H_A)$. Similarly, the Born probability $P_{\psi }(\beta \mid a,b)$ should be independent of a and in fact equals $P_{\psi _{|B}}(\beta \mid b)$.

However, I have repeatedly noted that the empirical probability extracted from a long measurement outcome sequence coincides with the corresponding Born probability only almost surely with respect to the induced probability measure on the space of outcome sequences, and hence outliers violating the property (5.4) exist (for finite sequences even with positive probability). If one such run is found, the door to superluminal signaling is open, at least in principle. To see this, recall that the crudest form of determinism is what is called Predictability by Cavalcanti and Wiseman [28], i.e. the property that

$$\begin{aligned} P_{\psi }(\alpha , \beta \mid a, b)\in \{0,1\}. \end{aligned}$$

(5.6)

It is easy to show that the conjunction of Predictability and Surface Locality implies factorization and hence, for random settings, the Bell (CHSH) inequalities, and therefore for suitable states $\psi$ this conjunction contradicts the statistical predictions of quantum mechanics as expressed by the Born rule. Accepting the latter, Surface Locality therefore implies unpredictability and hence some (very weak) form of randomness. There are many other results in this direction, ranging from e.g. Barrett, Kent, and Hardy [8] to Wolf (2015),⁵¹ involving varying definitions of randomness, all coming down to the implication

$$\begin{aligned} no\, signaling \Rightarrow randomness , \end{aligned}$$

(5.7)

assuming Freedom and random settings.⁵² What I’d like to argue for is the contrapositive

$$\begin{aligned} no\, randomness \Rightarrow signaling . \end{aligned}$$

(5.8)

My argument is only heuristic, since the terms are used differently in (5.7) and (5.8): to prove (5.7) one typically uses Born probabilities and other theoretical entities, whereas for (5.8) I use probabilities obtained from outcome sequences as limiting relative frequencies: (5.8) then comes from low-probability sequences that violate (5.4), which is satisfied with probability one by sufficiently random sequences. Indeed, this difference is the whole point:

Surface Locality is a statistical property, like the second law of thermodynamics.⁵³

Appendices

Appendix A: The Born Rule Revisited

Every argument on probabilities in quantum mechanics relies on the Born rule, which is so natural that it will probably (!) be with us forever. In order to explain its canonical mathematical origin, I use the C*-algebraic approach to quantum theory, which because of its ability to simultaneously talk about commutative and non-commutative algebras (and hence about classical and quantum probabilities) is especially useful in this context.⁵⁴ Indeed, the Born rule arises by restricting a quantum state $\omega$ on the non-commutative algebra B(H), i.e. the algebra of all bounded operators on some Hilbert space H, to the commutative C*-algebra $C^*(a)$ generated by some self-adjoint operator $a=a^*\in B(H)$ and the unit operator $1_H$ (so perhaps $C^*(a,1_H)$ would have been better notation).⁵⁵

This view has been inspired by the Copenhagen Interpretation, in that the associated probabilities originate in the classical description of a measurement setting, as called for by Bohr. However, precisely because of this, my derivation also seems to undermine the Copenhagen claim of the irreducibility of randomness, which may now clearly be traced back to a voluntary loss of information in passing from an algebra to one of its subalgebras and ignoring the rest (see also Sect. 3). Happily, the Copenhagen Interpretation will be saved from this apparent inconsistency by a mathematical property that has no classical analog.

As I see it, the Born rule is the quantum-mechanical counterpart of the following elementary construction in measure theory. Let X be a compact Hausdorff space,⁵⁶ with associated commutative C*-algebra $A=C(X)$ of complex-valued continuous functions on X. Let $f\in A$ be real-valued and denote its (automatically closed) range in ${{\mathbb {R}}}$ by $\sigma (f)$; the notation is justified by the fact that the spectrum of a self-adjoint operator in quantum theory is analogous to $\sigma (f)$ in every possible way. Let $\omega$ be a state on A, or, equivalently (by the Riesz representation theorem), a probability measure $\mu$ on X, related to $\omega$ by

$$\begin{aligned} \omega (f)=\int _X d\mu (x)\, f(x). \end{aligned}$$

(A.1)

The state $\omega$ then also defines a probability measure $\mu _f$ on the “spectrum” $\sigma (f)$ through

$$\begin{aligned} \mu _f(\Delta )=\mu (f\in \Delta ), \end{aligned}$$

(A.2)

where $\Delta \subseteq \sigma (f)$ and $f\in \Delta$ or $f^{-1}(\Delta )$ denotes the subset $\{x\in X\mid f(x)\in \Delta \}$ of X. Virtually all of probability theory is based on this construction, which I now rephrase. For any C*-algebra A with unit $1_A$, let $C^*(a)$ be the smallest C*-algebra in A (under the same operations and norm) that contains a and the unit $1_A$, as above for $A=B(H)$. If $a^*=a$, then $C^*(a)$ is the norm-closure of the set of all (finite) polynomials in a. Take $A=C(X)$ and $f\in C(X)$, assumed real-vaued (since $f^*=f$). This gives as isomorphism

$$\begin{aligned} C(\sigma (f))&{\mathop {\rightarrow }\limits ^{\cong }} C^*(f); \end{aligned}$$

(A.3)

$$\begin{aligned} g&\mapsto g\circ f, \end{aligned}$$

(A.4)

where $f\in C(X)$, $g\in C(\sigma (f))$, as follows from the Stone–Weierstrass Theorem.⁵⁷

If C is a C*-subalgebra of A with the same unit as A, then any state $\omega$ on A defines a state $\omega _{|C}$ on C. Thus $\omega _{|C^*(f)}$ is a state on $C^*(f)$. Furthermore, if $\varphi :B\rightarrow C$ is a unital homomorphism of unital C*-algebras (i.e., a linear map preserving all operations as well as the unit—in our case $\varphi$ is even an isomorphism), and if $\omega '$ is a state on C, then $\varphi _*\omega '= \omega '\circ \varphi$ is a state on B. Using (A.3)–(A.4), we apply this construction to

$$\begin{aligned} A=C(X),\,\,B=C(\sigma (f)),\,\,C= C^*(f); \end{aligned}$$

(A.5)

$$\begin{aligned} \omega \in S(C(X)),\,\,\omega '=\omega _{|C^*(f)},\,\,\varphi (g)=g\circ f, \end{aligned}$$

(A.6)

where S(A) is the state space of A, so that a state $\omega$ on C(X) defines a state $\omega '$ on $C(\sigma (f))$. Once again using the Riesz representation theorem then turns $\omega '$ into a probability measure $\mu '$ on $\sigma (f)$; and this is exactly the probability measure $\mu _f=\mu '$ defined in (A.2).

Copying this reasoning for quantum mechanics mutatis mutandis gives the Born measure. Instead of (A.5)–(A.6), for some given $a=a^*\in B(H)$ we now take

$$\begin{aligned} A=B(H),\,\,B=C(\sigma (a)),\,\,C= C^*(a); \end{aligned}$$

(A.7)

$$\begin{aligned} \omega \in S(B(H)),\,\,\omega '=\omega _{|C^*(a)},\,\,\varphi (g)=g (a), \end{aligned}$$

(A.8)

where (A.3)–(A.4) is replaced by the continuous functional calculus (CFC), that is,

$$\begin{aligned} C(\sigma (a))&{\mathop {\rightarrow }\limits ^{\cong }} C^*(a); \end{aligned}$$

(A.9)

$$\begin{aligned} g&\mapsto g(a). \end{aligned}$$

(A.10)

Here $\sigma (a)$ is the spectrum of a, defined as the set of $\lambda \in {{\mathbb {C}}}$ for which the operator $a-\lambda \cdot 1_H$ is not invertible in B(H); if $\dim (H)<\infty$, then $\sigma (a)$ is the set of eigenvalues of a. In most applications the state $\omega$ is normal, i.e., given by a density operator $\rho$ on H through

$$\begin{aligned} \omega (a)=\mathrm{Tr}\,(\rho a). \end{aligned}$$

(A.11)

We then replay our record: a state $\omega \in S(B(H))$ restricts to a state $\omega _{|C^*(a)}$ on $C^*(a)$, which is mapped to a state on $C(\sigma (a))$ by the CFC, which state in turn is equivalent to a probability measure on $\sigma (a)$. The probability measure $\mu _a$ on $\sigma (a)$ obtained by this construction is exactly the Born measure, which is more commonly defined as follows:

Theorem A.1

Let H be a Hilbert space, let $a^*=a\in B(H)$, and let $\omega$ be a state on B(H). There exists a unique probability measure $\mu _a$ on the spectrum $\sigma (a)$ of a such that

$$\begin{aligned} \omega (g(a))=\int _{\sigma (a)} d\mu _a(\lambda )\, g(\lambda ), \,\, g\in C(\sigma (a)). \end{aligned}$$

(A.12)

We now return to the issue raised in the introduction to this appendix: even short of hidden variables (see Sect. 3), to what extent is the probabilistic structure of quantum mechanics that is already inherent in the Born measure reducible to ignorance? From a naive perspective it is, for in the corresponding classical case the measure $\mu _f$ on $\sigma (f)$ does not determine the measure $\mu$ on X except when $C^*(f)=C(X)$ (which is certainly possible, take e.g. $X=[0,1]$ and $f(x)=x$). However, in the quantum case it so happens that if a is maximal (i.e. its spectrum is nondegenerate, or, equivalently, $C^*(a)$ is a maximal commutative C*-subalgebra of B(H)), then the measure $\mu _a$ on $\sigma (a)$does determine the state $\omega$, at least if $\omega$ is pure and normal, as in (A.11), despite the fact that $C^*(a)$ is just a small part of B(H). So the Copenhagen Interpretation is walking a tightrope here, but it doesn’t fall.⁵⁸

The Born measure is a mathematical construction; what is its relationship to experiment? This relationship must be the source of the (alleged) randomness of quantum mechanics, for the Schrödinger equation is deterministic. We start by postulating, as usual, that $\mu _a(A)$ is the (single case) probability that measurement of the observable a in the state $\omega$ (which jointly give rise to the pertinent Born measure $\mu _a$) gives a result $\lambda \in A\subset \sigma (a)$. What needs to be clarified in the above statement is the word probability.

Although I do not buy Lewis’s [78] “Best Systems Account” (BSA) of laws of nature,⁵⁹ I do agree with his identification of single-case probabilities as numbers (consistent with the probability calculus as a whole) that theory assigns to events, upon which long-run frequencies provide empirical evidence for the theory in question,⁶⁰ but do not define probabilities.⁶¹ The Born measure is a case in point: these probabilities are theoretically given, but have to be empirically verified by long runs of independent experiments. In other words, by the results reviewed below such experiments provide numbers whose role it is to test the Born rule as a hypothesis. This is justified by (4.3) in §4, which equates probabilities computed from long-run frequencies with the corresponding single-case probabilities that define the clause “with probability one” under which (4.3) holds, and without which clause the limit in (4.3) would be undefined. As explained in Sects. 4–5, the relevance of (4.3) to quantum physics comes from Theorem 5.1, which I now prove.

Proof of Theorem 5.1

Let $a=a^*\in B(H)$, where H is a Hilbert space and B(H) is the algebra of all bounded operators on H, and let $\sigma (a)$ be the spectrum of a. For simplicity (and since this is enough for our applications, where $H={{\mathbb {C}}}^2$) I assume $\dim (H)<\infty$, so that $\sigma (a)$ simply consists of the eigenvalues $\lambda _i$ of a (which may be degenerate). Let us first consider a finite number N of runs of an identical measurement of a. The first option in the theorem corresponds to a simultaneous measurement of the commuting operators

$$\begin{aligned} a_1&=a\otimes 1_H\otimes \cdots \otimes 1_H; \end{aligned}$$

(A.13)

$$\begin{aligned}&\cdots \nonumber \\ a_N&=1_H\otimes \cdots \otimes 1_H\otimes a, \end{aligned}$$

(A.14)

all defined on the N-fold tensor product $H^N\equiv H^{\otimes N}$ of H with itself.⁶² To put this in a broader perspective, consider any set $(a_1, \ldots , a_N)\equiv \underline{a}$ of commuting operators on any Hilbert space K (of which (A.13)–(A.14) is obviously a special case with $K=H^N$). These operators have a joint spectrum$\sigma (\underline{a})$, whose elements are the joint eigenvalues$\underline{\lambda }=(\lambda _1, \ldots , \lambda _N)$, defined by the property that there exists a nonzero joint eigenvector $\psi \in K$ such that $a_i\psi =\lambda _i\psi$ for all $i=1, \ldots , N$; clearly, the joint spectrum is given by

$$\begin{aligned} \sigma (\underline{a})=\{\underline{\lambda }\in \sigma (a_1)\times \cdots \times \sigma (a_N)\mid e_{\underline{\lambda }}\equiv e^{(1)}_{\lambda _1}\cdots e^{(n)}_{\lambda _n} \ne 0\} \subseteq \sigma (a_1)\times \cdots \times \sigma (a_N), \end{aligned}$$

(A.15)

where $e^{(i)}_{\lambda _i}$ is the spectral projection of $a_i$ on the eigenspace for the eigenvalue $\lambda _i\in \sigma (a_i)$. Von Neumann’s Born rule for the probability $p_{\underline{a}}(\underline{\lambda })$ of finding $\underline{\lambda }\in \sigma (\underline{a})$ then simply reads

$$\begin{aligned} p_{\underline{a}}(\underline{\lambda })=\omega (e_{\underline{\lambda }}), \end{aligned}$$

(A.16)

where $\omega$ is the state on B(K) with respect to which the Born probability is defined. If $\dim (K)<\infty$, as I assume, we always have $\omega (a)=\mathrm{Tr}\,(\rho a)$ for some density operator $\rho$, and for a general Hilbert space K this is the case iff the state $\omega$ is normal on B(K). For (normal) pure states we have $\rho =|\psi \rangle \langle \psi |$ for some unit vector $\psi \in K$, in which case

$$\begin{aligned} p_{\underline{a}}(\underline{\lambda })=\langle \psi ,e_{\underline{\lambda }}\psi \rangle . \end{aligned}$$

(A.17)

The Born rule (A.16) follows from the same reasoning as the single-operator case explained in the run-up to Theorem A.1 (Landsman [75], Sect. 4.1):⁶³ we have an isomorphism

$$\begin{aligned} C^*(\underline{a})\cong C(\sigma (\underline{a})) \end{aligned}$$

(A.18)

of (commutative) C*-algebras, and under this isomorphism the restriction of the state $\omega$, originally defined on B(K), to $C^*(\underline{a})$ defines a probability measure $\mu _{\underline{a}}$ on the joint spectrum $\sigma (\underline{a})$, which is just the Born measure whose probabilities are given by (A.16). For (A.13)–(A.14) we have equality in (A.15); since in that case $\sigma (a_i)=\sigma (a)$, we obtain

$$\begin{aligned} \sigma (\underline{a})=\sigma (a)^N, \end{aligned}$$

(A.19)

and, for all $\lambda _i\in \sigma (a)$ and states $\omega$ on $B(H^N)$, the Born rule (A.16) becomes

$$\begin{aligned} p_{\underline{a}}(\lambda _1, \ldots , \lambda _N)=\omega (e_{\lambda _1}\otimes \cdots \otimes e_{\lambda _N}). \end{aligned}$$

(A.20)

Now take a given state $\omega _1$ on B(H). Reflecting the idea that $\omega$ is the state on $B(H^N)$ in which N independent measurements of $a\in B(H)$ in the state $\omega _1$ are carried out, choose

$$\begin{aligned} \omega =\omega _1^N, \end{aligned}$$

(A.21)

i.e. the state on $B(H^N)$ defined by linear extension of its action on elementary tensors:

$$\begin{aligned} \omega _1^N(b_1\otimes \cdots \otimes b_n)=\omega _1(b_1) \cdots \omega _N(b_N), \end{aligned}$$

(A.22)

where $b_i\in B(H)$ for each $i=1, \ldots , N$. It follows that

$$\begin{aligned} \omega ^N(e_{\lambda _1}\otimes \cdots \otimes e_{\lambda _N})=\omega _1(e_{\lambda _1})\cdots \omega _1(e_{\lambda _N})=p_a(\lambda _1)\cdots p_a(\lambda _N), \end{aligned}$$

(A.23)

so that the joint probability of the outcome $(\lambda _1, \ldots , \lambda _N)\in \sigma (\underline{a})$ is simply

$$\begin{aligned} p_{\mathbf {a}}(\lambda _1, \ldots , \lambda _N)=p_a(\lambda _1)\cdots p_a(\lambda _N). \end{aligned}$$

(A.24)

Since these are precisely the probabilities for option 2 (i.e. the Bernoulli process), i.e.,

$$\begin{aligned} \mu _{\underline{a}}=\mu _a^N, \end{aligned}$$

(A.25)

this proves the claim for $N<\infty$.

To describe the limit $N\rightarrow \infty$, let B be any C*-algebra with unit $1_B$; below I take $B=B(H)$, $B=C^*(a)$, or $B=C(\sigma (a))$. We now take

$$\begin{aligned} A_N=B^{\otimes N}, \end{aligned}$$

(A.26)

the N-fold tensor product of B with itself.⁶⁴ The special cases above may be rewritten as

$$\begin{aligned} B(H)^{\otimes N}&\cong B(H\otimes \cdots \otimes H); \end{aligned}$$

(A.27)

$$\begin{aligned} C^*(a)^{\otimes N}&\cong C^*(a_1, \ldots , a_N); \end{aligned}$$

(A.28)

$$\begin{aligned} C(\sigma (a))^{\otimes N}&\cong C(\sigma (a) \times \cdots \times \sigma (a)), \end{aligned}$$

(A.29)

with N copies of H and $\sigma (a)$, respectively; in (A.28) the $a_i$ are given by (A.13)–(A.14).

We may then wonder if these algebras have a limit as $N\rightarrow \infty$. They do, but it is not unique and depends on the choice of observables, that is, of the infinite sequences ${\mathbf {a}}=({\mathsf {a}}_1,{\mathsf {a}}_2, \ldots )$, with ${\mathsf {a}}_N\in A_N$, that are supposed to have a limit. One possibility is to take sequences ${\mathbf {a}}$ for which there exists $M\in {{\mathbb {N}}}$ and ${\mathsf {a}}_M\in A_M$ such that for each $N\ge M$,

$$\begin{aligned} {\mathsf {a}}_N={\mathsf {a}}_M \otimes 1_B\cdots \otimes 1_B, \end{aligned}$$

(A.30)

with $N-M$ copies of the unit $1_B$. On that choice, one obtains the infinite tensor product $B^{\otimes \infty }$, see Landsman [75], Sect. C.14. The limit of (A.27) in this sense is $B(H^{\otimes \infty })$, where $H^{\otimes \infty }$ is von Neumann’s ‘complete’ infinite tensor product of Hilbert spaces,⁶⁵ in which $C^*(a)^{\otimes \infty }$ is the C*-algebra generated by the operators $(a_1,a_2, \ldots )$. The limit of (A.29) is

$$\begin{aligned} C(\sigma (a))^{\otimes \infty }\cong C(\sigma (a)^{{{\mathbb {N}}}}), \end{aligned}$$

(A.31)

where $\sigma (a)^{{{\mathbb {N}}}}$, which we previously saw as a measure space (as a special case of $X^{{{\mathbb {N}}}}$ for general compact Hausdorff spaces X), is now seen as a topological space with the product topology, in which it is compact.⁶⁶ As in the finite case, we have an isomorphism

$$\begin{aligned} C^*(a)^{\otimes \infty } \cong C(\sigma (a))^{\otimes \infty }, \end{aligned}$$

(A.32)

and hence, on the given identifications and the notation (A.13) - (A.14), we have

$$\begin{aligned} C^*(a_1, a_2, \ldots )\cong C(\sigma (a)^{{{\mathbb {N}}}}). \end{aligned}$$

(A.33)

It follows from the definition of the infinite tensor products used here that each state $\omega _1$ on B defines a state $\omega _1^{\infty }$ on $B^{\otimes \infty }$. Take $B=B(H)$ and restrict $\omega _1^{\infty }$, which a priori is a state on $B(H^{\otimes \infty })$, to its commutative C*-subalgebra $C^*(a_1, a_2, \ldots )$. The isomorphism (A.33) then gives a probability measure $\mu _{\underline{a}}$ on the compact space $\sigma (a)^{{{\mathbb {N}}}}$, where the label $\underline{a}$ now refers to the infinite set of commuting operators $(a_1, a_2, \ldots )$ on $H^{\otimes \infty }$. To compute this measure, use (A.12) and the fact that by construction functions of the type

$$\begin{aligned} f(\lambda _1, \lambda _2, \ldots )=f^{(N)}(\lambda _1, \ldots , \lambda _N), \end{aligned}$$

(A.34)

where $N<\infty$ and $f^{(N)}\in C(\sigma (a)^N)$, are dense in $C(\sigma (a)^{{{\mathbb {N}}}})$ with respect to the appropriate supremum-norm, and that in turn finite linear combinations of factorized functions $f^{(N)}(\lambda _1, \ldots , \lambda _N)=f_1(\lambda _1)\cdots f_N(\lambda _N)$ are dense in $C(\sigma (a)^{N})$. It follows from this that

$$\begin{aligned} \mu _{\underline{a}}=\mu _a^{\infty }. \end{aligned}$$

(A.35)

Since this generalizes (A.25) to $N=\infty$, this finishes the proof of Theorem 5.1. $\square$

Appendix B: 1-Randomness

As we have seen, randomness (of measurement outcomes) in quantum mechanics was originally defined by Born as indeterminism. This is what hidden variable theories like ’t Hooft’s and Bohmian mechanics challenge, to which in turn Bell’s Theorem and the Free Will Theorem (read in a specific way reviewed in Appendix C below) provide obstacles. Indeterminism is a physical definition of randomness, which in mathematics is most closely matched by incomputability. However, there is a much deeper notion of randomness in mathematics, which is what at least in my view quantum mechanics should aspire to produce. This notion, now called 1-randomness, has its roots in Hilbert’s Sixth Problem, viz. the Mathematical Treatment of the Axioms of Physics,⁶⁷ which was taken up independently and very differently by von Mises [128] and by Kolmogorov [70].

Von Mises was a strict frequentist for whom probability was a derived concept, predicated on first having a good notion of a random sequence from which relative frequencies defining probability could be extracted. Kolmogorov, on the other hand, started from an axiomatic a priori notion of probability from which a suitable mathematical concept of randomness was subsequently to be extracted. Despite the resounding and continuing success of the first step, Kolmogorov’s initial failure to achieve the follow-up led to his later notion of algorithmic randomness, which (subject to a technical improvement) was to become one of the three equivalent definitions of 1-randomness. In turn, von Mises’s failure to adequately define random sequences eventually led to the other two definitions.⁶⁸

Kolmogorov’s problem, which was noticed already by Laplace and perhaps even earlier probabilists, was that, specializing to a 50-50 Bernoulli process for simplicity,⁶⁹ any binary string $\sigma$ of length N has probability $P(\sigma ) = 2^{-N}$ and any (infinite) binary sequence x has probability $P(x) = 0$, although say $\sigma =0011010101110100$ looks much more random than $\sigma =111111111111111$. In other words, their probabilities say little or nothing about the randomness of individual outcomes. Imposing statistical properties helps but is not enough to guarantee randomness. It is slightly easier to explain this in base 10, to which I therefore switch for a moment. If we call a sequence xBorel normal if each possible string $\sigma$ in x has (asymptotic) frequency $10^{-|\sigma |}$ (so that each digit $0, \ldots , 9$ occurs 10% of the time, each block 00 to 99 occurs 1% of the time, etc.), then Champernowne’s number

$$\begin{aligned} 0123456789101112131415161718192021222324252629282930 \ldots \end{aligned}$$

can be shown to be Borel normal. The decimal expansion of $\pi$ is conjectured to be Borel normal, too (and has been empirically verified to be so in billions of decimals), but these numbers are hardly random: they are computable, which is an antipode to randomness.

Von Mises’s problem was that his definition of a random sequence (called a Kollektiv), despite his great insights, simply did not work. Back to binary sequences, his notion of randomness was supposed to guarantee the existence of limiting relative frequencies (which in turn should define the corresponding single-case probabilities), but of course he understood that a sequence like $01010101\cdots$ is not very random at all. His idea was that limiting relative frequencies should exist not only for the given sequence, but also for all its permutations defined by so-called place-selection functions, which he tried to find (in vain) by precluding successful gambling strategies on the digits of the sequence.

Of the three equivalent definitions of 1-randomness already mentioned in Sect. 2, namely:

$$Incompressibility; \; Patternlessness; \; Unpredictability$$

the first may be said to go back to Kolmogorov’s struggle, whereas the other two originate in later attempts by other mathematicians to improve the work of von Mises.⁷⁰

Although there isn’t a single “correct” mathematical notion of randomness, the notion of 1-randomness featured here stands out (and represents a consensus view) largely because it can be defined in these three equivalent yet fairly different ways, each of which realizes some basic intuition on randomness (of course defined through its obvious antipode!).

In what follows, these notions will be defined more precisely, followed by some of their consequences.⁷¹ We assume basic familiarity with the notion of a computable function $f:{{\mathbb {N}}}\rightarrow {{\mathbb {N}}}$, which may technically be defined through recursion theory or equivalently through Turing machines: a function is computable if it can be computed by a computer.

1.1 Incompressibility

The idea is that a string or sequence is random iff its shortest description is the sequence itself, but the notion of a description has to made precise to avoid Berry’s paradox:

The Berry number is the smallest positive integer that cannot be described in less than eighteen words.

The paradox, then, is that on the one hand this number must exist, since only finitely many integers can be described in less than eighteen words and hence the set of such numbers must have a lower bound, while on the other hand Berry’s number cannot exists by its own definition. This is, of course, one of innumerable paradoxes of natural language, which, like the liar’s paradox, will be seen to lead to an incompleteness theorem once the notion of a description has been appropriately formalized in mathematics, as follows.⁷²

The plain Kolmogorov complexity$C(\sigma )$ of $\sigma \in \underline{2}^{\underline{N}}$ is defined as the length (in bits) of the shortest computer program (run on some fixed universal Turing machine U) that computes $\sigma$. The choice of U affects this definition only up to a $\sigma$-independent constant. For technical reasons (especially for defining the randomness of sequences) it is preferable to work with prefix-free Turing machinesT, whose domain D(T) consists of a prefix-free subset of $\underline{2}^*$, i.e., if $\sigma \in D(T)$ then $\sigma \tau \notin D(T)$ for any $\sigma ,\tau \in \underline{2}^*$, where $\sigma \tau$ is the concatenation of $\sigma$ and $\tau$, as the notation suggests. This is also independent of U up to a constant, and defines the the prefix-free Kolmogorov complexity$K(\sigma )$ as the length of the shortest program (run on some fixed universal prefix-free Turing machine U) that computes $\sigma$. For fixed $c\in {{\mathbb {N}}}$ we then say that $\sigma \in \underline{2}^*$ is c-compressible if $K(\sigma )<|\sigma |-c$; of course, this depends on U via $K(\sigma )$, but nonetheless a simple counting argument shows:

At least $2^N-2^{N-c+1}+1$ strings $\sigma$ of length $|\sigma |=N$ are notc-compressible.

Clearly, as $\sigma$ grows and $c>0$, these will form the overwhelming majority. Finally, $\sigma$ is Kolmogorov c-random if it is not c-compressible, i.e., if $K(\sigma )\ge |\sigma |-c$, and Kolmogorov random if it is not c-compressible for any $c\in {{\mathbb {N}}}$, that is, if one even has $K(\sigma )\ge |\sigma |$. Since $K(\sigma )$ can at most be equal to the length of an efficient printing program plus the length of $\sigma$, this simply means that $\sigma$ is Kolmogorov random if

$$\begin{aligned} K(\sigma )\approx |\sigma |, \end{aligned}$$

(B.1)

where $\approx$ means ‘up to a $\sigma$-independent constant’. Finally, if $x_{|N}$ is the initial segment of length N of a sequence $x\in \underline{2}^{{{\mathbb {N}}}}$, i.e. $|x_{|N}|=N$, then x is Kolmogorov–Chaitin random if

$$\begin{aligned} \lim _{N\rightarrow \infty } \frac{K(x_{|N})}{N}=1. \end{aligned}$$

(B.2)

Equivalently,⁷³ a sequence x is Kolmogorov–Chaitin random if there exists $c\in {{\mathbb {N}}}$ such that each truncation $x_{|N}$ satisfies $K(x_{|N})\ge N - c$. I note with satisfaction that randomness of sequences, seen as idealizations of finite strings, as such satisfies Earman’s principle (Sect. 4).

This definition of randomness of both finite strings and infinite sequences looks very appealing, but in a way it is self-defeating, since although it is defined in terms of computability, the complexity function K is not computable, and hence one cannot even determine algorithmically if a finite string $\sigma$ is random (let alone an infinite one).⁷⁴

Moreover, Berry’s paradox strikes back through Chaitin’s incompleteness Theorem: For any consistent, sound, and sufficiently rich formal system F (containing basic arithmetic) there is a constant f such that the statement $K(\sigma ) > f$ cannot be proved in F for any $\sigma \in \underline{2}^*$, although it is true for all (random) strings $\sigma$ that satisfy $K(\sigma ) \ge |\sigma | > f$ (and there are infinitely many such strings, as the above counting argument shows). To get a very rough idea of the proof, let me just say that any proof in F of the sentence $K(\sigma ) > f$ would identify $\sigma$ and hence give a shorter description of $\sigma$ than its complexity $K(\sigma )$ allows.

Chaitin’s Theorem gives a new and inexhaustible class of unprovable but true statements (which also lie in a very different class from those provided by Gödel himself):

For almost all random strings their randomness cannot be proved.⁷⁵

To the extent that there are deep logico-philosophical truths, surely this is one! Compare:

It may be taken for granted that any attempt at defining disorder in a formal way will lead to a contradiction. This does not mean that the notion of disorder is contradictory. It is so, however, as soon as I try to formalize it.⁷⁶

The satisfactory definition of 1-randomness proves this wrong, but the statement preceding it is a correct version of a similar intuition: randomness is by its very nature elusive.

1.2 Patternlessness

This is the most direct attack on the “paradox of probability”, which states that each individual sequence has probability zero despite the huge differences in their (true or apparent) “randomness”. We use the notation of §4, so that $\mu ^{\infty }$ is the probability measure on the set $\underline{2}^{{{\mathbb {N}}}}$ of all binary sequences induced by the 50-50 measure on the outcome space $\{0,1\}$ of a single fair coin flip (the discussion is easily adapted to other cases).

As opposed to single outcomes, a key role will be played by tests, i.e. (measurable) sets$T\subset \underline{2}^{{{\mathbb {N}}}}$ of outcomes for which $\mu ^{\infty }(T)=0$. Such a test defines a property that a sequence $x\in \underline{2}^{{{\mathbb {N}}}}$ may or may not have, namely membership of T. Conversely, and perhaps less circularly, one may start with a given pattern that x might have, which would make it appear less random if it had it, and express this pattern as a test T. Conceptually, tests contain outcomes that are supposed to be “atypical”, and randomness of $x\in \underline{2}^{{{\mathbb {N}}}}$ will be defined by the property of not belonging to any such test, making x “typical”. For example, the property $\lim _{N\rightarrow \infty } N^{-1}\sum _{n=1}^N x_n=1/2$ is “typical” for a fair coin flip and indeed (by the strong law of large numbers) it holds with probability 1 (in that the set E of all x for which this limit exists and equals 1/2 has $\mu ^{\infty }(E)=1$, so that its complement $T=\underline{2}^{{{\mathbb {N}}}}\backslash E$ has $\mu ^{\infty }(T)=0$). One has to proceed very carefully, though, since all singletons $T=\{x\}$ for $x\in \underline{2}^{{{\mathbb {N}}}}$ are to be excluded; indeed, these all have measure zero and yet, returning to the paradox of probability, some uncontroversially random sequence x (on whatever definition) would fail to be random by the criterion just proposed if all measure zero sets were included in the list of tests. Kolmogorov’s former PhD student Martin-Löf [86] saw his way out of this dilemma, combining techniques from computability/recursion theory with some ideas from the intuitionistic/constructive mathematics of the great L.E.J. Brouwer:

• 1. One specializes to tests of the form $T=\cap _{n\in {{\mathbb {N}}}} U_n$, where $U_{n+1}\subseteq U_n$ and

  $$\begin{aligned} \mu ^{\infty }(U_n)\le 2^{-n}, \end{aligned}$$

  (B.3)

  which guarantees that $\mu ^{\infty }(T)=0$. Perhaps the simplest example is not the law of large numbers, for which see Li and Vitányi [79], §2.4, Example 2.4.2, but the test where $U_n$ consists of all sequences starting with n zeros; clearly, $\mu ^{\infty }(U_n)= 2^{-n}$. In this case, the test $T=\{x_{\mathrm {love}}\}$does consist of a singleton $x_{\mathrm {love}}=000\cdots$ (zeros only).

• 2. Both the sets $U_n$ and the map $n\mapsto U_n$ have to be computable in a suitable sense.⁷⁷

• 3. A sequence $x\in \underline{2}^{{{\mathbb {N}}}}$ is Martin-Löf random (= typical) if $x\notin T$ for any such test T.

If $x\notin T$, one says that xpasses the test T. Note that the computability requirement implies that the set of all tests T satisfying these two criteria is countable, which fact by itself already shows what a huge cut in the set of all measure-zero sets has been achieved.⁷⁸

It should be clear that this definition makes no sense if the sample space is finite, but in order to adhere to Earman’s principle one could still check to what extent randomness of sequences is determined by their finite initial segments. This is indeed the case, as follows from the detailed structure of the admissible sets $U_n$ above (see footnote 77).

1.3 Unpredictability

Unpredictability is a family resemblance, like randomness: the following definition is just one possibility, cf. e.g. footnote 41 and eq. (5.6). Having said this: unpredictability of a binary sequence $x = (x_1, x_2, \ldots )$ is formalized as the impossibility of a successful betting strategy on the successive digits of x. Suppose a punter with initial capital d(0) starts by betting $b(x_1=0)\equiv b(0)$ on $x_1 = 0$ and $b(x_1 = 1)\equiv b(1)$ on $x_1 = 1$, where $b(0) + b(1) = d(0)$, with fair payoff $d(x_1) = 2b(0)$ if $x_1 = 0$ and $d(x_1) = 2b(1)$ if $x_1 = 1$. It follows that

$$\begin{aligned} d(0) + d(1) =2(b(0) + b(1)) = 2d(0). \end{aligned}$$

(B.4)

This motivates the concept of a martingale as a function $d: \underline{2}^*\rightarrow [0,\infty )$ that satisfies

$$\begin{aligned} d(\sigma 0)+d(\sigma 1)=2d(\sigma ), \end{aligned}$$

(B.5)

for each $\sigma \in \underline{2}^*$. Both the betting strategy itself and the payoff (for given $x\in \underline{2}^{{{\mathbb {N}}}}$) can be reconstructed from d: the bet on the $N+1$’th digit $x_{N+1}$ is given by

$$\begin{aligned} b(x_{N+1}=0)&=\frac{1}{2}d(x_{N}0); \end{aligned}$$

(B.6)

$$\begin{aligned} b(x_{N+1}=1)&=\frac{1}{2}d(x_{N}1), \end{aligned}$$

(B.7)

and after N bets the punter owns $d(x_{|N})$. A martingale dsucceeds on $A\subset \underline{2}^{{{\mathbb {N}}}}$ if

$$\begin{aligned} \lim \sup _{N\rightarrow \infty } d(x_{|N})=\infty \,\,\, \text{ for } \text{ each } x\in A, \end{aligned}$$

(B.8)

in which case the punter beats the casino.⁷⁹ Our first impulse would now be to call $x\in \underline{2}^{{{\mathbb {N}}}}$ random if there exists no martingale that succeeds on $A=\{x\}$, but, Ville [125] proved:

Let $A\subset \underline{2}^{{{\mathbb {N}}}}$. Then $\mu ^{\infty }(A)=0$ if and only if there exists a martingale that succeeds on A.

Here A should be measurable. In particular, for any sequence $x\in \underline{2}^{{{\mathbb {N}}}}$ there exists a martingale that succeeds on x, and hence no sequence x would be random on the criterion that no martingale succeeds on it. Fortunately, the previous two definitions of randomness suggest that all that is missing is a suitable notion of computability for martingales. This notion was provided by Schnorr [102]: all that needs to be added is that the class of martingales d that succeed on A be uniformly left computably enumerable, in the sense that firstly each real number $d(\sigma )$ is the limit of a computable increasing sequence of rational numbers, and secondly that there is a single program that computes $d(\sigma )$ in that way.

Defining $x\in \underline{2}^{{{\mathbb {N}}}}$ to be Schnorr random if there exists no uniformly left computably enumerable martingale that succeeds on it,⁸⁰ one has the crowning theorem on 1-randomness:⁸¹

Theorem B.1

A sequence $x\in \underline{2}^{{{\mathbb {N}}}}$ is Kolmogorov–Chaitin random (i.e. incompressible) iff it is Martin-Löf random (i.e. patternless) and iff it is Schnorr random (i.e. unpredictable).

These equivalent conditions, then, define 1-randomness of sequences $x\in \underline{2}^{{{\mathbb {N}}}}$.

1.4 Some Reflections on 1-Randomness

Any sound definition of randomness (for binary sequences) has to navigate between Scylla and Charybdis: if the definition is too weak (such as Borel normality), counterexamples will undermine it (such as Champernowne’s number), but if it is too strong (such as being lawless as in Brouwer’s choice sequences, cf. §2), it will not hold almost surely in a 50-50 Bernoulli process. In this respect 1-randomness does very well: see Theorem 4.1 and the theorem below (which lies at the basis of Corollary 4.2):

Theorem B.2

Any 1-random sequence is Borel normal, incomputable, and contains any finite string infinitely often.⁸²

In fact, since computability is sometimes used as a mathematical metaphor for determinism, it is worth mentioning that 1-random sequences are incomputable in a very strong sense: each 1-random sequence x is immune in the sense that neither the set $\{n\in {{\mathbb {N}}}\mid x_n=1\}$ nor its complement $\{n\in {{\mathbb {N}}}\mid x_n=0\}$ is computably enumerable (c.e.), or can even contain an infinite c.e. subset. Thus 1-randomness is far stronger than mere incomputability (or perhaps indeterminism). On the other hand, as already noted in §5, Chaitin’s Incompleteness Theorem (see Appendix B) makes it impossible to prove that some given outcome sequence of a quantum-mechanical experiment with binary outcomes with 50-50 Born probabilities is 1-random even if it is (which is worrying, because one can never be sure, but only almost sure, that outcomes are 1-random). At best, in a bipartite (Alice and Bob) setting of the kind discussed in §5, one may hope for results showing that if an outcome is not 1-random, then some kind of superluminal signaling is possible, but, as reviewed in §5, even that much has not been rigorously established so far.⁸³

This suggests to try and prove randomness properties of such outcomes sequences that are on the one hand weaker than 1-randomness, like incomputability, but on the other hand are stronger, in that they hold surely. To this end, Abbott, Calude, and Conder [1] take a 3-level quantum system, prepare this in some state $\psi \in {{\mathbb {C}}}^3$ and repeatedly (indeed, infinitely often) measure any projection $|\varphi \rangle \langle \varphi |$ for which $\sqrt{5/14}\le |\langle \varphi ,\psi \rangle |\le \sqrt{9/14}$; for an experimental setup realizing these measurements see Kulikov et al. [73]. In that case there is a finite set ${\mathcal {P}}$ of 1-dimensional projections on ${{\mathbb {C}}}^3$ that contains $|\varphi \rangle \langle \varphi |$ as well as $|\psi \rangle \langle \psi |$ and admits no coloring (Abbott, Calude, and Svozil, [2]).⁸⁴ Some particular outcome sequence is described by a function $f:{{\mathbb {N}}}\rightarrow \{0,1\}$. The crucial assumption Abbott et al. then make is that if f is computable, then it extends to a function $\tilde{f}: {{\mathbb {N}}}\times {\mathcal {P}}\rightarrow \{0,1\}$ for which each $\tilde{f}_n: {\mathcal {P}}\rightarrow \{0,1\}$ is a coloring of ${\mathcal {P}}$, where $\tilde{f}_n(e)=\tilde{f}(n,e)$. Clearly, then, on this assumption f cannot be computable. This gives the desired result, but moves the burden of proof to justifying the assumption. Phrased by the authors in terms of “elements of physical reality” à la EPR, their assumption in fact postulates that if f is computable, then it originates in some (“morally” deterministic) non-contextual hidden-variable theory (to which their own sharpened Kochen–Specker Theorem applies).

A different line of attack (see e.g. Acín et al., [4]; Bendersky et al. [13]; Herrero-Collantes and Garcia-Escartin [61]; Kovalsky et al. [72]) assumes Hidden Locality and some amount of Freedom (cf. §3) to prove that outcome sequences cannot originate in a hidden-variable theory (to which Bell’s Theorem applies) and hence are indeterministic.

Appendix C: Bell’s Theorem and Free Will Theorem

In support of the analysis of hidden variable theories in §3 this appendix reviews Bell’s [10] Theorem and the Free Will Theorem, streamlining earlier expositions (Cator and Landsman [27]; Landsman [75], Chapter 6) and leaving out proofs and other adornments.⁸⁵ In the specific context of ’t Hooft’s theory (where the measurement settings are determined by the hidden state) and Bohmian mechanics (where they are not, as in the original formulation of Bell’s Theorem and in most hidden variable theories) a major advantage of my approach is that both determined und undetermined stettings fall within its scope; the latter case arises from the former by adding an independence assumption.⁸⁶

As a warm-up I start with a version of the Kochen–Specker Theorem whose logical form is very similar to Bell’s [10] Theorem and the Free Will Theorem, as follows:

Theorem C.1

Determinism, Nature, Non-contextuality, and Freedom are contradictory.

Of course, this slightly unusual formulation hinges on the precise meaning of these terms.

• Determinism is the conjunction of the following two assumptions:

  • 1. There is a state space X with associated functions $A: X\rightarrow S$ and $L:X\rightarrow O$, where S is the set of all possible measurement settings Alice can choose from, namely a suitable finite set of orthonormal bases of ${{\mathbb {R}}}^3$ (11 well-chosen bases will to to arrive at a contradiction),⁸⁷ and O is some set of possible measurement outcomes. Thus some $x\in X$ determines both Alice’s setting $a=A(x)$and her outcome $\alpha =L(x)$.

  • 2. There exists some set $\Lambda$ and an additional function $H:X\rightarrow \Lambda$ such that

    $$\begin{aligned} L=L(A,H), \end{aligned}$$

    (C.1)

    in the sense that for each $x\in X$ one has $L(x)={\hat{L}}(A(x),H(x))$ for a certain function ${\hat{L}}:S \times \Lambda \rightarrow O$. This self-explanatory assumption just states that each measurement outcome $L(x)={\hat{L}}(a,\lambda )$ is determined by the measurement setting $a=A(x)$ and the “hidden” variable or state $\lambda =H(x)$ of the particle undergoing measurement.

• Nature fixes $O=\{(0,1,1), (1,0,1), (1,1,0)\}$, which is a non-probabilistic fact of quantum mechanics with overwhelming (though indirect) experimental support.

• Non-contextuality stipulates that the function ${\hat{L}}$ just introduced take the form

  $$\begin{aligned} {\hat{L}}((\mathbf {e}_1,\mathbf {e}_2,\mathbf {e}_3),\lambda )=(\tilde{L}(\mathbf {e}_1,\lambda ), \tilde{L}(\mathbf {e}_2,\lambda ), \tilde{L}(\mathbf {e}_3,\lambda )), \end{aligned}$$

  (C.2)

  for a single function $\tilde{L}:S^2\times \Lambda \rightarrow \{0,1\}$ that also satisfies $\tilde{L}(-\mathbf {e},\lambda )=\tilde{L}(\mathbf {e},\lambda )$.⁸⁸

• Freedom finally states that the following function is surjective:

  $$\begin{aligned} A\times H:X\rightarrow S\times \Lambda ; \quad \quad x\mapsto (A(x),H(x)). \end{aligned}$$

  (C.3)

  In other words, for each $(a,\lambda )\in S\times \Lambda$ there is an $x\in X$ for which $A(x)=a$ and $H(x)=\lambda$. This makes A and H “independent” (or: a and $\lambda$ are free variables).

See Landsman [75], §6.2 for a proof of the Kochen–Specker Theorem in this language.⁸⁹

Bell’s [10] Theorem and the Free Will Theorem both take a similar generic form, namely:

Theorem C.2

Determinism, Nature, (Hidden) Locality, and Freedom, are contradictory.

Once again, I have to explain what these terms exactly mean in the given context.

• Determinism is a straightforward adaptation of the above meaning to the bipartite “Alice and Bob” setting. Thus we have a state space X with associated functions

  $$\begin{aligned} A: X\rightarrow S;\,\,quad B: X\rightarrow S;\quad L:X\rightarrow O;\quad R: X\rightarrow O, \end{aligned}$$

  (C.4)

  where the set S of all possible measurement settings Alice and Bob can each choose from differs a bit between the two theorems: for the Free Will Theorem it is the same as for the Kochen–Specker Theorem above, as is the set O of possible measurement outcomes, whereas for Bell’s Theorem (in which Alice and Bob each measure a 2-level system), S is some finite set of angles (three is enough), and $O=\{0,1\}$.

  • In the Free Will case, these functions and the state $x\in X$ determine both the settings $a=A(x)$ and $b=B(x)$ of a measurement and its outcomes $\alpha =L(x)$ and $\beta =R(x)$ for Alice on the Left and for Bob on the Right, respectively.

  • Although this is also true in the Bell case, his theorem relies on measurement statistics (as opposed to individual outcomes), so that one in addition assumes a probability measure $\mu$ on X (sampled by repeated measurements, see Sects. 4–5).⁹⁰

  Furthermore, there exists some set $\Lambda$ and some function $H:X\rightarrow \Lambda$ such that

  $$\begin{aligned} L=L(A,B,H);\qquad R=R(A,B,H), \end{aligned}$$

  (C.5)

  in the sense that for each $x\in X$ one has functional relationships

  $$\begin{aligned} L(x)={\hat{L}}(A(x),B(x),H(x));\qquad R(x)={\hat{R}}(A(x),B(x),H(x)), \end{aligned}$$

  (C.6)

  for certain functions ${\hat{L}}:S \times S\times \Lambda \rightarrow O$ and ${\hat{R}}:S \times S\times \Lambda \rightarrow O$.

• Nature reflects elementary quantum mechanics of correlated 2-level and 3-level quantum systems for the Bell and the Free Will cases, respectively, as follows:⁹¹

  • In the Free Will Theorem, $O=\{(0,1,1), (1,0,1), (1,1,0)\}$ is the same as for the Kochen–Specker Theorem. In addition perfect correlation obtains: if $a=(\mathbf {e}_1,\mathbf {e}_2,\mathbf {e}_3)$ is Alice’s orthonormal basis and $b=(\mathbf {f}_1,\mathbf {f}_2,\mathbf {f}_3)$ is Bob’s, one has

    $$\begin{aligned} \mathbf {e}_i=\mathbf {f}_j\, \Rightarrow \, {\hat{L}}_i(a,b,z)={\hat{R}}_j(a,b,z), \end{aligned}$$

    (C.7)

    where ${\hat{L}}_i, {\hat{R}}_j: S \times S\times \Lambda \rightarrow \{0,1\}$ are the components of ${\hat{L}}$ and ${\hat{R}}$, respectively. Finally,⁹² if $(a',b'$) differs from (a, b) by changing the sign of any basis vector,

    $$\begin{aligned} {\hat{L}}(a',b',\lambda )={\hat{L}}(a,b,\lambda );\,\,{\hat{R}}(a',b',\lambda )={\hat{R}}(a,b,\lambda ). \end{aligned}$$

    (C.8)

  • In Bell’s Theorem, $O=\{0,1\}$, and the statistics for the experiment is reproduced as conditional joint probabilities given by the measure $\mu$ through

    $$\begin{aligned} P(L\ne R|A=a,B=b)=\sin ^2(a-b). \end{aligned}$$

    (C.9)

• (Hidden) Locality, which replaces Non-contextuality in Theorem C.1, means that

  $$\begin{aligned} L(A,B,H)=L(A,H);\qquad R(A,B,H)=R(B,H). \end{aligned}$$

  (C.10)

  In words: Alice’s outcome given $\lambda$ does not depend on Bob’s setting, and vice versa.

• Freedom is an independence assumption that looks different for both theorems:

  • In the Free Will Theorem it means that each $(a,b,\lambda )\in S\times S\times \Lambda$ is possible in that there is an $x\in X$ for which $A(x)=a$, $B(x)=b$, and $H(x)=\lambda$.

  • In Bell’s Theorem, (A, B, H) are probabilistically independent relative to $\mu$.⁹³

This concludes the joint statement of the Free Will Theorem and Bell’s [10] Theorem in the somewhat unusual form we need for the main text. The former is proved by reduction to the Kochen–Specker Theorem, whilst the latter follows by reduction to the usual version of Bell’s Theorem via the Freedom assumption; see Landsman (1917), Chapter 6 for details.

For our purposes these theorems are equivalent, despite subtle differences in their assumptions. Bell’s Theorem is much more robust in that it does not rely on perfect correlations (which are hard to realize experimentally), and in addition it requires almost no input from quantum theory.⁹⁴ On the other hand, Bell’s Theorem uses probability theory in a highly nontrivial way: like the hidden variable theories it is supposed to exclude it relies on the possibility of fair sampling of the probability measure $\mu$. The factorization condition defining probabilistic independence passes this requirement of fair sampling on to both the hidden variable and the settings, which brings us back to Sect. 5.

All trusting Nature, different parties may now be identified by the assumption they drop:

• Following Copenhagen quantum mechanics, most physicists reject Determinism;

• Bohmians reject (Hidden) Locality;

• ’t Hooft rejects Freedom.

However, as we argued in Sect. 5, even the latter two camps do not really have a deterministic theory underneath quantum mechanics because of their need to randomly sample the probability measure they must use to recover the predictions of quantum mechanics.