Homer nodded: von Neumann's surprising oversight

We review the famous no-hidden-variables theorem in John von Neumann's 1932 book on the mathematical foundations of quantum mechanics. We describe the notorious gap in von Neumann's argument, pointed out by Grete Hermann in 1935 and, more famously, by John Bell in 1966. We disagree with recent papers claiming that Hermann and Bell failed to understand what von Neumann was actually doing.

quantities, and you measure the same physical quantity R on a large enough random sample of the systems, then the mean of all those measurement outcomes is called Exp(R). 6 Implicit in Assumption A ′ , and in the notation Exp(R), is the physical assumption, not always emphasized, that this mean value does not depend on which of several possible distinct ways of measuring R might be chosen.
One way to define a physical quantity is to specify a way to measure it. As an important example, if R is a physical quantity that one does know how to measure, and f is a function that takes real numbers to real numbers, then one can define another physical quantity f (R) by specifying that to measure f (R) you measure R and then apply f to the outcome r of the R-measurement.
We shall point out below that the criticisms of Hermann's and Bell's readings of von Neumann by Bub and Dieks are invalidated by the fact that von Neumann's four assumptions also provide another way to define physical quantities that makes no explicit mention of measurements.
Assumption A ′ also states explicitly that Exp(R) is non-negative if the physical quantity R is "by nature" non-negative. Nobody has any issues with this. Exp(aR + bS + · · ·) = a Exp(R) + b Exp(S) + · · · . (1) If several different physical quantities R, S, . . . can be simultaneously measured, then you can define a physical quantity that is a function f of them all by specifying that f (R, S . . .) is measured by measuring them jointly, and applying f to the results r, s, . . . of all those measurements. The linearity condition B ′ for jointly measurable quantities follows straightforwardly from this definition, applied to the function f (r, s, . . .) = ar + bs + · · · . 6 In 1932 most physicists talked about probabilities in terms of ensembles. In this paper we follow von Neumann's language. For a given physical system an ensemble can be understood as an assignment of probabilities to the outcomes of all possible ways to measure physical quantities (or sets of jointly measurable physical quantities) defined on that system. For a given ensemble, the function Exp(R) is then the standard expectation value for the probability distribution associated with the particular measurement of R. Now it is one of the most important features of quantum mechanics that not all physical quantities can be simultaneously measured. 7 Extending the scope of Assumption B ′ to quantities R, S, . . . that are not jointly measurable is problematic, however, since at this stage it is not even clear what aR + bS + · · · in B ′ might mean for such quantities.
Indeed, von Neumann immediately remarks that B ′ characterizes such a linear combination "only in an implicit way", since there is "no way to construct from the measurement [instructions] for R, S, . . . such [instructions] for R + S + · · · ." 8 Bub and Dieks both take this to mean that von Neumann uses assumption B ′ to define linear combinations of physical quantities that are not simultaneously measurable. This is the entire basis for their criticisms of Bell and Hermann. If B ′ is just a definition, it cannot also be an invalid assumption, as Hermann and Bell maintain. But as we shall see below, the full set of von Neumann's four assumptions contains another way to define linear combinations of physical quantities that are not simultaneously measurable. With that alternative definition, Assumption B ′ can indeed impose a nontrivial constraint on the values an Exp function can have for such linear combinations. There is no reason to insist that Assumption B ′ must be taken as a definition.
Assumption I: (p. 313) There is a 1-to-1 correspondence between physical quantities R and Hermitian operators R that act on a Hilbert space. For any real-valued function f , if the quantity R has the operator R, then the quantity f (R) has the operator f (R).
The requirement that this 1-to-1 correspondence must be preserved by functions is quite powerful. We have noted in our discussion of Assumption A ′ von Neumann's specification of how to define functions of a physical quantity. Standard Hilbert space mathematics tells us how to define functions of a Hermitian operator. Requiring, as Assumption I does, that these two quite different ways of evaluating functions should preserve the one-to-one correspondence between physical quantities and Hermitian operators has surprisingly strong consequences. Appendix 1 illustrates the power of this function-preserving 1-1 correspondence.
7 The acknowledgment that some physical quantities cannot be jointly measured introduces a crucial feature of quantum mechanics even into assumptions A ′ and B ′ .
Because this association of physical quantities with Hermitian operators is one-toone, it is possible to use Hermitian operators to define physical quantities, and vice-versa.
Assumption II provides a pertinent example of this.
Assumption II: (p. 314) If the physical quantities R, S, . . . have the Hermitian operators R, S, . . ., then the physical quantity aR+bS +· · · has the Hermitian operator aR+bS +· · ·, whether or not R, S, . . . are simultaneously measurable: 9 aR + bS + · · · ←→ aR + bS + · · · . (3) Assumption II is the obvious way to define aR+bS +· · · for sums of physical quantities Von Neumann's statement of Assumption II is only for the special case a = b = 1. But Assumptions I and A ′ tell us that aR is the Hermitian operator associated with the physical quantity aR, which leads directly to the more general form we give here.
10 Von Neumann's derivation of the density matrix form for the Exp-function, mentioned below, makes explicit use of Assumption II to construct such linear combinations, to which he then applies Assumption B ′ . 11 The reader interested only in where Bub and Dieks go astray can stop here. In what follows we expand on how von Neumann uses his four assumptions to arrive at his nohidden-variables theorem, and why his conclusions that hidden variables would undermine the fundamental principles of quantum mechanics are indeed not justified by his argument.

b. What von Neumann proves with his assumptions.
Von Neumann first proves 12 that if an ensemble of physical systems and the associated Exp function satisfy all four of his assumptions, then the Exp function for that ensemble must have the form where U is a non-negative 13 Hermitian operator characteristic of the ensemble but independent of the physical quantity R. In modern language there must be a density matrix U , such that the Exp function for the ensemble is the trace of the product of that density matrix with the Hermitian operator that corresponds to that physical quantity. 14 The Exp function characterizing a pure quantum state φ is indeed of the form (4) with the density matrix U given by |φ φ|. And, of course, the ensembles associated with ordinary quantum states do indeed satisfy all four of von Neumann's assumptions.
Von Neumann addresses the question of hidden variables on p. 323. 15 He asks whether the dispersion of any ensemble characterized by a wave function φ could result from the fact that such pure states are not the fundamental states, but only statistical mixtures of several more basic states. To specify such "actual states" one would need additional data -"hidden parameters", which we denote here collectively by λ. When adjoined to the quantum state φ these hidden parameters would determine everything -i.e. the resulting subensembles would be free of dispersion: for all physical quantities R. The statistics of the nondeterministic ensemble, characterized by (4) with U = U φ = |φ φ|, would result from appropriately weighted averages over 12 Pages 314-316, with some additional mopping up on pps. 316-320. 13 Von Neumann uses the term "definite". 14 Von Neumann's proof is quite straightforward. Three decades later Andrew Gleason proved what is now known as Gleason's Theorem: that the density matrix form (4) follows from premises essentially equivalent to Assumptions A ′ , I, and II. Gleason does not use assumption B ′ for physical quantities that are not jointly measurable. His argument is notoriously intricate.
15 He also brings up the question on pps. 209-210, but defers answering it, promising to show later that "an introduction of hidden parameters is certainly not possible without a basic change in the present theory." all the actual states, (φ, λ), into which the φ-ensemble was decomposed by the hidden parameters. 16 Von Neumann shows (again straightforwardly) that a φ ensemble cannot be so decomposed into dispersion-free (φ, λ) subensembles provided the Exp functions for the subensembles, Exp φ,λ , are also of the form (4) with density matrix U given by some U φ,λ . Therefore if the Exp functions for quantum states can be represented by weighted averages of Exp functions for dispersion-free subensembles, then some of those subensembles cannot have Exp functions of the form (4), and therefore some of von Neumann's four assumptions must fail for some of those subensembles.
Which assumptions might it be that fail for the dispersion-free subensembles?
c. Von Neumann nods. 17 Von Neumann clearly believes I and II to be the assumptions that must be abandoned if there are dispersion-free subensembles. When he states that "the established results of quantum mechanics can never be derived" (p. 324) if there are dispersion free subensembles," the reason he offers is that if they did exist, then "it [would be] impossible that the same physical quantities exist with the same function connections (i.e., that I and II hold)." That is indeed what I and II are about -functional relations among physical quantities, mediated by their corresponding Hermitian operators. Assumptions I and II, as noted above, make no mention of ensembles or statistical distributions. They specify broad structural relations, that it might be reasonable to expect to hold for physical quantities, regardless of what subensembles they might be measured in. 18 If indeed it was assumptions I and II that von Neumann expected to fail for the dispersion-free subensembles, then one can understand his now notorious "It is therefore not, as is often assumed, a question of a reinterpretation of quantum mechanics, -the 16 Putting it in terms of probability distributions p rather than ensembles, the question is whether p [φ] can be expressed as a weighted average of dispersion-free distributions, conditioned not just on the state φ, but also on the additional parameters λ.
17 "Homer nods.": Even the best of us sometimes slip up. From John Dryden's translation of line 359 of Horace's Ars Poetica: indignor quandoque bonus dormitat Homerus.
18 When von Neumann adds "Nor would it help if there existed other, as yet undiscovered, physical quantities, in addition to those represented by the operators in quantum mechanics, because the relations assumed by quantum mechanics (i.e., I, II) would have to fail already for the by-now known quantities," he underlines that he is blaming assumptions I and II. present system of quantum mechanics would have to be objectively false, in order that another description of the elementary processes than the statistical one be possible." (p. 325) So strong a conclusion might indeed be appropriate if assumptions I and II were the only suspects. But there are other suspects, A ′ and B ′ that von Neumann, unaccountably, fails to question. These have to do with the nature of physical quantities and the statistics of ensembles. They have nothing to do with "function connections" among physical quantities, or "relations assumed by quantum mechanics." Could assumptions A ′ or B ′ be sacrificed for the dispersion-free subensembles without making "the present system of quantum mechanics . . . objectively false"?
It might indeed be radical to abandon for subensembles the idea, A ′ , that single physical quantities and simultaneously measurable sets give rise to statistics that do not depend on the particular way in which they are measured. One could argue whether that would be more or less radical than abandoning I and II for the subensembles. But why bother to argue? Why not simply give up assumption B ′ for linear combinations of physical quantities that are not simultaneously measurable?
It is a peculiar feature of ordinary quantum mechanics that Assumption B ′ holds for the mean values over the φ-ensembles specified by quantum states, even when the physical quantities cannot be jointly measured. But there is no compelling reason to expect that B ′ should continue to hold for averages over the (φ, λ)-subensembles into which the φensembles might be subdivided by specifying additional hidden variables.
Bub and Dieks pass over B ′ , as a candidate for the assumption that fails for the dispersion-free subensembles, because they insist on interpreting it as nothing more than a definition. Dieks says that it would make no sense to reject B ′ for those subensembles because it is "analytic". But as emphasized above, Assumption II provides a powerful alternative way to define linear combinations of physical quantities that are not jointly measurable. In terms of that definition it is not only meaningful to reject B ′ for the hypothetical dispersion-free subensembles, but quite compatible with the general structure of ordinary quantum mechanics. Thanks to Hermann and Bell, Bub and Dieks are aware that they need a reason for not blaming B ′ . Von Neumann, who was unable to benefit from Bell's later criticisms 19 seems just to have overlooked the possibility. Homer not only nodded. He seems to have been fast asleep. Bell's describing his oversight as "silly" in a 19 It would be interesting to know if he ever became aware of Hermann's. magazine interview does not strike us as excessive. 20 There is no reason at all to require the Exp functions on possible dispersion-free subensembles to be linear on physical quantities that are not simultaneously measurable.
Maintaining "the established results of quantum mechanics" only requires B ′ to hold when those subensembles are recombined to make up the φ-ensemble characterizing the full quantum state φ. This is precisely the point made by John Bell fifty years ago, and, thirty years before Bell, by Grete Hermann.

Bell's criticism of von Neumann.
The most important part of Bell, 1966, is his better version of von Neumann's attempt at a no-hidden-variables theorem. Bell restricts von Neumann's assumption B ′ to physical quantities R, S, . . . that can be simultaneously measured. The linear combination W = aR + bS + · · · can then be measured by jointly measuring R, S, . . . and forming the corresponding linear combination of those measurement outcomes. With a a more elaborate argument, quite different from von Neumann's, Bell can still rule out dispersion-free subensembles, provided the Hilbert space has three or more dimensions. 21 To explain the point of his own refinement of von Neumann, Bell must explain the problem with von Neumann's then widely accepted result. He does this rather informally, condensing von Neumann's four assumptions into "Any real linear combination of any two Hermitian operators represents an observable, and the same linear combination of expectation values is the expectation value of the combination." This overly brisk summary 22 insufficiently emphasizes von Neumann's distinction between physical quantities and Hermitian operators. 23 It underemphasizes the importance of the mapping being 1-to-1. It does not distinguish between assumptions that refer to the statistical Exp functions and assumptions that do not. Nevertheless, this rough summary 20 See Mermin, 1993. 21 Bell then criticizes his own no-hidden-variables argument by challenging the implicit assumption that the result of measuring R should not depend on what other jointly measurable physical quantities R is measured with. But that's another story. 22 Bell does mention by name assumptions B ′ , I, and II in a footnote, but says nothing about their separate content. 23 Bell does make the distinction in an earlier introductory section, but unlike von Neumann he does not repeatedly insist on it. His use of "observable" to mean "physical quantity" is unfortunate, since by 1966 most physicists used the term for both physical quantities and Hermitian operators. 24 Bell often fails to distinguish between "noncommuting operators" and "not jointly measurable physical quantities". We show in Appendix 1 that the association of noncommuting operators with physical quantities that are not jointly measurable does indeed follow from assumptions A ′ , I, and II. 25 Bell mentions the nonadditivity of eigenvalues of non-commuting operators not because he thought von Neumann had overlooked this, but because it helps him explain the "nontriviality of the additivity of expectation values", which von Neumann, unaccountably, takes for granted in the dispersion-free subensembles. Similarly, when Bell mentions that "a measurement of a sum of noncommuting observables cannot be made by combining trivially the results of separate observations on the two terms" because "it requires a quite distinct experiment," he is not suggesting that von Neumann, who returns to this point repeatedly, was unaware of it. 26 The part of Hermann, 1935, that we address here is the short (small print!) Section 7, "The circle in von Neumann's proof", pps. 251-253. Hermann calls von Neumann "Neumann". We have restored his "von" for the sake of uniformity.

is enough to make clear what Bell objects to in von Neumann's assumptions, and this is
27 Perhaps in 1935 the distinctions von Neumann relied on had not yet been absorbed into a terminology that obscured important distinctions. von Neumann's oversight, she offers him some escape hatches that we cannot make much sense of. 28 Hermann considers an ensemble of physical systems. There are physical quantities R and S that can be measured on the systems of the ensemble. There is a function Exp(R) that gives the mean value of the measurement outcomes arising from an R-measurement on all the systems of the ensemble. "Von Neumann assumes that Exp(R + S) = Exp(R) + Exp(S). This crucial assumption is equivalent to von Neumann's B ′ . It is trivial, Hermann notes, for classical physics, and for quantum mechanical quantities that can be simultaneously measured, because then "the value of their sum is nothing other than the sum of the values that each of them separately takes, from which follows immediately the same relation for the mean values of these magnitudes. The relation is, however, not self-evident for quantum mechanical quantities between which uncertainty relations hold, and in fact for the reason that the sum of two such quantities is not immediately defined at all: since a sharp measurement of one of them excludes that of the other, so that the two quantities cannot simultaneously assume sharp values, the usual definition of the sum of two quantities is not applicable. Only by the detour over certain mathematical operators assigned to these quantities does the formalism introduce the concept of a sum also for such quantities." Hermann is saying here that because it is not clear how to define the sum in (6) or in Assumption B ′ of two quantities that are not jointly measurable, "to introduce the concept of a sum. . .for such quantities" requires a detour involving mathematical operators assigned to them -i.e. von Neumann's Assumptions I and II. By emphasizing the need for a detour into I and II she underlines that it is not necessary to take B ′ to define the sum of quantities that are not simultaneously measurable. Hermann is reading von Neumann just as we do. 29 28 We conjecture that she may have found von Neumann's blatant oversight so surprising that she tried, unsuccessfully, to guess what else he may have had in mind. It is these final remarks of hers that lead Dieks to state that her views are closer than Bell's to von Neumann.
29 Less anachronistically, we are reading von Neumann just as she does.
For an ensemble characterized by a wave-function φ, Hermann notes, and therefore (6) is valid by virtue of the quantum mechanical identity Here R and S are, she notes, "mathematical operators assigned to the quantities R and S." Since (8) holds whether or not R and S commute, (6) holds whether or not R and S are simultaneously measurable. 30 So B ′ does hold for ensembles characterized by wave functions.
But what about subsets of those ensembles "selected from them on the basis of any new features." For those subensembles "one can no longer infer from the asserted addition rule for (Rφ, φ), that also in these subsets the expectation value of the sum of physical quantities is the same as the sum of their expectation values. In this way, however, an essential step in Neumanns proof is missing." There it is: precisely the same problem that we describe in Section 2 and that Bell identified thirty years after Hermann.
We wish Hermann had stopped here. But she goes on. It is our guess that she goes on because she knows that this obvious problem did not stop von Neumann. What can he have been thinking? At this point we cannot paraphrase her account, because we can no longer follow it. We attach it as Appendix 2, in the hope that the reader may understand her better than we have done.
Setting aside what we take to be Hermann's efforts to find the motivation behind von Neumann's oversight, she has, in fact, read von Neumann more closely than Bell. She has the whole story. Once again, the culprit is Assumption B ′ . The only real difference between the reading we and Bell give and hers, is that she considers the possibility that von Neumann himself was aware of the obvious problem, and implicitly limited himself to subensembles for which the difficulty did not arise. But if he did that, then he had committed himself to the view that the hidden variables single out only those subensembles that lack features which make them any different from the larger φ-ensembles that they combine to give. So even if he did know what he was doing, he was begging the question.

Acknowledgment
We would like to thank Ulrich Mohrhoff for bringing Dieks, 2017, to our attention.

Appendix 1
Here are two examples of the power of von Neumann's Assumptions A ′ , I, and II.
The problematic assumption B ′ is not used; sums of physical quantities that are not jointly measurable are defined by sums of the corresponding Hermitian operators, using Assumption II. See also von Neumann, 1932, andPark andMargenau, 1968.
Theorem: The result of measuring a physical quantity must lie in the spectrum of the corresponding Hermitian operator.
Let R be the Hermitian operator corresponding to the physical quantity R, and consider a function f (x) that is 0 if x belongs to the spectrum of R, and 1 elsewhere. This means that f (R) = 0, the zero operator. So Assumption I requires that f (R) = 0, the physical quantity that is always 0. Thus, for every result r of measuring R, we have f (r) = 0, which means that r belongs to the spectrum of R.
Theorem: The correspondence between physical quantities and Hermitian operators must associate jointly measurable quantities with commuting operators and vice-versa.
If two Hermitian operators R and S commute, then it is a mathematical fact that there is a third Hermitian operator T of which they are both functions: By Assumption I the correspondence preserves functional relations, so Using Assumption A ′ , one can then simultaneously measure R and S by measuring T and applying to the outcome of that measurement the functions f and g.
The converse is trickier. 31 Let the physical quantities R and S be jointly measurable, and let R and S be the corresponding Hermitian operators. By Assumption A ′ , functions of 31 Unlike the proofs in von Neumann, 1932, or Park andMargenau, 1968, the elementary, but somewhat complicated, proof that follows makes no use of the spectral decomposition theorem for Hermitian operators.
such jointly measurable quantities can be measured by measuring the individual quantities R and S and evaluating the function at the individual outcomes r and s. So products of two jointly measurable physical quantities, that differ only in the order in which the quantities appear, can all be measured by the same experiment and have the same measurement outcomes. Such physical quantities are therefore identical. Since the correspondence is 1to-1, they must therefore all correspond to the same Hermitian operator. We show below that this leads to identities among the operators R and S which we can exploit to show that R and S must commute.
On the other hand, directly squaring both sides of (11) gives Since S(R(RS)), R(S(SR)), and (RS) 2 are all the same physical quantity, the sum of the right sides of (13) and (14) must be the same operator as twice the right side of (15). This gives us R 2 S 2 + S 2 R 2 = (RS)(SR) + (SR)(RS).
We also have, as a direct application of (11) to the pair R 2 and S 2 , and therefore, in view of (16), Since the operators on the right sides of (18) and (15) must be the same, we have and therefore This requires the square of the Hermitian operator C = i(RS − SR) to vanish, which in turn requires C itself to vanish. So R and S must indeed commute.

Appendix 2
We reproduce below the final paragraph and a half of the section on von Neumann in Hermann, 1935. The footnotes are our own comments. We have a sense of what Hermann is trying to say in the first half-paragraph below and the first half of the last paragraph.
We have no useful comments on the final part of the final paragraph. We would guess that she is struggling, unsuccessfully, to guess the kind of thinking that led to von Neumann's surprising oversight.
In this way, however, an essential step in von Neumann's proof is missing. 32 If instead -like von Neumann -one does not give up on this step, 33 then one has implicitly absorbed into the interpretation the unproven assumption that there can be no distinguishing features, of the elements of an ensemble of physical systems characterized by φ, on which the result of the R-measurement depends. 34 32 The step is being able to apply B ′ to the (φ, λ)-subensembles extracted from a φensemble by the value λ of the hidden variables. 33 Here she notes that von Neumann does take the step. 34 She is saying that since he does take the step he must be implicitly assuming that the (φ, λ)-subensembles have no features to distinguish them from the original φ-ensemble, for which B ′ is valid.
However, the impossibility of such features is precisely the claim to be proven.
Thus the proof runs in a circle. 35 On the other hand, from the standpoint of von Neumann's calculus one can argue against this, that it is an axiomatic requirement that all physical quantities are uniquely associated with certain Hermitian operators in a Hilbert space, and that through the discovery of new features invalidating the present limits of predictability, this association would inevitably be broken. 36 Indeed, any discovery that is representable in the operator calculus would have its contents specified only through the form of a wave function, which for quantities not simultaneously measurable exhibits the smearing out required by the uncertainty relations, and which finds application only by way of the probability interpretation. By this consideration, however, the crucial physical question of whether the progress of physical research can attain more precise predictions than are possible today, cannot be twisted into the impossibly equivalent mathematical question of whether such a development would be representable solely in terms of the quantum mechanical operator calculus. There would need to be a compelling physical reason, if not only the physical data known to date, but also all the results of research still to be expected in the future are related to each other according to the axioms of this formalism. But how should one find such a reason? The fact that the formalism has so far proven itself, so that one is justified in seeing in it the appropriate mathematical description of known natural connections, does not mean that the as yet undiscovered natural law connections should also have the same mathematical structure.