1 Introduction

By 1925, thanks to the work of the quantum pioneers, enough evidence had accumulated to accept the quantization of physical variables such as the atomic energies or the angular momentum, but the (classical) kinematic content of the old quantum theory seemed to fail. This caught the attention of Heisenberg, who, with great insight, set out to develop a description of quantum systems in which the dynamical variables were replaced by matrices [1]. Subsequent developments consolidated the operator formalism and extended it with great success to the radiation field. However, the underlying physical reason for the substitution—both in quantum theory and in quantum optics—of classical variables in phase space by operators in a Hilbert space has remained as obscure as it was to the founders of quantum theory.

In recent work [2, 3] we have analyzed the process that takes a system composed typically of a charged subatomic particle coupled to an external potential plus the zero-point radiation field, zpf, from its initially classical deterministic behavior to its final quantum behavior. As a result of this process, the canonical particle variables x(t), p(t) turn into (dipolar) response functions to a specific set of field modes. The respective response coefficients are the matrix elements of \(\hat{x}\) and \(\hat{p}\), satisfying the basic commutator \([\hat{x},\hat{p}]=i\hbar\). The variables involved in the (apparently) mechanical description provided by the Heisenberg formalism thus cease to be continuous (phase-space) variables to become the response functions to relevant field modes. This endows the theory with a clear physical rationale for the function\(\rightarrow\)operator transition.

A natural—and necessary step for coherence—is to investigate the concomitant process of quantization of the field. Having shown that matter has become quantized as a result of its interaction with the field, we propose to analyze the consequences of this interaction on the field itself. For this purpose we follow a procedure that builds on the preceding experience but now applied to those field components that interact with matter.Footnote 1 The result that the respective field variables become quantized would hardly surprise anyone; it is the outright novelty of the path leading to quantization that warrants attention. How does field quantization come about, and what do we learn along the way? What conclusions can be drawn, if any, about the quantum nature of the radiation field more generally? This is the main subject of the present work.

In preparation for the main topic, Sect. 2 contains a brief description of the process that leads to the quantization of matter and the emergence of the operator formalism with emphasis on the role of the zpf in this process. The central part of the paper deals with the quantization of the canonical field variables, starting with a description of an electromagnetic field component interacting with quantized matter. A procedure analogous to that leading to the particle operators is followed to obtain the field creation and annihilation operators \(\hat{a}^{\dagger }\), \(\hat{a}\) and the commutator \([\hat{a},\hat{a}^{\dagger }]=1\). The main implications of these results are briefly discussed in the final part of the paper.

2 Matter quantization: why operators instead of functions

Heisenberg was on the right track when he assumed that the structure of the quantum laws corresponds to that of classical physics. In both cases the Hamiltonian of the system determines the time evolution of any dynamical variable. The correspondence between the quantum commutator and the respective Poisson bracket, subsequently established by Dirac, has become commonplace and has led to the usual practice of putting a caret on the (classical) dynamical variables as a recipe for quantization. But this seemingly innocent (though actually radical) change in the mathematics entails a no less radical change in physical meaning, which has remained largely unaddressed. The question, therefore, is inevitable: what is the physics behind the transformation of continuous phase-space functions into operators, of dynamical variables into matrices?

We have set out to take the question thoughtfully and address it with the tools provided by stochastic electrodynamics, sed. From previous work on sed [5], we recall that the description of a typical quantum-mechanical system is completed by considering that it is composed not only of the material part but also of the radiation field interacting with it. By default, this field includes its ground-state component, the zpf. From a careful analysis of the dynamics of the compound system, it has become clear that the central effect of this interaction on matter is precisely to bring it to the quantum regime. In the following we will briefly review how the transition to this new regime drastically changes the nature of the dynamical variables used in the description, resulting in the replacement of the classical phase-space variables x, p by the corresponding operators \(\hat{x}\), \(\hat{p}\). As expected, although there is a formal one-to-one correspondence between the variables and the operators, they differ substantially in their physical meaning.

2.1 Kinematics of the SED system

We consider a particle bound by an external conservative force (typically an atomic electron) that gets connected to the zpf at some instant \(t_{o}\). For \(t<t_{o}\), the motion of the particle is classical; when it begins to interact with the field, the motion becomes complicated by the effect of stochasticity. As is shown in [2, 3], the solution of the particle’s equation of motion consists of two terms: a transient one, which as a result of the radiation reaction decays over a time \(\tau _{d}\approx 10^{-11}\) s for a system of the size of an atom, and a stationary one, which is purely stochastic and persists due to the permanent action of the zpf. The particle thus ends up in a stationary state of motion driven by the electric component of the zpf; it has entered the quantum regime.

Because the system as a whole is Hamiltonian, the entire set of canonical variables at time t (a semicolon is used for the set of canonical variables to avoid confusion with the Poisson bracket)

$$\begin{aligned} (q;p)=(x_{i},q_{\alpha };p_{i},p_{\alpha }) \end{aligned}$$
(1)

is related to the set of canonical variables at time \(t_{o}\) (or any other time),

$$\begin{aligned} (q_{o};p_{o})=(x_{io},q_{\alpha o};p_{io},p_{\alpha o}), \end{aligned}$$
(2)

via a canonical transformation. The full set of canonical variables includes also those of the (in principle infinite number of) field modes, with \(q_{\alpha }\), \(p_{\alpha }\) the (random) canonical variables corresponding to the mode of the field of frequency \(\omega _{\alpha }\). A discrete set of frequencies is considered here for reasons that will become clear later.

To analyze the evolution of the kinematics of the particle’s response to the field, we consider the Poisson bracket of the particle’s canonical variables at time t,

$$\begin{aligned} \{x_{i}(t),p_{j}(t)\}_{qp}=\delta _{ij}, \end{aligned}$$
(3)

with \(i,j=1,2,3\). In this and the following equations the subindex implies summing the Poisson brackets calculated with respect to all relevant variables. Since the sets (1) and (2) are related by a canonical transformation, and Poisson brackets are invariant under such transformations, the Poisson bracket (3) can be taken indistinctly with respect to the canonical variables at time t or at time \(t_{o}\),

$$\begin{aligned} \{ x_{i},p_{j}\} _{qp}&=\{ x_{i},p_{j}\} _{q_{o}p_{o}}\nonumber \\&=\{ x_{i},p_{j}\} _{x_{o}p_{o}}+\{x_i, p_{j}\} _{q_{\alpha o}p_{\alpha o}} \end{aligned}$$
(4)

whence from Eq. (3) we obtain

$$\begin{aligned} \{x_{i}(t),p_{j}(t)\}_{x_{o}p_{o}}+\{x_{i}(t), p_{j}(t)\}_{q_{\alpha o}p_{\alpha o}}=\delta _{ij}. \end{aligned}$$
(5)

As mentioned earlier, the transient solution decays over a time of the order of \(\tau _{d}\), due to the radiation reaction, so that the dependency of \(x_{i}(t),p_{j}(t)\) on the initial conditions \(\textbf{x}_{o}\), \(\textbf{p}_{o}\), disappears. Therefore the first term in Eq. (5), the only one carrying the particle’s initial conditions, vanishes and we are left with

$$\begin{aligned} \{x_{i}(t),p_{j}(t)\}_{q_{\alpha o}p_{\alpha o}} =\delta _{ij}\qquad (t>\tau _{d}). \end{aligned}$$
(6)

Note that the Poisson bracket now refers to the stationary part of the solution, which contains the response of the particle variables to the set of field modes \(\{\alpha \}\). In other words, the field has taken control of the kinematics of the particle.

Using the rules of transformation from canonical to normal field variables (i. e., the usual random dimensionless variables such that \(|a_{\alpha }|^{2}=1\)),

$$\begin{aligned} \omega _{\alpha }q_{\alpha }^{o}&=\sqrt{\frac{\hbar \omega _{\alpha }}{2}} (a_{\alpha }+a_{\alpha }^{*}),\nonumber \\ p_{\alpha }^{o}&=-i\sqrt{\frac{\hbar \omega _{\alpha }}{2}} (a_{\alpha }-a_{\alpha }^{*}), \end{aligned}$$
(7)

as corresponds to the zpf, the Poisson bracket \(\{x_{i},p_{j}\} _{q_{\alpha o}p_{\alpha o}}\) transforms into

$$\begin{aligned} \{ x_{i},p_{j}\} _{q_{\alpha o}p_{\alpha o}} =\frac{1}{i\hbar }[x_{i},p_{j}], \end{aligned}$$
(8)

where the bilinear form \([x_{i},p_{j}]\) stands for the transformed Poisson bracket,

$$\begin{aligned} {[}x_{i},p_{j}]\equiv \{x_{i},p_{j}\}_{aa^{*}}=\sum _{\alpha } \left( \frac{\partial x_{i}}{\partial a_{\alpha }} \frac{\partial p_{j}}{\partial a_{\alpha }^{*}} -\frac{\partial p_{j}}{\partial a_{\alpha }} \frac{\partial x_{i}}{\partial a_{\alpha }^{*}}\right) . \end{aligned}$$
(9)

According to Eq. (6), for times \(t>\tau _{d}\) this bilinear form must satisfy the condition

$$\begin{aligned} {[}x_{i},p_{j}]=i\hbar \delta _{ij}. \end{aligned}$$
(10)

2.2 From dynamical variables to operators

Given its universal character, Eq. (10) applies to the particle in any stationary state n. We therefore tag the variables \(x_{i}\), \(p_{j}\) with the subindex n and write, using (10) (from now on, we limit the discussion to 1D for simplicity)

$$\begin{gathered} \left[ x,p\right] _{nn}=\sum _{\alpha } \left( \frac{\partial x_{n}}{\partial a_{\alpha }} \frac{\partial p_{n}}{\partial a_{\alpha }^{*}} -\frac{\partial p_{n}}{\partial a_{\alpha }} \frac{\partial x_{n}}{\partial a_{\alpha }^{*}}\right) =i\hbar . \end{gathered}$$
(11)

This implies that the variables \(x_{n}\) and \(p_{n}=m\dot{x}_{n}\) are linear functions of the normal field variables \((a_{\alpha },a_{\alpha }^{*})\), corresponding to the field modes to which the particle responds resonantly [4]. We therefore write

$$\begin{aligned} x_{n}(t)&=\sum _{l}x_{ln}a_{ln}e^{-i\omega _{ln}t}+\mathrm{c.c.},\nonumber \\ {p}_{n}(t)&=\sum _{l}p_{ln}a_{ln}e^{-i\omega _{ln}t}+\mathrm{c.c.}, \end{aligned}$$
(12)

where \(a_{ln}\) is the normal variable associated with the field mode that connects state n with any other accessible state l, and \(x_{ln}\), \(p_{ln}=-im\omega _{ln}x_{ln}\) are the respective response coefficients, and introduce these expressions into Eq. (11), thus obtaining

$$\begin{aligned} {[}x,p]_{nn}=2im\sum _{l}\omega _{ln}|x_{ln}|^{2}=i\hbar . \end{aligned}$$
(13)

Since the \(a_{nl}\), \(a_{ml}\) (\(n\ne m\)) are independent random variables, Eq. (11) generalizes to

$$\begin{aligned} {[}x,p]_{nm}=i\hbar \delta _{nm}. \end{aligned}$$
(14)

The \(x_{ln}\) and \(a_{ln}\) refer to the transition \(n\rightarrow l\) involving the frequency \(\omega _{ln}\), while \(x_{nl}\) and \(a_{nl}\) refer to the inverse transition with \(\omega _{nl}=-\omega _{ln}\); therefore, from (12), \(x_{nl}^{*}(\omega _{nl})=x_{ln}(\omega _{ln}),\ p_{nl}^{*}(\omega _{nl})=p_{ln}(\omega _{ln}),\ a_{nl}^{*}(\omega _{nl})=a_{ln}(\omega _{ln}),\) and (14) takes the form

$$\begin{aligned} \sum _{l}\left( x_{nl}p_{lm}-p_{nl}x_{lm}\right) =i\hbar \delta _{nm}, \end{aligned}$$
(15)

which is precisely Heisenberg’s quantization rule. The \(x_{nl}\) and \(p_{nl}\) in this equation are elements of a couple of matrices \(\hat{x}\) and \(\hat{p}\), respectively (as was observed by Born), with as many rows and columns as there are different states,

$$\begin{aligned} {[}\hat{x},\hat{p}]_{nm}=i\hbar \delta _{nm}, \end{aligned}$$
(16)

or in terms of the respective operators,

$$\begin{aligned} {[}\hat{x},\hat{p}]=i\hbar . \end{aligned}$$
(17)

The relationship between Poisson brackets and commutators established by Dirac in more general terms, thus finds a plausible physical explanation as a kinematic transformation that gives a new meaning to the quantities involved in the description. The canonical qm commutator is no longer a postulate, it is an expression of the particle’s response to the field once a stationary state has been reached. Seen as a component of \(\hat{x}\) acting on the particle in state n, \(x_{ln}\) represents the amplitude (or coefficient) of the linear, resonant response of the particle to the field mode (ln) that takes it to state l, fully in line with the quantum description and as expressed in the selection rules for dipole transitions, which, as is well known, depend directly on the matrix coefficients \(x_{ln}\).

Once x and p become operators, all dynamical variables become operators acting on the states, which are represented by column matrices. It is clear that \(\hat{x}\), \(\hat{p}\) do not describe particle trajectories, so no phase-space description is associated with them. The Hilbert-space formalism provides thus a compact description based on the elements connecting the stationary quantum states, rather than a description of the states themselves. It should be noted that the (joint) appearance of i and \(\hbar\) results from the fact that in the quantum regime, the description refers to the response of the mechanical system to the field variables. Although the zpf disappears from the picture, it leaves Planck’s constant as an indelible mark everywhere; equally important, the imaginary unit i enters qm at a fundamental level.

3 Field quantization: why operators instead of functions

In his celebrated 1905 article on the photoelectric effect, Einstein stated that radiation within the domain of Wien’s formula behaves as if it were composed of mutually independent quanta of energy \(\hbar \omega\). In 1916 he further demonstrated that a momentum \(\hbar \omega /c\) is transferred to a molecule when it emits or absorbs a quantum of light [6, 7]. Thereafter, Dirac claimed that the canonical field variables had to satisfy the canonical commutation relationFootnote 2 for radiation to have quantum properties [8]. This statement gave birth to the field’s ladder operators and has spread to become the cornerstone of all quantum field theory. The recipe is simple and works great: canonical quantization is achieved by associating with each pair of classical canonically conjugate variables, two operators with commutator \(i\hbar\) (see, e.g., [9,10,11]). However, as mentioned above, the physical reason behind the recipe remains obscure.

3.1 Description of a field component in interaction with matter

In the following we develop a description of the radiation field interacting with matter consistent with the results presented above. The fact that quantized matter, as shown above, interacts resonantly with the radiation field of a set of well-defined frequencies \(\{ \omega _{nl}\}\) allows us to focus on a field component of one of these frequencies, which we shall call simply \(\omega\), and describe it in terms of its canonical variables. This description serves to express the vector potential \(\textbf{A}(\textbf{x},t)\) (and any function derived from it) of the field in interaction with matter, which may be the zpf alone or in combination with an external radiation field.

We consider a component of given wave vector \(\textbf{k}\) and polarization \(\varvec{\varepsilon }\), and focus on its (scalar) coefficient. In the usual dipole (long-wavelength) approximation, the dependence on \(\textbf{x}\) is neglected, so we have a simple harmonic oscillator of frequency \(\omega\) being associated with each pair of vectors \((\textbf{k},\varvec{\varepsilon })\).

Let \({q}_{n}(t)\), \({p}_{n}(t)\) be the canonical variables that describe this field when in state n; we write them as linear combinations of the same normal field variables \((a_{\alpha },a_{\alpha }^{*})\) used in the previous section, which we denote here with the indices \({n},{n}'\) that connect state n with state \({n'}\), i. e., \(a_{{nn}'},a_{{nn}'}^{*}\), with respective coefficients \({q}_{{nn}'}\), \({p}_{{nn}'}\):

$$\begin{aligned} {q}_{{n}}(t)&=\sum _{{n}'}{q}_{{nn}'}a_{n{n'}} e^{-i\omega _{{n'n}}t}+\mathrm{c.c.},\nonumber \\ {p}_{{n}}(t)&=\sum _{{n'}}{p}_{{nn}'}a_{{nn}'} e^{-i\omega _{{n'n}}t}+\mathrm{c.c.}, \end{aligned}$$
(18)

where \({p}_{{nn}'}=-i\omega _{{n'n}}{q}_{{nn}'}\).

The fact that this field component corresponds to a single frequency \(\omega\) means that \(|\omega _{{n'n}}|=\omega\), i.e., \(\omega _{{n'n}}=\pm \omega\). Consequently, only two of the coefficients \({q}_{{nn}'}\) connecting state n with state \(n'\) are different from zero. Since there are no intermediate states, we identify the immediately upper state of the field (corresponding to \(\omega _{{n'n}}=\omega\)) with \(n'=n+1\) and the immediately lower state (corresponding to \(\omega _{{n'n}}=-\omega\)) with state \({n}'=n-1\), so that Eq. (18) become

$$\begin{aligned} {q}_{n}(t)&={q}_{{nn}+1}a_{{nn}+1}e^{-i\omega t} +{q}_{nn-1}{a}_{nn-1}e^{i\omega t}+\mathrm{c.c.}, \end{aligned}$$
(19)
$$\begin{aligned} {p}_{{n}}(t)&=-i\omega {q}_{{nn}+1}a_{{nn}+1} e^{-i\omega t}\\ & \quad +i\omega {q}_{{nn-1}}{a}_{{nn}-1} e^{i\omega t}+\mathrm{c.c.}, \end{aligned}$$
(20)

and

$$\begin{aligned}&\omega {q}_{{nn}+1}-i{p}_{{nn}+1}=0,\nonumber \\&\omega {q}_{{nn}-1}+i{p}_{{nn}-1}=0. \end{aligned}$$
(21)

By analogy with the results of the previous section, we identify \({q}_{{nn}+1}\), \({q}_{{nn}-1}\) with the response coefficients that determine the change of state of the field component of frequency \(\omega\) in interaction with matter from n to \(n+1\) and \(n-1\), respectively.

3.2 The field operators

According to Eqs. (19) and (20), in the quantum regime the Poisson bracket of the canonical variables of the field component of frequency \(\omega\) is taken only with respect to the couple of field quadratures of the elementary field modes having that frequency,

$$\begin{aligned} \{{q}_{n}(t),{p}_{n}(t)\} _{ q_{{nn}'}p_{{nn}'} }=1, \end{aligned}$$
(22)

with \(n'=n\pm 1\). With the transformation from quadratures q, p to normal field variables a, \(a^{*}\), Eqs. (7), this becomes

$$\begin{aligned} {[}{q}_{n}(t),{p}_{n}(t)]=i\hbar , \end{aligned}$$
(23)

which gives explicitly, using Eqs. (19) and (20),

$$\begin{aligned} {q}_{{nn}'}{p}^{*}_{{nn}'} -{p}_{{nn}'}{q}^{*}_{{nn}'}=i\hbar . \end{aligned}$$
(24)

From the same equations applied to n and to \(n'\), we note that \({q}^{*}_{{nn}'}(\omega _{{n'n}})={q}_{{n'n}} (\omega _{{nn'}})\), \({p}^{*}_{{nn}'}(\omega _{{n'n}})={p}_{{n' n}}(\omega _{{nn'}})\), and \(a_{{nn}'}^{*}(\omega _{{n'n}})=a_{{n'n}} (\omega _{{nn'}})\) so that Eq. (24) can be written in the alternative form

$$\begin{aligned} {q}_{{nn}'}{p}_{{n'n}}-{p}_{{nn}'}{q}_{{n'n}}=i\hbar . \end{aligned}$$
(25)

By identifying the coefficients \({q}_{{nn}'}\) and \(\textrm{p}_{{nn'}}\) as the elements of matrices \(\hat{{q}}\) and \(\hat{{p}}\), respectively, Eq. (25) becomes

$$\begin{aligned} {[}\hat{{q}},\hat{\textrm{p}}]=i\hbar , \end{aligned}$$
(26)

for any state n of the field.

Therefore, the field commutator is the Poisson bracket of the canonical variables \({q}_{n}(t)\), \({p}_{n}(t)\) of a field component of a given frequency with respect to the normal field variables \((a,a^{*})\) that can take this field to an upper or lower state. Since \({n'=n}\pm 1\), \(\hat{{q}}\) and \(\hat{{p}}\) have off diagonal elements immediately above and below the diagonal only. Further, because of (21), the normalized matrix \(\hat{a}\) and its adjoint, defined as

$$\begin{aligned} \hat{a}&=\frac{1}{\sqrt{2\hbar \omega }}(\omega \hat{{q}} +i\hat{{p}}),\nonumber \\ \hat{a}^{\dagger }&=\frac{1}{\sqrt{2\hbar \omega }}(\omega \hat{{q}} -i\hat{{p}}), \end{aligned}$$
(27)

have off diagonal elements immediately above and below the diagonal, meaning that they play the role of annihilation and creation operators, respectively. Equation (26) gives for their commutator

$$\begin{aligned} {[}\hat{a},\hat{a}^{\dagger }]=1. \end{aligned}$$
(28)

Since normal field variables a, \(a^{*}\) associated with different modes are statistically independent of each other, it is straightforward to generalize this equation to

$$\begin{aligned} {[}\hat{a}_{\textbf{k},\lambda },\hat{a}_{\textbf{k}',\lambda '}^{\dagger }] =\delta _{\textbf{k}\textbf{k}'}^{(3)}\delta _{\lambda \lambda '}, \end{aligned}$$
(29)

which is the cornerstone of the quantum theory of radiation. The quantum formalism is then completed by expressing the Hamiltonian operator for a single field component in terms of \(\hat{{p}}\) and \(\hat{{q}}\) (or \(\hat{a}\) and \(\hat{a}^{\dagger }\)),

$$\begin{aligned} \hat{H}_{rad}=\frac{1}{2}\left( \omega ^{2}\hat{{q}}^{2} +\hat{{p}}^{2}\right) =\frac{\hbar \omega }{2} \left( \hat{a}\hat{a}^{\dagger }+\hat{a}^{\dagger }\hat{a}\right) \end{aligned}$$
(30)

and incorporating the state vectors denoted by \(\vert n\rangle\) with \(\vert 0\rangle\) for the ground or vacuum state.

We must bear in mind, however, that Eq. (30), like all the previous ones, refers to a field component that exchanges energy as a result of its interaction with quantized matter, which means that it should be considered in the context of the full Hamiltonian (in the Coulomb gauge)

$$\begin{aligned} \hat{H}&=\frac{1}{2m}\left( \hat{\textbf{p}}-\frac{e}{c}\hat{\textbf{A}} (\hat{\textbf{x}})\right) ^{2}+\hat{V}(\hat{\textbf{x}}) \\ & \quad+\sum _{\alpha } \frac{\hbar \omega }{2}\left( \hat{a}_{\alpha }\hat{a}_{\alpha }^{\dagger } +\hat{a}_{\alpha }^{\dagger }\hat{a}_{\alpha }\right) , \end{aligned}$$
(31)

where \(\alpha\) has the same connotation as in Sect. 2: it denotes the field modes with which the particle can exchange energy and momentum, resulting in a change of state of both matter and field. Consequently, these results tell us nothing about the nature (quantum or non-quantum) of the free radiation field. Consider for example a beam of synchrotron radiation produced by highly accelerated orbiting electrons, which may have any (continuous) range of energies, in principle. When such radiation strikes a detector, a resonant response of the detector’s atoms to a well-defined frequency \(\omega\) may lead to the absorption of a corresponding energy \(\hbar \omega\); it is this (absorbed) energy that is quantized. Quantized matter and field form a single entity.

4 Concluding remarks

The image of an isolated atom or any quantum entity in empty space appears, from the point of view of sed, as an idealization with no counterpart in the real world. The inclusion of the zpf changes the (apparently) mechanical nature of quantum systems to an electrodynamic one. In addition to ensuring (and explaining) the atomic stability by compensating for the energy lost by radiation, sed has provided a possible explanation for a whole series of quantum phenomena, including entanglement, the electron spin and the (nonrelativistic) radiative corrections of qed [5]. Notably, in this paper we have shown that it also explains one of the most obscure quantum features: the substitution of dynamical variables by operators. Once in the quantum regime, the central elements of the description are the response amplitudes of the mechanical system to the field modes with which it interacts. In turn, the field variables contain the matrix elements associated with transitions involving a well-defined energy exchange with matter. Both the mechanical and the field commutators preserve their classical symplectic structure, and the respective operators lend themselves to a unified description of the composite system in the product Hilbert space.