1 Introduction

Much of the modern-era precision in hadron spectroscopy is not gleaned from scattering reactions, but rather from production or decay processes [1,2,3,4,5]. For two-body states, the universality of final-state interactions [6] provides an important and fundamental link between scattering and production amplitudes, guaranteeing their phases to be identical in the region where scattering is elastic. The presence of a third strongly interacting decay product complicates rigorous analyses considerably. An approximation to the decay amplitude where the two-particle interaction is modeled by a resonant amplitude without accounting for the interaction with the spectators is referred to as the isobar model. The lineshape of the resonant pairwise interaction is often parameterized by the Breit–Wigner function [7]. While one might expect that under certain circumstances, the influence of spectator particles on the two-body resonance signal ought to be small – the resonance in question being narrow, or the spectators having large relative momenta – the impact of the spectator interaction on the resonance lineshape has only rarely been quantified. The goal of this article is to start the endeavor to survey such more complicated final-state interactions numerically, beginning with the simplest processes: decays into three pions.

A tool to perform a theoretically rigorous evaluation of three-body decays is given by the so-called Khuri–Treiman (KT) dispersion relations [8]. These are coupled integral equations that describe all sequential pairwise two-body rescattering, summed to all orders. While the two-body phase shifts are assumed to be known, the solutions of these equations depend on a set of free parameters, subtraction constants of the dispersion integrals, which can be fixed by comparison to experimental data or by matching to effective field theories. Instead of analyzing data for a particular process, we pose the question: how much statistics needs to be collected in order for lineshape modifications due to a third pion to be discernible? We choose the processes to study based on the condition that their amplitude representations can be reduced to one single subtraction constant. This then serves as a mere normalization, and hence allows us to study subtle variations of the resonance lineshape in an unambiguous manner. In many three-pion decays, the \(\rho \) meson is the most prominent pion–pion resonance, and interactions of higher angular momentum are suppressed. An important criterion is therefore that we select decays in which S-waves are forbidden by conservation laws, reducing the problem to the P-wave interaction only. One process of significant interest disregarded here is the \(\tau \rightarrow 3\pi \nu _\tau \) decay, in which the \(a_1(1260)\) resonance with \(J^{PC}=1^{++}\) appears prominently; however, there are S-waves involved, whose strength relative to the P-waves cannot be fixed a priori without data (cf. a similar analysis in Ref. [9]).

The outline of this article is as follows. In Sect. 2, we introduce and discuss the KT equations and derive the amplitude decompositions of four different three-pion decays that are dominated by the \(\rho \pi \) intermediate states. Subsequently, in Sect. 3, we describe the statistical method used to distinguish the KT solutions from the simpler Omnès model that neglects spectator interactions entirely. Section 4 shows our numerical results and discusses the dependence of the crossed-channel rescattering effects on mass and quantum numbers of the three-pion system in detail. Our findings are summarized in Sect. 5. Some technical details are relegated to the appendices.

2 Khuri–Treiman equations

The Khuri–Treiman equations [8] were first derived in the 1960s to analyze \(K\rightarrow 3\pi \) decays. With the advent of very precise parameterizations of low-energy pion–pion (\(\pi \pi \)) phase shifts [10,11,12], the approach has experienced a remarkable renaissance and has been applied to various decays [13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28] and scattering processes [29,30,31,32,33] since.

We here consider four different \(3\pi \) decays with quantum numbers \(J^{PC} = 0^{--}\), \(1^{--}\), \(1^{-+}\), and \(2^{++}\), which fulfill the criterion introduced in Sect. 1. For all these, the unpolarized distribution over the three-body phase space as represented by the Dalitz plot contains the full information on the decay dynamics, as only one helicity amplitude contributes in every case. In order to apply the KT equations, we decompose each amplitude into the so-called single-variable amplitudes (SVAs), which are complex functions with a right-hand cut only. These decompositions are known as reconstruction theorems, proven in chiral perturbation theory in a give order using fixed-variable dispersion relations [34,35,36]. Thereby, we restrict ourselves to P-waves and neglect all higher partial waves.

2.1 Reconstruction theorem for \(1^{--}\) decay

We begin our discussion with the consideration of isoscalar vector quantum numbers, \(I^G(J^{PC})=0^-(1^{--})\), where the isospin I is forced to be zero by the negative G-parity of the odd-pion system and negative C-parity. Decays into three neutral pions are forbidden by charge conjugation. The decays \(\omega /\phi \rightarrow 3\pi \) have been studied extensively using the KT formalism [15, 16, 25, 26], as well as extended to the \(J/\psi \rightarrow 3\pi \) decays [37], and the general reactions \(e^+e^-\rightarrow 3\pi \) [38,39,40,41]. Experimentally, the Dalitz plots both for \(\omega \rightarrow 3\pi \) [42, 43] and \(\phi \rightarrow 3\pi \) [44, 45] have been investigated in detail.

The decay reads

$$\begin{aligned} V(p)&\rightarrow \pi ^0(p_1)\pi ^+(p_2)\pi ^-(p_3), \end{aligned}$$
(1)

where p and \(p_i\) denote the four momenta of the decay particle V and the pions, respectively. The Mandelstam variables [46] are defined as \(s=(p-p_1)^2\), \(t=(p-p_2)^2\), and \(u=(p-p_3)^2\). The amplitude \(\mathcal {M}\) is decomposed into a scalar amplitude \(\mathcal {F}\) and a kinematic factor in the following form:

$$\begin{aligned} \mathcal {M}(s,t,u)&=i\epsilon ^\mu K_\mu \mathcal {F}(s,t,u), \nonumber \\ K_\mu&=\varepsilon _{\mu \nu \alpha \beta }p_1^\nu p_2^\alpha p_3^\beta , \end{aligned}$$
(2)

where the Levi-Civita tensor is employed due to the odd intrinsic parity. By squaring the matrix element and averaging over the initial polarization, one obtains

$$\begin{aligned} \overline{|\mathcal {M}|^2}&=\mathcal {K}(s,t,u)|\mathcal {F}(s,t,u)|^2, \end{aligned}$$
(3)

where \(\mathcal {K}\) is a factor proportional to the Kibble function [47],

$$\begin{aligned} \mathcal {K}(s,t,u)&=\frac{1}{4}\left( stu-M_\pi ^2(M^2-M_\pi ^2)^2\right) . \end{aligned}$$
(4)

Here, M denotes the mass of the decay particle, and \(M_\pi \) refers to the pion mass in the isospin limit. Using the fixed-variable dispersion relations, one can show that the scalar amplitude is decomposed into the P-wave SVAs, \(\mathcal {F}(x)\) [15, 48,49,50]:

$$\begin{aligned} \mathcal {F}(s,t,u)=\mathcal {F}(s)+\mathcal {F}(t)+\mathcal {F}(u), \end{aligned}$$
(5)

where discontinuities in F- and higher partial waves have been neglected. The decay amplitude is invariant under a shift

$$\begin{aligned} \mathcal {F}(s) \rightarrow \mathcal {F}(s) + \alpha (s-s_0) , \end{aligned}$$
(6)

where \(3s_0 \equiv M^2+3M_\pi ^2 = s+t+u\), and \(\alpha \) is an arbitrary complex constant. This means that the decomposition (5) only defines \(\mathcal {F}(s)\) up to such a polynomial ambiguity. The strategy to eliminate the ambiguity is discussed in Sect. 2.5.

2.2 Reconstruction theorem for \(1^{-+}\) decay

Mesons with quantum numbers \(I^G(J^{PC})=1^-(1^{-+})\) are exotic in the quark model. The lightest candidate for such hybrid mesons is the \(\pi _1(1600)\), which has been searched for experimentally in different final states such as \(\eta \pi \) [51,52,53,54,55], \(\eta '\pi \) [52, 53, 55], and the most relevant for the present study \(\rho \pi \) [56].

The amplitude decomposition for the decays \(\pi _1\rightarrow 3\pi \) is very similar to the one for resonances with \(I^G(J^{PC})=0^-(1^{--})\) discussed in the previous section, however, the positive charge conjugation implies odd isospin. For the isovector decay, the explicit decomposition of the decay amplitude in terms of isospin indices is required,

$$\begin{aligned} X^i(p)&\rightarrow \pi ^j(p_1)\pi ^k(p_2)\pi ^l(p_3), \end{aligned}$$
(7)

where i, j, k, and l are the isospin indices in Cartesian basis. The decay is once more of odd intrinsic parity. The decomposition reads

$$\begin{aligned} \mathcal {M}^{ijkl}(s,t,u)=i\epsilon ^\mu K_\mu \mathcal {H}^{ijkl}(s,t,u), \end{aligned}$$
(8)

with the isospin amplitude \(\mathcal {H}^{ijkl}\) following from the well-known isospin relations for \(\pi \pi \) scattering [57]:

$$\begin{aligned} \mathcal {H}^{ijkl}(s,t,u)&=\delta ^{ij}\delta ^{kl}\mathcal {H}(s,t,u)+\delta ^{ik}\delta ^{jl}\mathcal {H}(t,u,s)\nonumber \\&\quad +\delta ^{il}\delta ^{jk}\mathcal {H}(u,s,t). \end{aligned}$$
(9)

The reconstruction theorem retaining P-waves only, i.e., neglecting discontinuities in D-waves and higher, reads

$$\begin{aligned} \mathcal {H}(s,t,u)=\mathcal {H}(t)-\mathcal {H}(u), \end{aligned}$$
(10)

derived in Appendix A.

The decomposition is ambiguous by shifting

$$\begin{aligned} \mathcal {H}(s)\rightarrow \mathcal {H}(s)+\alpha . \end{aligned}$$
(11)

This ambiguity in principle allows us to write down a twice-subtracted dispersion integral for \(\mathcal {H}(s)\) that still depends on one subtraction constant only. Since the subtraction constants of these two representations are connected by a sum rule, the amplitude \(\mathcal {H}(s,t,u)\) remains unchanged. The allowed charge configurations are

$$\begin{aligned} X^+&\rightarrow \pi ^+\pi ^0\pi ^0,\quad X^+\rightarrow \pi ^+\pi ^+\pi ^-,\quad X^0\rightarrow \pi ^+\pi ^-\pi ^0,\\ X^-&\rightarrow \pi ^-\pi ^0\pi ^0,\quad X^-\rightarrow \pi ^-\pi ^-\pi ^+, \end{aligned}$$

which all lead to the same result for the absolute squared of the amplitude.Footnote 1 Due to the analogy between Eqs. (2) and (8), the latter can be written in the same form as Eq. (3).

2.3 Reconstruction theorem for \(2^{++}\) decay

For isovector tensor mesons, \(I^G(J^{PC})=1^-(2^{++})\), the lightest state is the \(a_2(1320)\) that dominantly decays into \(3\pi \) [58]. Hence, we need to consider isospin explicitly,

$$\begin{aligned} T^i(p)&\rightarrow \pi ^j(p_1)\pi ^l(p_2)\pi ^k(p_3). \end{aligned}$$
(12)

Due to the high spin of the decaying particle, the complete amplitude decomposition is considerably more complicated than for the vector decays and involves different helicity amplitudes; this is discussed in Appendix B. The isospin decomposition of \(\mathcal {M}^{ijkl}(s,t,u)\) involves invariant isospin amplitudes, which can be defined with respect to different Mandelstam variables. These are related to each other by crossing symmetry; see Appendix B for details. The s-channel amplitude is given as

$$\begin{aligned} \mathcal {M}_s(s,t,u)&=i\sqrt{2}\epsilon _{\mu \nu }K^\mu \left[ (p_2+p_3)^\nu \mathcal {B}(s,t,u) \right. \nonumber \\&\left. \quad +(p_2-p_3)^\nu \mathcal {C}(s,t,u)\right] . \end{aligned}$$
(13)

The resulting spin-averaged squared amplitude is then given as

$$\begin{aligned} \overline{|\mathcal {M}_s|^2}&=\tilde{\mathcal {K}}_1(s,t,u)|\mathcal {B}(s,t,u)|^2\nonumber \\&\quad +2\tilde{\mathcal {K}}_2(s,t,u)\text {Re}\left( \mathcal {B}(s,t,u)\mathcal {C}(s,t,u)^*\right) \nonumber \\&\quad +\tilde{\mathcal {K}}_3(s,t,u)|\mathcal {C}(s,t,u)|^2, \end{aligned}$$
(14)

where

$$\begin{aligned} \tilde{\mathcal {K}}_i(s,t,u)&=\frac{\mathcal {K}(s,t,u)}{40M^2}k_i(s,t,u),\quad \forall i\in \{1,2,3\},\nonumber \\ k_1(s,t,u)&=\lambda (s,M^2,M_\pi ^2)\nonumber ,\\ k_2(s,t,u)&=(s+M^2-M_\pi ^2)(u-t)\nonumber ,\\ k_3(s,t,u)&=(t-u)^2+4M^2\left( s-4M_\pi ^2\right) , \end{aligned}$$
(15)

and \(\lambda (a,b,c)=a^2+b^2+c^2-2(ab+ac+bc)\) is the standard Källén function. The reconstruction theorem of the scalar functions has the form

$$\begin{aligned} \mathcal {B}(s,t,u)&=\mathcal {B}(t)-\mathcal {B}(u),\nonumber \\ \mathcal {C}(s,t,u)&=\mathcal {B}(t)+\mathcal {B}(u), \end{aligned}$$
(16)

neglecting discontinuities that lead to \(\pi \pi \) D-waves and higher. This decomposition is unambiguous.

2.4 Reconstruction theorem for \(0^{--}\) decay

The decays of the \(\eta ^{(\prime )}\) mesons, \(I^G(J^{PC})=0^+(0^{-+})\), into \(3\pi \) necessarily violate G-parity. In the Standard Model, where both the strong and the electromagnetic interactions preserve charge conjugation, the decays proceed via breaking of isospin symmetry, while more exotic scenarios of physics beyond the Standard Model involving C-parity violation are suggested for the \(\pi ^+\pi ^-\pi ^0\) final state [27, 59]. We here concentrate on the latter, with total three-pion isospin \(I=0\) and negative charge conjugation, which would be equally applicable for the three-pion decay of a quark-model-exotic resonance with \(I^G(J^{PC})=0^-(0^{--})\) in QCD. Such states have been predicted as hybrid mesons with precisely this decay channel [60], although first lattice-QCD calculations at unphysically high pion masses suggest them to appear at higher masses than the \(1^{-+}\) hybrids [61]; constituent-gluon models partly come to different conclusions [62]. The decay amplitude written in terms of P-waves only is given by

$$\begin{aligned} \mathcal {M}(s,t,u)=(t-u)\mathcal {G}(s)+(u-s)\mathcal {G}(t)+(s-t)\mathcal {G}(u). \nonumber \\ \end{aligned}$$
(17)

We note that due to the decaying particle being a (pseudo) scalar, there is no additional kinematic factor in the relation to the Dalitz-plot distribution. In contrast to the fully symmetric reconstruction theorem under pairwise exchange of Mandelstam variables for \(J^{PC}=1^{--}\), this one is fully antisymmetric.

The amplitude stays invariant under a three-parameter polynomial shift

$$\begin{aligned} \mathcal {G}(s)\rightarrow \mathcal {G}(s)+\alpha +\beta s+\gamma s^2(3s_0-s), \end{aligned}$$
(18)

where \(\alpha \), \(\beta \), and \(\gamma \) are arbitrary complex numbers. Similar to the discussion in Sect. 2.2, this ambiguity allows us to write \(\mathcal {G}(s)\) as a twice- or three-times-subtracted dispersion integral, depending on a single subtraction constant.

2.5 Partial-wave unitarity, Omnès solutions

The form of the partial-wave series deviates due to the different spins of the decaying particles. While the one for the pseudoscalar decay in Eq. (17) proceeds in terms of standard Legendre polynomials, the one for the vector decays has the form [63]

$$\begin{aligned} \mathcal {F}(s,t,u)=\sum _{\ell =1}P_\ell '(z_s)f_\ell (s) , \end{aligned}$$
(19)

and similarly for \(\mathcal {H}(s,t,u)\), where \(P_\ell '(z_s)\) refers to the derivatives of the Legendre polynomials. The cosine of the s-channel scattering angle, denoted by \(z_s\), can be expressed via the Mandelstam variables

$$\begin{aligned} z_s=\frac{t-u}{\kappa (s)}, \quad \kappa (s)=\sqrt{1-\frac{4M_\pi ^2}{s}}\lambda ^{1/2}(s,M^2,M_\pi ^2)\,. \end{aligned}$$
(20)

In the s-channel center-of-mass system, t and u are related to the scattering angle via

$$\begin{aligned} t(s,z_s)&=u(s,-z_s)=\frac{1}{2}\big (3s_0-s+\kappa (s)z_s\big ) \,. \end{aligned}$$
(21)

The partial-wave expansion for the tensor-meson decay is slightly more cumbersome as discussed in Appendix B.

We consider elastic unitarity for the two-pion states. The \(\pi \pi \) P-wave phase shift \(\delta (s)=\delta _1^1(s)\) is parameterized according to Ref. [64]. A partial wave \(\chi _1(s)\) of angular momentum \(\ell =1\) obeys a unitarity relation of the form

$$\begin{aligned} \textrm{disc}\,\chi _1(s)&= \lim _{\epsilon \rightarrow 0}\,[\chi _1(s+i\epsilon )-\chi _1(s-i\epsilon )] \nonumber \\ {}&= 2i\chi _1(s)\sin \delta (s)e^{-i\delta (s)}\theta \big (s-4M_\pi ^2\big ). \end{aligned}$$
(22)

It can be decomposed into parts with right-hand and left-hand cuts only, \(\chi _1(s)=\mathcal {X}(s)+{\widehat{\mathcal {X}}}(s)\), with \(\mathcal {X}\in \{\mathcal {F},\mathcal {H},\mathcal {B},\mathcal {G}\}\), where \({\widehat{\mathcal {X}}}\) is the so-called inhomogeneity that has no discontinuity along the right-hand cut. It results from the partial-wave projection of the t- and u-channel SVAs. The inhomogeneities are given by

$$\begin{aligned} {\widehat{\mathcal {F}}}(s)&=3\Big \langle (1-z_s^2)\mathcal {F}\Big \rangle \quad [J^{PC} = 1^{--}], \nonumber \\ \widehat{\mathcal {H}}(s)&=-\frac{3}{2}\Big \langle (1-z_s^2)\mathcal {H}\Big \rangle \quad [J^{PC} = 1^{-+}] , \nonumber \\ {\widehat{\mathcal {B}}}(s)&=\frac{3}{4}\Bigg [\Big \langle (1-z_s^2)\mathcal {B}\Big \rangle - \xi (s) \Big \langle (1-z_s^2)z_s\mathcal {B}\Big \rangle \Bigg ]\nonumber \\&\qquad \qquad \quad [J^{PC} = 2^{++}], \nonumber \\ {\widehat{\mathcal {G}}}(s)&=-\frac{3}{\kappa (s)}\Bigg [3(s-s_0)\langle z_s\mathcal {G}\rangle +\kappa (s)\langle z_s^2\mathcal {G}\rangle \Bigg ] \nonumber \\&\qquad \qquad \quad [J^{PC} = 0^{--}], \end{aligned}$$
(23)

where we employ the notation

$$\begin{aligned} \langle z_s^n \mathcal {X}\rangle = \frac{1}{2}\int _{-1}^1\textrm{d}z_s z_s^n \mathcal {X}\big (t(s,z_s)\big ) , \end{aligned}$$
(24)

and \(\xi (s)\) is defined via

$$\begin{aligned} \xi (s)=\sqrt{1-\frac{4M_\pi ^2}{s}}\frac{s+M^2-M_\pi ^2}{\lambda ^{1/2}(s,M^2,M_\pi ^2)}. \end{aligned}$$
(25)

As a consequence, the unitarity relations for the partial waves can be reduced to those for the SVAs \(\mathcal {X}(s)\), which read

$$\begin{aligned} \textrm{disc}\,\mathcal {X}(s)= & {} 2i\Big (\mathcal {X}(s)+{\widehat{\mathcal {X}}}(s)\Big )\nonumber \\{} & {} \cdot \sin \delta (s)e^{-i\delta (s)}\theta \big (s-4M_\pi ^2\big ). \end{aligned}$$
(26)

The solution for the homogeneous problem, setting \({\widehat{\mathcal {X}}}=0\), is given by the well-known Omnès function \(\varOmega (s)\) [65]

$$\begin{aligned} \mathcal {X}_{\text {hom.}}(s)&=P(s)\varOmega (s),\nonumber \\ \varOmega (s)&=\exp \left( \frac{s}{\pi }\int _{4M_\pi ^2}^\infty \textrm{d}s' \frac{\delta (s')}{s'(s'-s)}\right) , \end{aligned}$$
(27)

where P(s) is a polynomial and \(\varOmega (0)=1\). Such representations are used, e.g., in descriptions of the pion vector form factor; see Ref. [64] and references therein. Using the Omnès function as an approximation for a SVA in a three-pion final state, we describe the rescattering of a two-pion subsystem only, with the third pion being a spectator, as shown in Fig. 1. To include the full rescattering effects, cf. Fig. 2, we need to solve the inhomogeneous equation. The solution is given by [14]

$$\begin{aligned} \mathcal {X}(s)&=\varOmega (s)\Bigg (P_{n-1}(s)\nonumber \\&\quad +\frac{s^n}{\pi }\int _{4M_\pi ^2}^\infty \frac{\textrm{d}s'}{s'^n}\frac{\sin \delta (s'){\widehat{\mathcal {X}}}(s')}{|\varOmega (s')|(s'-s)}\Bigg ), \end{aligned}$$
(28)

where n determines the number of subtractions. As we aim for the most predictive model, without the need to fix the relative strength of various subtraction constants to concrete data, we set \(n=1\). The high-energy behavior of the SVAs is dictated by the one of the Omnès function, which in turn is given by the asymptotic limit of the input phase shift: \(\delta (s\rightarrow \infty ) \rightarrow \pi \) implies \(\varOmega (s\rightarrow \infty )\asymp s^{-1}\), and as a consequence, also the SVAs vanish asymptotically, \(\mathcal {X}(s) \asymp s^{-1}\). As a result, none of the polynomial ambiguities discussed in the previous sections survive: they would alter this asymptotic behavior and violate the high-energy constraint imposed.

We note that the mass of the decay particle enters \({\widehat{\mathcal {X}}}\) via the partial-wave projection integral. Physically, \({\widehat{\mathcal {X}}}\) incorporates the crossed-channel effects, which depend on the relative momenta of all three final-state pions. The resulting differences are discussed in Sect. 2.6. The solution (28) is generated iteratively for each process and decay mass, which can justify the diagrammatic representation in Fig. 2.

Fig. 1
figure 1

Diagrammatic representation of iterated bubble sums for the \(2\pi \) subsystems, implemented by Omnès functions. In general, due to interchange of the rescattered pions, three different bubble sums may contribute

Fig. 2
figure 2

Diagrammatic representation of the amplitudes based on the full Khuri–Treiman equations

Fig. 3
figure 3

Absolute values (left) and phases (right) of the SVAs: for \(J^{PC}=1^{--}\) with different decay masses M (top), and at the same decay mass \(M=7M_\pi \) for the three different reconstruction theorems (middle). The two plots at the bottom show again the absolute values for both comparisons, but this time with the basis functions normalized to the peak of the Omnès function

2.6 Comparison of different SVAs

In Fig. 3, we compare the KT solutions for the different SVAs to the Omnès solution, both for different decay masses (for the \(J^{PC}=1^{--}\) case; cf. also Ref. [48]) and comparing the SVAs with different reconstruction theorems or inhomogeneities at the same decay mass. These pictures suggest we already have an answer to the question to what extent the \(\rho \) lineshape and phase are modified by crossed-channel interactions, and how this modification varies with quantum numbers and decay mass. However, this impression is misleading to some extent, as can be seen by the following considerations.

  1. 1.

    As our reconstruction theorems all depend on one single SVA only, it is obvious that an overall shift of its phase by a constant is not observable. A significant part of the changes in phase compared to the input phase shift seen in Fig. 3 can already be undone by such a shift.

  2. 2.

    Although we have theoretically constrained our SVAs to fulfill a certain, restrictive, high-energy behavior, this still means that a polynomial shift according to the corresponding ambiguity is not observable in a finite Dalitz plot. This suggests that any change between Omnès and full KT solution that is, in fact, polynomial-like will not be experimentally verifiable.

  3. 3.

    Finally, the single subtraction or normalization constants of our dispersive amplitude representations are not a priori fixed; changes in the SVAs that can be absorbed in a change of normalization will therefore also not allow us to verify non-trivial rescattering effects. This is demonstrated in the bottom row of Fig. 3, where the SVAs are not commonly normalized at \(s=0\), but in the \(\rho \) peak: the differences between the different solutions already appear significantly muted.

All three points demonstrate that it is very difficult to quantify the observable changes by considering complex, interfering decay amplitudes only. We therefore choose a different, unambiguous, path in the following and immediately study the Dalitz-plot distributions, which are direct observables.

3 Log-likelihood estimator

In experiments measuring Dalitz plots, the data is often binned. The binning scheme is determined by each experiment individually to obtain distributions with reasonable statistical and systematic uncertainties; therefore, there is no unique prescription even for a given number of overall events. We thus seek an unbinned method to characterize the Dalitz-plot distribution, relying on the total number of events for the full Dalitz plot only. In the following, the deviations between the Omnès and KT solutions are quantified using the Kullback–Leibler (KL) divergence [66] based on the log-likelihood estimator.

As the description of the decay amplitudes in terms of KT equations is physically more complete, we interpret their outcome as the “truth” and test to which extent the Omnès functions are capable of reproducing this. The decay amplitudes are denoted by \(\mathcal {M}^\text {KT}\) and for the KT and Omnès solutions, respectively, where the isospin and helicity indices are suppressed for clarity. For the Omnès solution, we replace the SVAs by plain Omnès functions in the corresponding reconstruction theorems. We define the probability density function (pdf) as

$$\begin{aligned} f(s,t)&=\frac{\overline{|\mathcal {M}|^2}(s,t,u)}{\int _D \overline{|\mathcal {M}|^2}(s,t,u)\textrm{d}s\textrm{d}t}. \end{aligned}$$
(29)

Additionally, we explore an unphysical quantity defined by

$$\begin{aligned} |\tilde{\mathcal {M}}(s,t,u)|^2&=\frac{\overline{|\mathcal {M}|^2}(s,t,u)}{\mathcal {K}(s,t,u)}, \nonumber \\ {\tilde{f}}(s,t)&=\frac{|\tilde{\mathcal {M}}(s,t,u)|^2}{\int _D |\tilde{\mathcal {M}}(s,t,u)|^2\textrm{d}s\textrm{d}t}. \end{aligned}$$
(30)

This construct appears to be more sensitive to the distribution at the edge of the Dalitz plots and therefore visualizes the \(\rho \) bands discussed in Sect. 4. Our representation is fixed up to a normalization constant, therefore the pdf is free of any undetermined parameters. The integral region D is the three-body decay phase space, captured by the Dalitz plot. For the decay mass M and the pion mass \(M_\pi \), its boundaries are determined by

$$\begin{aligned} 4M_\pi ^2&\le s \le (M-M_\pi )^2,\nonumber \\ t(s,z_s=-1)&\le t \le t(s,z_s=1), \end{aligned}$$
(31)

where \(t(s,z_s)\) is given by Eq. (21). We generate a sample \({\mathbb {D}}\) by drawing \(N\in {\mathbb {N}}\) pairs \((s_i,t_i)\) from \(f^\text {KT}\). The likelihood function L of a pdf f with respect to the data sample \({\mathbb {D}}\) is defined as

$$\begin{aligned} L({\mathbb {D}},f)=\prod _{i=1}^N f(s_i,t_i). \end{aligned}$$
(32)

The likelihood ratio of the Omnès and KT solutions

(33)

indicates which one is favored. In the following, we will use the log-likelihood and its difference

(34)

We note that is possible despite drawing data from the KT solution, since N is finite. This gives us precisely the handle we need to determine the value of N to observe crossed-channel rescattering effects.

We can now perform \(B\in {\mathbb {N}}\) runs, which generate B datasets \({\mathbb {D}}_b\), \(b=1,\ldots ,B\), of size N. On each of these datasets, one can compute and access its probabilistic distribution. For large values of B, this distribution is Gaussian, with the mean and variance given by

(35)

where

(36)

The expressions are known as the Kullback–Leibler divergence [66] and variance. The cumulative distribution function reads

$$\begin{aligned} \mathcal {N}(x,\mu (N),\sigma (N))&=\frac{1}{2}\left( 1+\text {erf}\left( \frac{x-\mu (N)}{\sqrt{2}\sigma (N)}\right) \right) \nonumber \\ \text {with} \quad \mu (N)&=-Nd_\text {KL}, \quad \sigma (N)=\sqrt{N\nu _\text {KL}}, \end{aligned}$$
(37)

where erf is the error function. To validate the assumption of a normal distribution, we use the comparison in Fig. 4, which indicates a very good description.Footnote 2 From here on we can calculate our results using the pdfs as defined in Eq. (29).

Fig. 4
figure 4

Histogram for \(B=10^6\) datasets of different sample size N. For this plot, the amplitudes are computed at \(M=M_\phi \) and for \(J^{PC}=1^{--}\). Additionally, we plot Gaussians with the mean and standard deviation from Eq. (37)

In the region we reject the hypothesis that the Omnès solutions are sufficient to describe the data. The probability that the hypothesis is not rejected then reads

$$\begin{aligned} q(N)=1-\mathcal {N}(0,\mu (N),\sigma (N)). \end{aligned}$$
(38)

The inversion of the equation gives the number of events with the confidence determined by q via

$$\begin{aligned} N(q)=2\nu _\text {KL}\left( \frac{\text {erf}^{-1}(1-2q)}{d_\text {KL}}\right) ^2. \end{aligned}$$
(39)

For a \(5\sigma \) confidence level we need to have \(q=2.87\cdot 10^{-7}\) [58] and can now compute the resulting N.

Fig. 5
figure 5

Left panel: Probability of the Omnès model not being rejected as a function of the mass of the decaying particle, for all four reconstruction theorems and \(N=1000\). Right panel: Statistics needed to set the probability that the Omnès model is not rejected to \(5\sigma \) (right). The gray vertical lines, from left to right correspond to the masses of the \(\omega \), \(\phi \), \(\omega (1420)\), and \(\omega (1650)\) resonances

Fig. 6
figure 6

Left panel: Comparison for the \(1^{--}\) reconstruction theorem with normal SVAs and \(1^{-+}\) SVAs (“mixed”). Right panel: Comparison for \(1^{-+}\) reconstruction theorem with normal SVAs and \(1^{--}\) SVAs (“mixed”). We show the probability of the Omnès model not being rejected as a function of the mass of the decaying particle. Vertical gray lines as in Fig. 5

Fig. 7
figure 7

Dalitz plots for the \(J^{PC}=1^{-+}\) reconstruction theorem. From top to bottom the decay mass grows according to \(M=5/7.25/8/14\,M_\pi \), while the Dalitz plot without phase space \({\tilde{f}}(s,t)\) defined via Eq. (30) is shown in the left and \({\tilde{d}}_\text {KL}(s,t)\) (Eq. (36)) in the right column

Fig. 8
figure 8

Dalitz plots for the \(J^{PC}=1^{--}\) reconstruction theorem. From top to bottom the decay mass grows according to \(M=5/7.25/8/14\,M_\pi \), while the Dalitz plot without phase space \({\tilde{f}}(s,t)\) defined via Eq. (30) is shown in the left and \({\tilde{d}}_\text {KL}(s,t)\) (Eq. (36)) in the right column

4 Results

Among the three-pion decays studied in this paper, only two Dalitz plots have been studied experimentally with sufficiently high statistics: \(\omega \rightarrow 3\pi \) [42, 43], and \(\phi \rightarrow 3\pi \) [44, 45].

For \(\omega \rightarrow 3\pi \), WASA-at-COSY [42] has performed a Dalitz plot study with \(44\,080\) events, while the analysis by BESIII [43] is based on \(260\,520\) events. Both experiments parameterize the distribution by a polynomial expansion and present results testing one- and two-parameter models. Applying the formalism of the preceding section, we find that the statistics of WASA-at-COSY is sensitive to rescattering effects only at \(2.1\sigma \). On the other hand, in agreement with Ref. [26], BESIII reaches a \(5\sigma \) level for the solution containing one subtraction. However, as pointed out by Ref. [26], an additional subtraction leads to a better agreement for the Dalitz plot parameters.

For \(\phi \rightarrow 3\pi \), KLOE [44] provides a Dalitz plot analysis using \(2\cdot 10^6\) events, while CMD-2 [45] has studied almost \(8\cdot 10^4\) decays. For both, rescattering effects are clearly observable, as concluded by Ref. [15].

Using the statistical method explained above and the derived reconstruction theorems, we compute q(N) and N(q) for a large mass range as shown in Fig. 5. The mass dependence of N and q for the \(1^{--}\) decay looks strikingly different from the \(1^{-+}\) one, even though they share the same kinematic factor. To investigate the source of this difference in sensitivity between the two reconstruction theorems, we perform the following, unphysical, test. We plug the SVAs, calculated as KT solutions for the \(1^{-+}\) decay, into the linear combination given by the reconstruction theorem for \(1^{--}\), see Eq. (5), and vice versa. These unphysical amplitudes are denoted by “mixed” in Fig. 6. We observe that this changes the absolute values of q(N), while the qualitative behavior is the same. We therefore conclude that much of the sensitivity to rescattering effects is not actually due to the size of the lineshape modification of the SVAs, as shown in Fig. 3, but rather due to the specific linear combination in which they form the full decay amplitude.

In a log-plot for N (for fixed q) as a function of the decay mass, we find a similar form as for q (with fixed N); cf. left and right panels of Fig. 5. For large decay masses, the necessary number of events rises for all processes.Footnote 3 This is due to the fact that the KT solutions converge to the Omnès function in the infinite-mass limit. However, in the high-mass region, \(M\gtrsim 15M_\pi \), our approximations are no longer valid: inelastic effects and higher partial waves play a non-negligible role. For low decay masses, approaching the three-pion threshold, the necessary event numbers for the \(1^{--}\), \(1^{-+}\), and \(2^{++}\) decays rise due to limited phase space and kinematic suppression of the Dalitz-plot borders, far away from the \(\rho \) resonance. For \(0^{--}\) this is different, since here the amplitude does not vanish at the edge of the Dalitz plot.

The mass scan manifests several prominent features in the significance plot. For \(1^{-+}\) decays, N starts at \(10^5\) events around the \(\omega \) mass and then rises steeply to approximately \(2\cdot 10^6\) events at the \(\phi \) mass. At higher masses, it falls off up to about \(12M_\pi \). The very high number of necessary events is mainly due to the fact that crossing symmetry requires a zero in the Dalitz plot along the line \(t=u\), and hence any differences due to rescattering have to appear at the edge of the Dalitz plot, where the phase space is suppressed by the Kibble function. The Dalitz plots for decays of a particle with \(1^{-+}\) quantum numbers are shown in Fig. 7 for different masses. Here the difference decreases until the \(\rho \) bands are inside the Dalitz plot, and then falls off again when the size increases further.

Fig. 9
figure 9

Sketches of different kinematic configurations for the \(1^{--}\) decay with increasing decay mass. The straight lines denote the (qualitative) position of the three \(\rho \) bands in the Dalitz plot. The masses below the diagrams denote the decay masses for which the specific kinematic configurations are reached

Fig. 10
figure 10

Minimal number of events N to exclude the Omnès model at \(5\sigma \) significance for \(J^{PC}=1^{--}\), as a function of the decay mass decomposed into two different effects. The red line corresponds to the full solution from the right panel in Fig. 5. For the green line (“mass fixed”), we fix the size of the Dalitz plot to \(M=7.5M_\pi \) and only use the SVA varying with the running mass. For the blue line (“basis fixed”), we use the same SVA solution for \(M=7.5M_\pi \) for all masses and vary the size of the Dalitz plot. Vertical gray lines as in Fig. 5

For the \(1^{--}\) decays, we find a different behavior. The event number N starts at high values for the \(\omega \) resonance and then shows an overall decline with rising decay mass, with two small peaks between the \(\phi \) and \(\omega (1650)\). At the \(\phi \) resonance mass, the \(\rho \) bands are completely inside the Dalitz plot. The Dalitz plots for different decay masses are shown in Fig. 8. The first peak occurs due to the third kinematic configuration of the Dalitz plot as shown in Fig. 9. The difference increases again when the three \(\rho \) bands cross in the middle of the Dalitz plot. The second peak is also due to a peculiar structure in the Dalitz plot: in this decay mass region we find a ring-shaped local minimum, clearly visible for an unphysically narrow \(\rho \) width; see Appendix D. The ring affects the sensitivity even for the physical \(\rho \) width, and is responsible for the second peak.

In order to disentangle the origin of the various maxima and minima in the sensitivity of the \(1^{--}\) decays in dependence on the decay mass a little better, we separate, somewhat unphysically, two different effects in Fig. 10: the size of the Dalitz plot, and modifications of the SVAs. We once keep the SVA basis function fixed as calculated for decay mass \(M=7.5M_\pi \) and only vary the size of the Dalitz plot; and secondly, we vice versa keep the Dalitz plot fixed at \(M=7.5M_\pi \), and only vary the SVA with its implicit decay-mass dependence. The precise plot depends heavily on the choice for the fixed mass, but we observe both peaks when using a SVA for a fixed mass and only varying the size of the Dalitz plot. Fixing the size of the Dalitz plot and only varying the SVAs, on the other hand, results in a rather smooth decrease of the difference towards higher masses. We therefore conclude that the peaks in the \(1^{--}\) mass dependence are dominated by the structure of the Dalitz plot and not by the difference in the SVAs.

The main takeaway however is that with less than \(10^5\) events above the \(\phi \) mass, rescattering effects will play an important role in analyses of Dalitz plot data. For the \(\omega \) resonance and lower masses of the decaying particle, one requires more than \(10^6\) events to observe them.

For \(0^{--}\) decays, we find some small numerical fluctuations in the low-mass region, which are due to the multiple regions where the total decay amplitude is kinematically suppressed in the Dalitz plots, shown in Fig. 11: it vanishes along the three lines of \(s=t\), \(s=u\), and \(t=u\). The number of events slightly rises up to approximately \(N=10^6\) at the mass of the \(\omega \) and stays constant up to \(9M_\pi \). At higher masses, a slow decline sets in, including a small peak around \(14M_\pi \). Due to the structure of the reconstruction theorem, the differences are located near the Dalitz plot boundaries. The six regions of intensity are then split into regions with larger pdfs for the KT equations and the Omnès solutions, respectively.

For \(2^{++}\) decays, the low-mass behavior looks similar to the \(1^{--}\) decays. The difference, however, is that we find a small plateau up to the \(\omega \) mass, such that the number of events is increased by orders of magnitude. Furthermore the decline stops at the \(\phi \) mass above \(10^5\) events, such that the behavior at larger masses is more similar to the other quantum numbers. The Dalitz plots in Fig. 12 suggest that we do not have any visible features of the \(\rho \) bands for the small decay masses. For larger masses we observe the two expected bands in t- and u-channel with one crossing point inside the Dalitz plot. Similar to the \(1^{--}\) decays the Dalitz plot is split into two regions, where the pdfs for the KT equations and the Omnès functions are of different size. These regions vary in shape for the different decay masses.

Fig. 11
figure 11

Dalitz plots for the \(J^{PC}=0^{--}\) reconstruction theorem. From top to bottom the decay mass grows according to \(M=5/7.25/8/14\,M_\pi \), while the Dalitz plot f(st) defined via Eq. (29) is shown in the left and \({\tilde{d}}_\text {KL}(s,t)\) (Eq. (36)) in the right column

Fig. 12
figure 12

Dalitz plots for the \(J^{PC}=2^{++}\) reconstruction theorem. From top to bottom the decay mass grows according to \(M=5/7.25/8/14\,M_\pi \), while the Dalitz plot without the \(\mathcal {K}(s,t,u)\) factor, \(\tilde{f}(s,t)\), defined via Eq. (29) is shown in the left and \({\tilde{d}}_\text {KL}(s,t)\) (Eq. (36)) in the right column

Finally, to test the dependence of our findings on the \(\rho \) resonance width, we repeat the above exercises using a phase shift extracted from a simple Breit–Wigner model with an energy-dependent width [67], whose nominal width we fix to the smaller value \(\varGamma _\rho =30\,\text {MeV}\). We find that the number of events required to distinguish rescattering effects is significantly increased by about two orders of magnitude. This confirms the expected trend that rescattering effects vanish in the limit of small widths. A short analytic derivation of this limit is presented in Appendix E.

5 Conclusion

In this article, we have investigated the feasibility of unambiguously identifying the crossed-channel rescattering effects beyond the simplest isobar model in three-pion decays dominated by \(\rho \pi \) intermediate states. For four different quantum numbers of the decaying particles we have solved the Khuri–Treiman equations, integral equations that sum iterated two-pion rescattering in each pion pair to all orders. We have determined the minimal sample sizes for which Dalitz-plot distributions allow us to distinguish the KT solutions from the naive picture that ignores all effects beyond two-body resonances. The dependence on the mass of the decaying resonance has been studied in detail throughout.

The significance of the rescattering effects in \(3\pi \) final states is heavily dependent on the decay kinematics. In particular, the appearance and position of the \(\rho \) bands in the Dalitz plots plays a major role. For \(J^{PC}=1^{--}\) we found a strong dependence on the mass of the decaying particle: rescattering effects are small for the \(\omega \) resonance, where at least a few times \(10^5\) events in a Dalitz plot are necessary to identify them at \(5\sigma \) significance, while they are easily observable for the \(\phi \) resonance, with of the order of \(10^4\) Dalitz plot events sufficient; a similar sensitivity is expected for decaying isoscalar vector resonances up to almost \(2\,\text {GeV}\). However, the predictive power for large masses, e.g., the three-pion decays of the vector charmonia \(J/\psi \) or \(\psi '\) [68], is clearly limited due to inelastic effects and higher partial waves.

In general, the processes with zeros in the amplitude due to crossing symmetry need more statistics to resolve rescattering effects. This is the case for the isovector \(3\pi \) system with quantum numbers \(1^{-+}\) and \(2^{++}\),Footnote 4 as well as isoscalar \(0^{--}\) states, where for a wide mass range up to \(2\,\text {GeV}\), we predict necessary statistics between \(10^5\) and \(10^6\) events to identify non-trivial rescattering effects at \(5\sigma \) significance.

Throughout, our investigation should be understood as a pilot study towards the thorough implementation of Dalitz plot fits beyond the simplest isobar models. Theoretical limitations at this point clearly concern the constraint to Khuri–Treiman systems with one single free parameter: it is by no means guaranteed that the neglect of additional subtraction constants, which inter alia allow us to absorb effects of inelastic intermediate states, is justified in all circumstances. Furthermore, decays with several relevant partial waves, in particular isoscalar \(\pi \pi \) S-waves with their strong coupling to \(K{\bar{K}}\) above \(1\,\text {GeV}\), pose additional difficulties to pin down corrections to two-pion lineshapes unambiguously; these are known to play an important role in the interpretation of resonance signals in the \(a_1\) spectrum [69,70,71]. The systematic study of such more complicated KT systems remains a both formidable and highly rewarding challenge for future research.