1 Introduction

Rare \({{b}} \!\rightarrow {s} {\gamma } \) flavour-changing neutral-current transitions are expected to be sensitive to New Physics (NP) effects. These transitions are allowed only at loop level, and NP could arise from the exchange of a heavy particle in the electroweak penguin loop. In the Standard Model (SM), the recoil \(s \) quark that couples to a \(W \) boson is left-handed, causing the photon emitted in \({{b}} \!\rightarrow {s} {\gamma } \) transitions to be almost completely left-handed. Several theories beyond the SM predict a significant right-handed component for the photon polarisation: in the minimal supersymmetric model (MSSM), left-right squark mixing causes a chirality flip along the gluino line in the electroweak penguin loop [1], while in some grand unification models right-handed neutrinos (and the associated right-handed quark coupling) are expected to enhance the right-handed photon component [2].

Various complementary approaches have been proposed for the determination of the polarisation of the photon in \({{b}} \!\rightarrow {s} {\gamma } \) transitions. The first one consists in studying the time-dependent decay rate of \({B} ^{0}_{({s})} \rightarrow f^{{C\!P}} \gamma \) decays, where \(f^{{C\!P}}\) is a particle or system of particles in a \(C\!P\) eigenstate [3]. An alternative approach involves the study of angular distributions of the four-body final state in \({{B} ^0} \rightarrow K^{*0} \ell ^+ \ell ^-\) decays [4]. Yet another proposed method involves exploiting the angular distributions of the photon and the proton in the final state of \(\varLambda _{{b}} \rightarrow \varLambda _X (\rightarrow p h) \gamma \) decays, where \(\varLambda _X\) is either the ground state or an excited state of the \(\varLambda \) hyperon and h is a kaon or a pion [5].

Information on the photon polarisation can also be obtained from \(B\) decays to three hadrons and a photon. This approach is enabled by the fact that the three final-state hadrons allow the construction of a parity-odd triple product that inverts its sign with a change in the photon chirality, and by the existence of interference between the amplitudes of the hadronic system [6, 7].

In \(B \rightarrow {K} _\mathrm{{res}} \gamma \) decays, where \({K} _\mathrm{{res}}\) is a kaonic resonance decaying to a \({K} {\pi } {\pi } \) final state, the required interference in the \({K} {\pi } {\pi } \) system can arise from several sources. In the case of a single \({K} _\mathrm{{res}}\) state, the helicity amplitudes must contain at least two terms with a non-vanishing relative phase. This can occur between intermediate resonance amplitudes in the decay \({K} _\mathrm{{res}}\) \(\rightarrow \) \({K} {\pi } {\pi } \), between S and D wave amplitudes in the decay, or between two intermediate \({K} ^*\) \(\pi \) states with different charges, related by isospin symmetry.Footnote 1 Interference can also appear in the presence of different overlapping \({K} _\mathrm{{res}}\) states; in fact, the presence of a multitude of interfering resonances makes it very difficult to distinguish them, thus complicating the interpretation of the observed distributions.

A simplified approach to the study of the photon polarisation consists in exploiting the distribution of the polar angle of the photon with respect to the hadronic decay plane integrating over the resonance content of the \({K} {\pi } {\pi } \) system [6]. Using \(3\,\,\hbox {fb}^{-1} \) of pp collisions at the LHC, the LHCb collaboration determined the shape of this distribution and the up-down asymmetry between the number of events with photons emitted on either side of the plane [8]. The up-down asymmetry was found to differ from zero by 5.2 standard deviations. As this asymmetry is expected to be proportional to the photon polarisation parameter \(\lambda _{\gamma }\), this result represents the first observation of a parity-violating nonzero photon polarisation in \({{b}} \!\rightarrow {s} {\gamma } \) transitions. The proportionality coefficient between the up-down asymmetry and \(\lambda _{\gamma }\) depends on the resonance content of the \({K} {\pi } {\pi } \) system, and in particular on the interference pattern between the various decay modes. Without precise knowledge of these amplitudes, a measurement of the up-down asymmetry cannot be translated into a photon polarisation value.

In this paper, a method to determine the value of the photon polarisation parameter by means of an amplitude analysis of the \({K} {\pi } {\pi } {\gamma } \) system is proposed. It is organised as follows: a description of the up-down asymmetry and its limitations in extracting a value for the photon polarisation parameter are detailed in Sect. 2. In Sect. 3, a general expression for the \(B\!\rightarrow K\pi \pi {\gamma } \) decay rate in terms of a photon polarisation parameter is derived, the amplitude formalism is described, and the fit method used for the amplitude analysis is explained. In Sect. 4, results for simulated data sets with assumed models of \({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) and \({{B} ^0} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^0} {\gamma } \) decays are presented. Statistical sensitivities on the photon polarisation parameter are quoted for these models, assuming no background and no experimental effect. Conclusions are drawn in Sect. 5.

2 Motivation

\({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) and \({{B} ^0} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^0} {\gamma } \) decays can be described in terms of five independent variables: two angles (\(\cos \theta \) and \(\chi \)) that describe the direction of the photon in the rest frame of the kaonic resonance \({K} _\mathrm{{res}}\), and three squared invariant masses (\(s_{123}, s_{12}, s_{23}\)), where the indices 1, 2 and 3 refer respectively to the final-state \(\pi ^+\), \(\pi ^-\) and \(K^+\) for the charged decay mode, and to \(\pi ^-\), \(\pi ^0\) and \(K^+\) for the neutral decay mode.

As illustrated in Fig. 1 for \({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) decays, in the rest frame of the kaonic resonance \({K} _\mathrm{{res}}\), the normal to the hadronic decay plane is denoted by \({\hat{\mathbf {n}}} = (\mathbf {p}_{1} \times \mathbf {p}_{2}) / |\mathbf {p}_{1} \times \mathbf {p}_{2}|\). The polar angle \(\theta \) is the angle between \({\hat{\mathbf {n}}}\) and the opposite of the photon momentum, so that \(\cos \theta = - {\hat{\mathbf {n}}}\cdot \mathbf {p}_{\gamma }/ |\mathbf {p}_{\gamma }| \).Footnote 2 The angle \(\chi \) is defined from

$$\begin{aligned} \cos \chi&= \frac{({\hat{\mathbf {n}}} \times \mathbf {p}_{1}) \cdot ({\hat{\mathbf {n}}} \times \mathbf {p}_{\gamma })}{|{\hat{\mathbf {n}}} \times \mathbf {p}_{1}| \, |{\hat{\mathbf {n}}} \times \mathbf {p}_{\gamma }|}, \end{aligned}$$
(1)
$$\begin{aligned} \sin \chi&= \frac{({\hat{\mathbf {n}}} \times \mathbf {p}_{1}) \times ({\hat{\mathbf {n}}} \times \mathbf {p}_{\gamma })}{|{\hat{\mathbf {n}}} \times \mathbf {p}_{1}| \, |{\hat{\mathbf {n}}} \times \mathbf {p}_{\gamma }|} \cdot {\hat{\mathbf {n}}}. \end{aligned}$$
(2)
Fig. 1
figure 1

Definitions of the angular variables used to describe the \(K \pi \pi \gamma \) system. The indices 1, 2 and 3 refer respectively to the final-state \(\pi ^+\), \(\pi ^-\) and \(K^+\) in \({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) decays, and to \(\pi ^-\), \(\pi ^0\) and \(K^+\) in \({{B} ^0} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^0} {\gamma } \) decays

The \(B\!\rightarrow K\pi \pi {\gamma } \) differential branching fraction has the following dependence on \(\cos \theta \) [7]:

$$\begin{aligned}&\frac{{\text {d}}\!\varGamma ({B} \!\rightarrow {K} _\mathrm{{res}} \gamma \!\rightarrow {K} {\pi } {\pi } {\gamma } )}{{\text {d}}\!s_{123}{\text {d}}\!s_{12}{\text {d}}\!s_{23}{\text {d}}\!\chi {\text {d}}\!\cos \theta }\nonumber \\&\quad = \sum _{i=0,2,4} a_i(s_{123}, s_{12}, s_{23},\chi )\cos ^i\theta \nonumber \\&\quad \quad + \lambda _\gamma \sum _{j=1,3} a_j(s_{123}, s_{12}, s_{23}, \chi )\cos ^j\theta . \end{aligned}$$
(3)

Integrating Eq. 3 over the squared invariant masses and \(\chi \), the up-down asymmetry (\(\mathcal {A}_\mathrm{{ud}}\)) is defined as [6, 7]

$$\begin{aligned} \mathcal {A}_{\text {ud}} \equiv \frac{\int _0^1 {\text {d}}\!\cos {\theta }\frac{{\text {d}}\!\varGamma }{{\text {d}}\!\cos {\theta }}-\int _{-1}^0{\text {d}}\!\cos {\theta }\frac{{\text {d}}\!\varGamma }{{\text {d}}\!\cos {\theta }}}{\int _{-1}^1 {\text {d}}\!\cos {\theta }\frac{{\text {d}}\!\varGamma }{{\text {d}}\!\cos {\theta }}}, \end{aligned}$$
(4)

where the terms in even powers of \(\cos {\theta }\) disappear, and the resulting asymmetry is directly proportional to \(\lambda _\gamma \) with a proportionality coefficient that depends on the resonance content of the \({K} {\pi } {\pi } \) system.

The effects of the resonant structure of the \({K} {\pi } {\pi } \) system on \(\mathcal {A}_{\text {ud}}\) can be illustrated using a simplified \({{{B} ^+}} \!\rightarrow K^+_{\text {res}} \gamma \) model containing only two amplitudes corresponding to the decays \(K_1(1270)^+\!\rightarrow {{K} ^+} \rho (770)^0\) and \(K_1(1270)^+\!\rightarrow K^{*}(892)^0 {{\pi } ^+} \) with \(\rho (770)^0\!\rightarrow {{\pi } ^+} {{\pi } ^-} \) and \(K^{*}(892)^0\!\rightarrow {{K} ^+} {{\pi } ^-} \). Simulated samples of decays containing only right-handed photons are generated with different relative fractions (as defined in Eq. 18) and phase differences between these amplitudes, and the up-down asymmetry is computed for each of them. The results in Fig. 2 show that the up-down asymmetry varies widely depending on the phase difference between the amplitudes, while it is less dependent on the relative fraction. This implies that, even in this simple model, the proportionality coefficient that relates the up-down asymmetry to the photon polarisation parameter depends strongly on the phase difference between the amplitudes, making the knowledge of this phase essential to measure the value of \(\lambda _{\gamma }\); additionally, for some values of the relative phase, the proportionality coefficient is null, indicating that the measurement of the up-down asymmetry is not sensitive to \(\lambda _{\gamma }\) in such configurations.

Fig. 2
figure 2

Up-down asymmetry \(\mathcal {A}_\mathrm{{ud}}\) for simulated samples of \(B^+ \rightarrow K_1(1270)^+ \gamma \) decays governed by two amplitudes only, \(K_1(1270)^+ \rightarrow {{K} ^+} \rho (770)^0\) and \(K_1(1270)^+ \rightarrow K^{*}(892)^0 \pi ^+\), shown as a function of the generated ratio of fractions (radial coordinate, from 0.1 to 9.0) and phase difference between the two amplitudes (polar coordinate)

To overcome these difficulties and measure the photon polarisation, we propose an analysis that combines information from the angular variables and the squared invariant-mass distributions in order to characterise the interferences between decay processes and their effect on \(\lambda _{\gamma }\).

3 Method

3.1 Photon polarisation parameter

The differential decay rate for \(B\!\rightarrow K\pi \pi {\gamma } \) decays that proceed through a single resonance \(K_{\text {res}}^{i}\) can be written as [7]

$$\begin{aligned}&\dfrac{{\text {d}}\!\varGamma (B\rightarrow K_{\text {res}}^{i} (\rightarrow K \pi \pi ) \gamma )}{{\text {d}}\!s_{123}} \nonumber \\&\quad = |c^{i}_{\text {R}} \mathcal {T}^{i}(s_{123}) A^{i}_{\text {R}}|^{2} + |c^{i}_{\text {L}} \mathcal {T}^{i}(s_{123}) A^{i}_{\text {L}} |^{2}, \end{aligned}$$
(5)

where \(s_{123}\) is the invariant mass of the \(K \pi \pi \) system, \(c^{i}_{\text {R}}\) and \(c^{i}_{\text {L}}\) are the right- and left-handed weak radiative decay amplitudes, \(\mathcal {T}^{i}(s_{123})\) is the propagator associated to resonance \(K_{\text {res}}^{i}\), and \(A^{i}_{\text {R}}\) and \(A^{i}_{\text {L}}\) are the strong decay amplitudes for \(K_{\text {res, R/L}}^{i} \rightarrow K \pi \pi \). The right- and left-handed amplitudes do not interfere since the photon polarisation is an observable quantity. For a given resonance \(K_{\text {res}}^{i}\), a photon polarisation parameter \(\lambda ^{i}_{\gamma }\) is defined in terms of the weak radiative decay amplitudes,

$$\begin{aligned} \lambda _\gamma ^{i}\equiv \frac{|c^{i}_{\text {R}}|^2-|c^{i}_{\text {L}}|^2}{|c^{i}_{\text {R}}|^2+|c^{i}_{\text {L}}|^2}. \end{aligned}$$
(6)

Using an argument of parity invariance in strong interactions, detailed in Ref. [2], the weak radiative decay amplitudes associated with a resonance \(K_{\text {res}}^{i}\) in decays of a \(B^{+}\) or \(B^{0}\) meson can be written as [7, 9]

$$\begin{aligned} \begin{pmatrix} c^{i}_{\text {R}} \\ c^{i}_{\text {L}} \end{pmatrix} = - \dfrac{4G_{\text {F}}}{\sqrt{2}} V_{tb} V_{ts}^{*} \begin{pmatrix} C_{7}^{\text {eff}}\,g^{i}(0) + h_{\text {R}}^{i} \\ C^{\prime }_{7}\,P_{i} (-1)^{J_{i}-1}\,g^{i}(0) + h_{\text {L}}^{i} \end{pmatrix}, \end{aligned}$$
(7)

where \(G_{\text {F}}\) is the Fermi constant, \(V_{tb}\) and \(V_{ts}^{*}\) are CKM matrix elements, \(P_{i}\) and \(J_{i}\) are the parity and spin of the \(K_{\text {res}}^{i}\) resonance, \(g^{i}(0)\) is the process-dependent hadronic form factor, \(C^{\text {eff}}_{7}\) and \(C^{\prime }_{7}\) are the radiative Wilson coefficients, and the quantities \(h_{\text {R/L}}^{i}\) encode remaining contributions from the \(Q_{1-6}\) and \(Q_8\) hadronic operators (see Ref. [9] for more details). The coefficient \(C^{\text {eff}}_{7}\) includes “effective” linear contributions from the other coefficients \(C_{1-6}\) in order to make it regularisation- and renormalisation-scheme independent, as discussed in Ref. [10]. Assuming that the \(h_{\text {R/L}}^{i}\) terms are small enough to be neglected in the expressions of \(c^{i}_{\text {R}}\) and \(c^{i}_{\text {L}}\), the photon polarisation parameter reduces to

$$\begin{aligned} \lambda _\gamma ^{i}=\frac{|C_{7}^{\text {eff}}|^2 - |C^{\prime }_{7}|^2}{|C_{7}^{\text {eff}}|^2 + |C^{\prime }_{7}|^2}\equiv \lambda _{\gamma }, \end{aligned}$$
(8)

i.e., the photon polarisation in the weak decay \({{{B} ^+}} \!\rightarrow {K} ^{i+}_{\text {res}} {{\gamma }} \) is the same for all kaonic resonances \(K_{\text {res}}^{i}\) and it can be expressed only as a function of Wilson coefficients.Footnote 3 In the SM, the value of \(\lambda _{\gamma }\) is expected to be \(+1\) (up to corrections of the order of \(m_s^2/m_b^2\)) for decays of a \(B^+\) or \(B^{0}\) meson while it is expected to be \(-1\) for decays of a \(B^-\) or \(\bar{B}^{0}\) meson.

3.2 Amplitude formalism

To develop our formalism, decays of B mesons to \(K\pi \pi \gamma \) are assumed to proceed through a cascade of quasi-independent two-body decays, an approximation known as the isobar model [11, 12]. In this study, decay topologies of the form \(B \rightarrow R_i \gamma \), \(R_i \rightarrow R_j P_{1} \), and \(R_j \rightarrow P_{2} P_{3}\) are considered, where \(R_i\) is a \(K\pi \pi \) intermediate state, \(R_j\) is either a \(K\pi \) or \(\pi \pi \) resonant state and \(P_{\alpha }\) is a final-state kaon or pion. The function used to describe \(B\!\rightarrow K\pi \pi {\gamma } \) decays with the above topologies is therefore written as

$$\begin{aligned} \mathcal {P}_{\text {s}} = \dfrac{(1+\lambda _{\gamma })}{2}|\mathcal {M}_{\text {R}}|^{2} + \dfrac{(1-\lambda _{\gamma })}{2} |\mathcal {M}_{\text {L}}|^{2}, \end{aligned}$$
(9)

where amplitudes for various decay modes associated with right-handed (or left-handed) photons are summed coherently,

$$\begin{aligned} \mathcal {M}_{\text {R}/\text {L}} = \sum _{k} f_{k}\mathcal {A}_{k, \text {R}/\text {L}}(\varvec{x}), \end{aligned}$$
(10)

with \(f_{k}=a_{k}e^{i\phi _{k}}\).

The decay amplitude \(\mathcal {A}_{k, \text {R}/\text {L}}(\varvec{x})\) corresponds to a \(B\!\rightarrow K\pi \pi {\gamma } \) process k involving resonances \(R_i\) and \(R_j\) and a right- or left-handed photon, and \(\varvec{x}\) is the set of four-vectors associated with the final-state particles in the rest frame of the B meson. The complex coefficient \(f_{k}=a_{k}e^{i\phi _{k}}\) accounts for the magnitude \(a_{k}\) and phase \(\phi _{k}\) of decay amplitude k and is assumed to be the same for decays with right- or left-handed photons. The amplitude for a given decay mode k is a product of resonance propagators \(\mathcal {T}\) for each intermediate two-body decay with relative angular momentum L, a normalised Blatt–Weisskopf coefficient \(B_{L_B}\) for the two-body decay of the B characterised by relative angular momentum \(L_B\) and breakup momentum \(q_B\), and an overall spin factor \(\mathcal {S}_{ij}\) that encodes the dependence of the amplitudes on angular momenta,

$$\begin{aligned}&\mathcal {A}_{\text {R}}^{k}(\varvec{x}) = B_{L_B}(q_B(\varvec{x}),0) \mathcal {T}_{i}^{k}(\varvec{x}) \mathcal {T}_{j}^{k}(\varvec{x}) \mathcal {S}_{ij,\text {R}}^{k}(\varvec{x}), \nonumber \\&\mathcal {A}_{\text {L}}^{k}(\varvec{x}) = P_{i} (-1)^{J_{i}-1} B_{L_B}(q_B(\varvec{x}),0) \mathcal {T}_{i}^{k}(\varvec{x}) \mathcal {T}_{j}^{k}(\varvec{x}) \mathcal {S}_{ij, \text {L}}^{k}(\varvec{x}). \end{aligned}$$
(11)

In these expressions, \(P_{i}\) and \(J_{i}\) are the parity and spin of the \(K_{\text {res}}^{i}\) resonance as defined in Eq. 7.

Resonances are described by the product of a normalised Blatt–Weisskopf coefficient and a relativistic Breit–Wigner [13] lineshape,Footnote 4

$$\begin{aligned} \mathcal {T} (s,q,L) = \dfrac{\sqrt{c}B_{L}(q,0)}{m_0^2 - s - im_{0}\varGamma (s,q,L)}, \end{aligned}$$
(12)

where \(m_0\) is the nominal mass of the resonance, q denotes the breakup momentum of the outgoing particle pair in the rest frame of the resonance and \(\varGamma (s,q,L)\) is its energy-dependent width. The normalisation constant

$$\begin{aligned} c = \dfrac{m_{0}\varGamma _{0}\gamma _{0}}{\sqrt{m_{0}^2 + \gamma _{0}}}, \quad \text {with}\quad \gamma _{0} = m_{0}\sqrt{m_{0}^2 + \varGamma _{0}^2}, \end{aligned}$$
(13)

reduces correlations between the coupling to the decay channel and the mass and width of the resonance. The width of the resonance for a decay into two particles is parametrised as

$$\begin{aligned} \varGamma (s,q,L) = \varGamma _{0} \dfrac{m_0}{\sqrt{s}}\bigg (\dfrac{q}{q_0}\bigg )^{2L+1} B_{L}(q,q_{0})^2, \end{aligned}$$
(14)

where \(q_0\) is the value of the breakup momentum at the resonance pole \(s={m_0}^{2}\), and \(B_{L}(q,q_{0})\) is the normalised Blatt–Weisskopf barrier factor, listed in Table 1.

Table 1 Normalised Blatt–Weisskopf centrifugal barrier factors for angular momentum L. The meson radial parameter R is set to \(1.5\,(\text {GeV}/c)^{-1}\) following a measurement by Belle [15]

The spin factors \(\mathcal {S}_{ij,\text {R}/\text {L}}\) are constructed using the Rarita–Schwinger (covariant tensor) formalism, following the method described in Ref. [16]. The spin factors used in this study, as well as a brief description of their computation, are given in Appendix A.

3.3 Amplitude fit

The proposed method to determine the photon polarisation parameter \(\lambda _{\gamma }\) utilises all the degrees of freedom of the system to perform a maximum likelihood fit to the data using a probability density function (PDF) that depends explicitly on \(\lambda _{\gamma }\). This amplitude fit allows the direct measurement of \(\lambda _{\gamma }\), as well as of the relative magnitudes and phases of the different decay-chain amplitudes included in the model. The PDF is computed using the function \(\mathcal {P}_{\text {s}}\) given in Eq. 9 as

$$\begin{aligned} \mathcal {F}(\varvec{x} | \varOmega ) = \dfrac{\xi (\varvec{x}) \mathcal {P}_{\text {s}}(\varvec{x} | \varOmega ) \varPhi _{4}(\varvec{x}) }{\int \xi (\varvec{x}) \mathcal {P}_{\text {s}}(\varvec{x} | \varOmega )\varPhi _{4}(\varvec{x}) {\text {d}}\!\varvec{x}} , \end{aligned}$$
(15)

where \(\varOmega = \lambda _{\gamma }, \{a_{k}\},\{\phi _{k}\} \) is the set of fit parameters, \(\varPhi _{4}(\varvec{x})\) is the four-body phase-space density, and \(\xi (\varvec{x})\) is the efficiency, which accounts for effects related to detector acceptance, reconstruction, and event selection. The magnitude and phase of each amplitude k (\(a_{k}\) and \(\phi _{k}\)) are measured with respect to those of amplitude 1, for which \(a_{1}\) and \(\phi _{1}\) are fixed to 1 and 0, respectively.

The normalisation integral of Eq. 15 is computed numerically using a large sample of simulated events, generated according to an approximate model \(\mathcal {P}_{\text {gen}}\). The signal acceptance \(\xi (\varvec{x})\) is inherently taken into account by applying the event selection used in data to these simulated events; the normalisation integral can then be estimated as

$$\begin{aligned} \int \xi (\varvec{x}) \mathcal {P}_{s}(\varvec{x}| \varOmega ) \varPhi _{4}(\varvec{x}) {\text {d}}\!\varvec{x} = \dfrac{I_{\text {gen}}}{N_{\text {sel}}}\sum _{j}^{N_{\text {sel}}} \dfrac{\mathcal {P}_{s}(\varvec{x}_{j}| \varOmega )}{\mathcal {P}_{\text {gen}}(\varvec{x}_{j})} \end{aligned}$$
(16)

with

$$\begin{aligned} I_{\text {gen}} = \int \xi (\varvec{x}) \mathcal {P}_{\text {gen}}(\varvec{x}) \varPhi _{4}(\varvec{x}) {\text {d}}\!\varvec{x}, \end{aligned}$$
(17)

and where \(N_{\text {sel}}\) is the total number of generated events that pass the selection criteria. Note that \(I_{\text {gen}}\) does not depend on the parameters of the fit, and therefore does not need to be evaluated to perform the maximisation.

For the studies presented here, the effect of the application of a selection is not considered, i.e., \(\xi (\varvec{x}) = 1\).

The fraction of a decay mode k is defined as the ratio of the phase-space integral of the sum of right- and left-handed contributions over the phase-space integral of the function \(\mathcal {P}_{\text {s}}\),

$$\begin{aligned} F_{k}&= (1+\lambda _{\gamma })\frac{\int |f_{k}\mathcal {A}_{k, \text {R}}(\varvec{x})|^2 \varPhi _{4}(\varvec{x}){\text {d}}\!\varvec{x}}{2 \int \mathcal {P}_{s}(\varvec{x}| \varOmega ) \varPhi _{4}(\varvec{x}) {\text {d}}\!\varvec{x}}\nonumber \\&\quad + (1-\lambda _{\gamma })\frac{\int |f_{k}\mathcal {A}_{k, \text {L}}(\varvec{x})|^2 \varPhi _{4}(\varvec{x}){\text {d}}\!\varvec{x}}{2 \int \mathcal {P}_{s}(\varvec{x}| \varOmega ) \varPhi _{4}(\varvec{x}) {\text {d}}\!\varvec{x}} \end{aligned}$$
(18)

Due to interferences between the decay modes, the sum of these fractions may not be equal to unity. The interference term between the decay modes k and l, where \(k > l\), can be expressed as

$$\begin{aligned} F_{kl}&= (1+\lambda _{\gamma }) \frac{\int \mathcal {R}e\left\{ f_{k}\mathcal {A}_{k, \text {R}}(\varvec{x})f^*_{l}\mathcal {A}^*_{l, \text {R}}(\varvec{x})\right\} \varPhi _{4}(\varvec{x}) {\text {d}}\!\varvec{x}}{\int \mathcal {P}_{s}(\varvec{x}| \varOmega ) \varPhi _{4}(\varvec{x}) {\text {d}}\!\varvec{x}}\nonumber \\&\quad + (1-\lambda _{\gamma }) \frac{ \int \mathcal {R}e\left\{ f_{k}\mathcal {A}_{k, \text {L}}(\varvec{x})f^*_{l}\mathcal {A}^*_{l, \text {L}}(\varvec{x})\right\} \varPhi _{4}(\varvec{x}) {\text {d}}\!\varvec{x}}{\int \mathcal {P}_{s}(\varvec{x}| \varOmega ) \varPhi _{4}(\varvec{x}) {\text {d}}\!\varvec{x}}\,, \end{aligned}$$
(19)

such that the sum of all the fractions and interference terms is equal to unity:

$$\begin{aligned} \sum _k F_{k} + \sum _{k > l} F_{kl} = 1. \end{aligned}$$
(20)

4 Sensitivity

The amplitude formalism described in Sect. 3 is implemented in a generator and fitter software framework developed for the amplitude analysis of \(D^0 \rightarrow K^+ K^- {{\pi } ^+} {{\pi } ^-} \) decays at CLEO [16, 17]. The performance of the amplitude fitter is studied initially by generating and subsequently fitting simulated data sets of \(B\!\rightarrow K\pi \pi {\gamma } \) decays using models containing two or three amplitudes. Once the methodology is validated, more realistic models of the \({K} {\pi } {\pi } \) system are used in order to obtain prospects for measurements of the photon polarisation parameter in B-physics experiments.

Fig. 3
figure 3

Uncertainty on \(\lambda _{\gamma }\) obtained from amplitude fits of simulated samples of \(B \rightarrow K_1(1270)^+ \gamma \) decays governed by two amplitudes only, \(K_1(1270)^+ \rightarrow {{K} ^+} \rho (770)^0\) and \(K_1(1270)^+ \rightarrow K^{*}(892)^0 \pi ^+\), shown as a function of the relative fraction (radial coordinate, from 0.1 to 9.0) and phase (polar coordinate) of the two amplitudes. Each bin contains the average uncertainty of 10 amplitude fits performed on samples generated with the same model

Fig. 4
figure 4

Uncertainty on \(\lambda _{\gamma }\) obtained from amplitude fits of simulated samples of \(B \rightarrow K_1(1270)^+ \gamma \) decays governed by two amplitudes only, \(K_1(1270)^+ \rightarrow {{K} ^+} \rho (770)^0\) and \(K_1(1270)^+ \rightarrow K^{*}(892)^0 \pi ^+\), shown as a function of the generated \(\lambda _{\gamma }\) value and the relative fraction (left) or phase difference (right) of the two amplitudes. Each bin contains the average uncertainty of 10 amplitude fits performed on samples generated with the same model

4.1 Proof-of-concept using simplified models

As illustrated in Fig. 2, the sensitivity to the photon polarisation parameter obtained from the up-down asymmetry depends primarily on the relative phase. The same set of simplified models of the \({{{B} ^+}} \!\rightarrow K_1(1270)^+\gamma \) channel, which include the two previously studied decay modes \(K_1(1270)^+\!\rightarrow {{K} ^+} \rho (770)^0\) and \(K_1(1270)^+\!\rightarrow K^{*}(892)^0{{\pi } ^+} \), is used to test the performance of the full amplitude fit, as well as its stability and the accuracy of the obtained uncertainties. The free parameters of the fit are the photon polarisation parameter \(\lambda _{\gamma }\), and the modulus and phase associated with the \(K_1(1270)^+\!\rightarrow {{K} ^+} \rho (770)^0\) channel, hereafter referred to as the relative magnitude and phase, where the \(K_1(1270)^+\!\rightarrow K^{*}(892)^0{{\pi } ^+} \) channel is chosen as a reference.

For each pair of relative magnitude and phase considered, 10 simulated data sets of 8 000 events are generated with \(\lambda _{\gamma } =+1\) (close to the SM value) and fitted independently. The average uncertainty on \(\lambda _{\gamma }\) as a function of relative fraction (as defined in Eq. 18) and phase is shown in Fig. 3, where areas of higher colour saturation indicate regions with higher sensitivity to \(\lambda _{\gamma }\): unlike \(\mathcal {A}_\mathrm{{ud}}\), the amplitude analysis is sensitive to \(\lambda _{\gamma }\) for all values of relative fractions and phases, with statistical uncertainties ranging from 0.01 to 0.05. A higher average uncertainty on \(\lambda _{\gamma }\) is seen for models in which the fraction of one amplitude is much larger than the other, and the maximum sensitivity is obtained for a phase difference of around \(3\pi /2\) and a relative fraction of 1.5.

To evaluate the performance of the fit as a function of the photon polarisation parameter, the study is repeated for various generated values of \(\lambda _{\gamma }\), and the results are shown in Fig. 4. The highest sensitivities to \(\lambda _{\gamma }\) are obtained for \(\lambda _{\gamma } =\pm 1\), with increasing uncertainties observed as the generated absolute value of \(\lambda _{\gamma }\) decreases.

To study the fit accuracy and error estimation, 100 simulated data sets are generated and fitted for selected values of the model parameters (relative magnitude, relative phase and \(\lambda _{\gamma }\)). As asymmetric uncertainties are used in these fits, the quality of the parameter estimation is evaluated by checking that the distribution of the pull variable g is compatible with a standard normal distribution, where g is defined as

$$\begin{aligned} g=\frac{\text {(true value)}-\text {(fit result)}}{\text {(positive\, uncertainty)}}, \end{aligned}$$
(21)

if \(\text {(fit result)}\le \text {(true value)}\), and

$$\begin{aligned} g=\frac{\text {(fit result)}-\text {(true value)}}{\text {(negative\, uncertainty)}}, \end{aligned}$$
(22)

otherwise.

The mean values and standard deviations of the fitted parameters and the associated pull parameters can be found in Tables 7,  8, and 9 of Appendix B. For all models, each fit parameter has a Gaussian distribution centered on the generated value with a pull distribution of width consistent with unity, resulting in an unbiased measurement and correct error estimation.

Fig. 5
figure 5

Uncertainty on \(\lambda _{\gamma }\) obtained from amplitude fits of simulated samples of \(B^{0} \rightarrow K_1(1270)^0 \gamma \) decays, shown as a function of the phase differences of the \(K_1(1270)^0 \rightarrow {{K} ^+} \rho (770)^-\) and \(K_1(1270)^0 \rightarrow K^{*}(892)^0 \pi ^0\) decay modes relative to the \(K_1(1270)^0 \rightarrow K^{*}(892)^+ \pi ^-\) decay mode, denoted as \(\phi _{1}\) and \(\phi _{2}\) respectively. Each bin contains the average uncertainty of 10 amplitude fits performed on samples generated with the same model

Table 2 Model used to describe the \(K_{\text {res}}^+ \rightarrow K^+ \pi ^- \pi ^+\) hadronic system in the \({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) decays. The table is divided in sections according to the spin-parity \(J^{P}\) of the \({K} _\mathrm{{res}}\) resonance. The amplitude with the S-wave decay \(K_1(1270)^+ \rightarrow K^{*}(892)^0 \pi ^+\) is chosen as a reference for the magnitudes and phases
Fig. 6
figure 6

Squared invariant-mass (\(m^2_{K^+{{\pi } ^-} {{\pi } ^+}}, m^2_{K^+{{\pi } ^-}}, m^2_{{{\pi } ^-} {{\pi } ^+}}\)) and angular (cos\(\,\theta \) and \(\chi \)) distributions for a single data set of 14 000 \({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) decays generated with the 14 amplitudes listed in Table 2. The red histograms represent the projections of the PDF obtained from the fit

As a final test, we study decays of B mesons to \(K\pi \pi \gamma \) with a \(\pi ^0\) in the final state, which can have an additional source of interference from intermediate states that include a \(K^{*}(892)\) resonance. It has been claimed that the presence of these additional interference terms results in a higher maximum possible up-down asymmetry [7], and thus that the analysis of \({{B} ^0} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^0} {\gamma } \) decays could be potentially more sensitive to the photon polarisation than that of \({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) decays. The effect of an additional decay amplitude (and therefore additional interference terms) is studied using a \({{B} ^0} \!\rightarrow K_1(1270)^0 \gamma \) model with three different \(K_1(1270)^0\) decay channels, \(K_1(1270)^0\!\rightarrow {{K} ^+} \rho (770)^-\), \(K_1(1270)^0\!\rightarrow K^{*}(892)^+ \pi ^-\), and \(K_1(1270)^0\!\rightarrow K^{*}(892)^0 \pi ^0\). Ten simulated data sets, each containing 8 000 events, are generated for different values of the phase differences of the \(K_1(1270)^0\!\rightarrow {{K} ^+} \rho (770)^-\) and \(K_1(1270)^0\!\rightarrow K^{*}(892)^0 \pi ^0\) amplitudes relative to the \(K_1(1270)^0\!\rightarrow K^{*}(892)^+ \pi ^-\) mode; all samples are generated with \(\lambda _{\gamma } =+1\), with the decay rate for all amplitudes being equal. The uncertainty on the photon polarisation parameter for all models studied, shown in Fig. 5, is within the same range as seen in the two-amplitude \({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) model, showing that the amplitude analysis is not very sensitive to the number of interference terms in the \({K} {\pi } {\pi } \) system. We conclude that this amplitude analysis is sensitive to the photon polarisation parameter for all simplified models studied, for both charged and neutral decay modes.

Table 3 Pull parameters of the fit to \({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) samples for all magnitudes and phases relative to the amplitude with the S-wave decay \(K_1(1270)^+ \rightarrow K^{*}(892)^0 \pi ^+\)

4.2 Prospects for future measurements

4.2.1 \({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) decays

In light of the results of the proof-of-concept model, the most promising measurement of the photon polarisation parameter is expected to come from \({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) decays, which are the most abundantly reconstructed at LHCb and Belle II.

An estimate of the statistical sensitivity of a measurement of the photon polarisation from an amplitude analysis of \({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) decays is obtained by studying the model described in Table 2, which provides a good approximation to the \(K\pi \), \(\pi \pi \) and \({K} {\pi } {\pi } \) invariant mass spectra observed in a data sample of \(3\,\,\hbox {fb}^{-1} \) collected by LHCb during Run 1 of the LHC [8, 18]. A total of 100 data sets of 14 000 events each, corresponding to the LHCb signal yield of Run 1 [8], are generated with \(\lambda _{\gamma } =+1\). The fits of these samples yield a mean uncertainty on \(\lambda _{\gamma }\) of 0.015. Figure 6 shows the distributions for the five variables for one of these simulated data sets along with the corresponding projections of the fit PDF. The pull means (\(\mu _{\text {pull}}\)) and widths (\(\sigma _{\text {pull}}\)) of the complex coefficients \(a_{k}\) and \(\phi _k\), listed in Table 3, show that the uncertainties are mostly well estimated. The pull distribution associated with \(\lambda _{\gamma }\) has a mean of \(-0.38 \pm 0.11\) and a width of \(1.18 \pm 0.08\), indicating that the obtained uncertainty on \(\lambda _{\gamma }\) is underestimated by about \(20\%\), and that there is an evidence for a bias in the fitted value of \(\lambda _{\gamma }\) which amounts to around \(40\%\) of the statistical uncertainty. This bias cannot be linked to a specific fit parameter as the magnitude of the correlation coefficients between \(\lambda _{\gamma }\) and the other fit parameters typically lie below \(20\%\). Taking into account a corrected uncertainty of 0.018, the comparison of this result with the simplified models discussed in the previous section suggests that the model complexity does not have a large effect on the sensitivity to \(\lambda _{\gamma }\).Footnote 5 This fact can be used to evaluate the gain in sensitivity that could be obtained by exploiting the additional \(6\,\,\hbox {fb}^{-1} \) of data that have been recorded by LHCb at a pp energy of \(13\,\)TeV in Run 2, where the B production cross-section is almost twice that at the Run 1 energy of \(7-8\,\)TeV: assuming that a total of 70 000 signal decays are selected using the LHCb Run 1 and Run 2 data sets, the resulting corrected statistical uncertainty on the measurement of the photon polarisation parameter could reach 0.009. The bias in the value of \(\lambda _{\gamma }\) would have to be corrected for or accounted as a systematic uncertainty.

Table 4 Model used to describe the \(K_{\text {res}} \rightarrow K^+ \pi ^- \pi ^0\) hadronic system in \({{B} ^0} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^0} {\gamma } \) decays. The table is divided in sections according to the spin-parity \(J^{P}\) of the \({K} _\mathrm{{res}}\) resonance. The amplitude with the S-wave decay \(K_1(1270)^0 \rightarrow K^{*}(892)^0 \pi ^0\) is chosen as a reference for the magnitudes and phases

4.2.2 \({{B} ^0} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^0} {\gamma } \) decays

As discussed in Sect. 4.1, \({{B} ^0} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^0} {\gamma } \) decays can also be used to measure the photon polarisation parameter. The main difference with the \({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) decays used above is that the hadronic part of the decays is a priori more complex due to an additional source of interference involving \(K^{*}(892)^0 \pi ^0\) and \(K^{*}(892)^+ \pi ^-\) intermediate states in the decays of the heavy kaonic resonances \(K_{\text {res}} \rightarrow K^+ \pi ^- \pi ^0\).

Samples of 10 000 simulated signal events (corresponding to the number of expected \({{B} ^0} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^0} {\gamma } \) decays to be reconstructed by Belle II with \(5\,\hbox {\,ab}^{-1} \) of integrated luminosity) are used to evaluate the sensitivity of a measurement of the photon polarisation parameter using \({{B} ^0} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^0} {\gamma } \) decays. As little is known about the hadronic system in such decays, a model of the \({K} {\pi } {\pi } \) system is obtained from the model used for the charged modes, assuming the relative magnitudes and phases of all allowed decay modes without a \(K^{*}(892)\pi \) to be identical to those of the charged mode. In the case of modes with intermediate states that include a kaonic resonance and a pion, the branching fraction is divided equally between the \(K_1(1270)^0\!\rightarrow K^{*}(892)^0 (\rightarrow {{K} ^+} {{\pi } ^-}){{\pi } ^0} \) and \(K_1(1270)^0\!\rightarrow K^{*}(892)^+(\rightarrow {{K} ^+} {{\pi } ^0}) {{\pi } ^-} \) modes assuming isospin conservation. The unknown phase differences are set to the same values for both modes, which is satisfactory in the absence of a strong dependence of the sensitivity of the measurement on the phase difference. The resulting model, containing 23 amplitudes, is presented in Table 4 and distributions from a single simulated data set are shown in Fig. 7, along with the corresponding fit PDF projections.

Fig. 7
figure 7

Squared invariant-mass (\(m^2_{K^+{{\pi } ^0} {{\pi } ^-}}, m^2_{K^+{{\pi } ^0}}, m^2_{{{\pi } ^0} {{\pi } ^-}}\)) and angular (cos\(\,\theta \) and \(\chi \)) distributions for a single data set of 10 000 \({{B} ^0} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^0} {\gamma } \) decays generated with the 23 amplitudes listed in Table 4. The red histograms represent the projections of the PDF obtained from the fit

Using the same procedure as for the charged mode, an uncertainty on the measurement of the photon polarisation of 0.016 is obtained from simulated signal samples. The pull mean of \(-\,0.64 \pm 0.11\) shown in Table 5 corresponds to the observation of a bias in the fitted value of \(\lambda _{\gamma }\) which amounts to around \(60\%\) of the statistical uncertainty obtained from the fit, where the magnitude of the correlation coefficients with the other fit parameters typically lies below \(20\%\). This bias would have to be corrected for in the final result of the fit, or to be taken into account as a systematic uncertainty. The associated pull width of \(1.19 \pm 0.08\) indicates that this uncertainty is also underestimated by around \(20\%\); the corrected value of 0.019 is comparable to the one obtained with the charged mode, confirming that the additional interference patterns and the higher complexity of the \({K} {\pi } {\pi } \) system do not provide a significant improvement on the precision of the measurement. As a higher number of signal events is expected for the charged mode, our method would perform better using these decays, but the amplitude analysis of the neutral mode would provide a very interesting independent measurement of the \(\lambda _{\gamma }\) parameter.

Table 5 Pull parameters of the fit to \({{B} ^0} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^0} {\gamma } \) samples for all magnitudes and phases relative to the amplitude with the S-wave decay \(K_1(1270)^0 \rightarrow K^{*}(892)^0 \pi ^0\)

5 Conclusions

A new method to measure the photon polarisation parameter in \(B\!\rightarrow K\pi \pi {\gamma } \) decays from an amplitude analysis is presented. Using simplified models of the hadronic part of the decay, it is shown that the sensitivity of the photon polarisation parameter measurement does not depend strongly on the configuration or complexity of the \({K} {\pi } {\pi } \) system.

The performed studies demonstrate that, in the ideal case of a background-free sample without distortions due to experimental effects, and ignoring the differences between non-factorisable hadronic parameters between the resonances in the \({K} {\pi } {\pi } \) system, this method allows the measurement of the photon polarisation with a statistical uncertainty of around 0.009 on a sample of 70 000 \({{{B} ^+}} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^+} {\gamma } \) decays corresponding to the signal statistics assumed for LHCb in Runs 1 and 2. Belle II is assumed to reconstruct about 10 000 \({{B} ^0} \!\rightarrow {{K} ^+} {{\pi } ^-} {{\pi } ^0} {\gamma } \) decays with a data set corresponding to an integrated luminosity of \(5\,\hbox {\,ab}^{-1} \). The analysis of these data could also determine independently the photon polarisation with a statistical uncertainty of the order of 0.019, again ignoring background and experimental effects, as well as non factorisable hadronic uncertainties.

The uncertainty on the measurement of the photon polarisation parameter \(\lambda _{\gamma }\) can be translated in terms of constraints on the Wilson coefficients \(C_{7}^{\text {eff}}\) and \(C^{\prime }_{7}\) using Eq. 8. In principle, the same method would also apply in the presence of process independent corrections to the Wilson coefficients and could also be translated in terms of \(C_{7}^{\text {eff}}\) and \(C^{\prime }_{7}\) with theoretical input on these corrections. These constraints could then be compared to those set by other relevant observables such as the \({{B} ^0} \!\rightarrow {{K} ^{*0}} {e ^+e ^-} \) angular observables, the time-dependent decay rate of \({{B} ^0_{s}} \!\rightarrow \phi \gamma \) decays, the \(C\!P\) asymmetry in \({{B} ^0} \!\rightarrow {{K} ^{*0}} \gamma \) decays or the inclusive \({B} \!\rightarrow X_s\gamma \) branching fraction, which are discussed extensively in Ref. [9]. While the particular dependence of \(\lambda _{\gamma }\) on the Wilson coefficients makes this observable a priori less interesting to size non-SM effects, the statistical power of the studies shown here will compensate this limitation. Additionally, since the dependence of \(\lambda _{\gamma }\) on the Wilson coefficients is different from that of the other observables, its measurement provides complementary information; in particular, a measurement of \(\lambda _{\gamma }\) in \(B\!\rightarrow K\pi \pi {\gamma } \) decays could help break an ambiguity that arises in the determination of \(\mathcal {R}e(C_{7}^\prime )\) when constraints from all radiative observables are combined assuming both Wilson coefficients to be real [9].

However, as already mentioned, theory calculations of the hadronic contributions are crucial to be able to perform this interpretation of \(\lambda _{\gamma }\) in terms of the Wilson coefficients. Additionally, the effect of the process dependent corrections, which are disregarded at the moment, should be estimated or taken into account as nuisance parameters.

In summary, the measurement of the photon polarisation parameter through an amplitude analysis of \(B\!\rightarrow K\pi \pi {\gamma } \) decays is a very promising method that could exploit the large data samples available at LHCb and Belle II in the near future. If the current shortcomings in the interpretation are overcome, the proposed approach will allow to set very competitive and complementary new constraints on the Wilson coefficients \(C_{7}^{\text {eff}}\) and \(C_{7}^\prime \), and will pave the way to a new array of measurements involving decays of \({b} \) hadrons to three hadrons and a photon.