1 Introduction

The Standard Model (SM) description of charge-parity (\(C\!P\)) violation can be tested by measuring the lengths and angles of the Unitary Triangle, which is a geometrical representation of the Cabibbo-Kobayashi-Maskawa quark-mixing matrix [1, 2]. In particular, the \(C\!P\)-violating phase \(\gamma \equiv \arg (-{V_{ud}^{}} {V_{ub}^*}/{V_{cd}^{}} {V_{cb}^*})\) is the only angle that can be measured at tree level with negligible theoretical uncertainties [3]. Therefore, it makes an excellent SM benchmark that can be compared with other indirect measurements of \(\gamma \) that are more likely to be affected by physics beyond the SM.

A powerful decay channel for the measurement of \(\gamma \) is \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \), which proceeds through both favoured \(b\rightarrow c{\bar{u}}s\) and a suppressed \(b\rightarrow u{\bar{c}}s\) transitions. Interference occurs when the \({D} \) meson, which is a superposition of the \({{D} ^0} \) and \({{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \) states, decays to a final state common to both of these flavour eigenstates. The interference effects are sensitive to \(\gamma \), which can in general be determined from measurements of the appropriate \(C\!P\) asymmetries and related observables. This strategy has been pursued for a wide range of \({D} \) final states at LHCb [4,5,6,7,8] and other b-physics experiments [9,10,11]. Effects from \(C\!P\)-violation also occur in the process \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) through the interference of \(b\rightarrow c{\bar{u}}d\) and \(b\rightarrow u{\bar{c}}d\) transitions, but these are in general significantly smaller in magnitude.

An interesting class of \({D} \) final states are self-conjugate multi-body \({D} \) decays. Since the strong-phase difference between the \({{D} ^0} \) and \({{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} \) decays varies across the multi-dimensional phase space of the charm-meson decay, the sensitivity to \(\gamma \) is diluted when considering the decay inclusively. However, by performing measurements in suitably chosen localised regions of phase space, the dilution effects can be minimised and the sensitivity to \(\gamma \) enhanced [12,13,14,15].

The \({D} \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} \) decay mode has been proposed as a promising decay mode for measuring \(\gamma \) [16]. It has a rich resonance structure and contains only charged particles in the final state, which is advantageous for experiments at a hadron collider. The availability of a detailed amplitude model for this process, based on an analysis of LHCb data [17], opens up the possibility of identifying those regions of phase space that have high sensitivity to \(\gamma \), which then can be probed in an analysis of \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) decays. The interpretation of the \({C\!P} \) asymmetries and other observables in these regions requires knowledge of the strong-phase difference between the \({{D} ^0} \) and \({{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \) meson decays. This knowledge can be obtained from the same amplitude model used to guide the measurements, but it is preferable to take the information from direct determinations made by experiments at charm threshold, such as BESIII [18], as this approach ensures the determination of \(\gamma \) has no dependence on model assumptions. An analogous study has recently been performed using the decay mode \(D \rightarrow {{K} ^\pm } {{\uppi } ^\mp } {{\uppi } ^\pm } {{\uppi } ^\mp } \) [19].

This paper presents the first study of \(C\!P\) violation in \({{B} ^\pm } \rightarrow [{{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} h^\pm \) decays, with \(h = K, \pi \). The analysis exploits proton–proton (pp) collision data collected by LHCb in Runs 1 and 2 of the LHC, corresponding to 9 \(\text {\,fb} ^{-1}\) of integrated luminosity. The study is performed in localised regions of phase space, defined with guidance from the amplitude model presented in Ref. [17], and the measurements are used to extract a value of \(\gamma \), using model predictions for the strong-phase variation in the charm-meson decay. In addition, first measurements of the global \(C\!P\) asymmetries are made for this decay and global measurements are updated for the mode \(B^\pm \rightarrow [\pi ^+\pi ^-\pi ^+\pi ^-]_D h^\pm \) with respect to those reported in Ref. [4], which are also interpreted in terms of \(\gamma \) and the underlying physics parameters.

2 Analysis strategy

This analysis follows the formalism described in Ref. [6]. The \({{{B} ^-}} \rightarrow [{{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} {{K} ^-} \) decay can proceed via the favoured \({{{B} ^-}} \rightarrow {{D} ^0} {{K} ^-} \) amplitude, or via the suppressed \({{{B} ^-}} \rightarrow {{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} {{K} ^-} \) amplitude. The overall amplitude of this decay is a coherent sum of the two decay paths,

$$\begin{aligned} {\mathcal {A}}_{{{B} ^-}} (\Phi ) = {\mathcal {A}}_{{{B} ^-}} ^{{{D} ^0} {{K} ^-}}\Big ({\mathcal {A}}_{{D} ^0} (\Phi ) + r_{B} ^{{D} {K}}\exp \big (i(\delta _{B} ^{{D} {K}} - \gamma )\big ){\mathcal {A}}_{{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} (\Phi )\Big ), \nonumber \\ \end{aligned}$$
(1)

where \({\mathcal {A}}_{{{B} ^-}} ^{{{D} ^0} {{K} ^-}}\) is the amplitude of the favoured \({{{B} ^-}} \rightarrow {{D} ^0} {{K} ^-} \) decay, \({\mathcal {A}}_{{D} ^0} \) (\({\mathcal {A}}_{{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \)) is the amplitude of the \({{D} ^0} \) (\({{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \)) decay, \(r_{B} ^{{D} {K}}\) is the magnitude of the ratio of the \({B} \)-decay amplitudes and \(\delta _{B} ^{{D} {K}}\) is the strong-phase difference of the amplitudes. Here \(\Phi \) labels the position in the five-dimensional phase space of the decay \({{D} ^0} \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} \). The corresponding expression for the \({{{B} ^+}} \) decay is obtained by making the substitutions \(\gamma \rightarrow -\gamma \) and \({\mathcal {A}}_{{D} ^0} \leftrightarrow {\mathcal {A}}_{{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \). The \(C\!P\)-conjugated amplitude \({\mathcal {A}}_{{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} (\Phi )\) is equal to \({\mathcal {A}}_{{D} ^0} ({\bar{\Phi }})\), where \({\bar{\Phi }}\) is obtained by swapping the charges and momentum directions of the \({D} \)-decay products. Here and in subsequent discussion \(C\!P\) violation in the \({D} \)-meson system is neglected, which is a good assumption at the current level of experimental sensitivity [20].

The \({D} \)-decay phase space is split into \(2\times {\mathcal {N}}\) bins, labelled from \(i = -{\mathcal {N}}\) to \(i = {\mathcal {N}}\), excluding zero. A \(C\!P\) transformation relates the bins with indices \(-i\) and \(+i\). The choice of binning scheme is described in Sect. 3. The expected yield of \({{{B} ^-}} \) decays in bin i is obtained by integrating Eq. (1), and the corresponding expression for \({{{B} ^+}} \) decays, over the phase space \(\Phi _i\) that belongs to that particular bin. Defining the \(C\!P\)-violating observables

$$\begin{aligned} x_\pm ^{{D} {K}} \equiv r_{B} ^{{D} {K}}\cos (\delta _{B} ^{{D} {K}}\pm \gamma ), \quad y_\pm ^{{D} {K}} \equiv r_{B} ^{{D} {K}}\sin (\delta _{B} ^{{D} {K}}\pm \gamma ), \nonumber \\ \end{aligned}$$
(2)

the yields \(N_i^\pm \) of \({{B} ^\pm } \) candidates in bin i are given by

$$\begin{aligned} N_{+i}^+= & {} h_{{{B} ^+}} ^{{D} {K}}\Big (F_{-i} + \big ((x_+^{{D} {K}})^2 + (y_+^{{D} {K}})^2\big )F_{+i} \nonumber \\{} & {} \quad + 2\sqrt{F_{+i}F_{-i}}\big (x_+^{{D} {K}}c_i - y_+^{{D} {K}}s_i\big )\Big ), \end{aligned}$$
(3)
$$\begin{aligned} N_{-i}^-= & {} h_{{{B} ^-}} ^{{D} {K}}\Big (F_{-i} + \big ((x_-^{{D} {K}})^2 + (y_-^{{D} {K}})^2\big )F_{+i} \nonumber \\{} & {} \quad + 2\sqrt{F_{+i}F_{-i}}\big (x_-^{{D} {K}}c_i - y_-^{{D} {K}}s_i\big )\Big ), \end{aligned}$$
(4)

where \(h_{{B} ^\pm } ^{{D} {K}}\) are normalisation constants. Since \(h_{{{B} ^+}} ^{{D} {K}}\) and \(h_{{{B} ^-}} ^{{D} {K}}\) are independent fit parameters, the binned measurement is insensitive to the \({{B} ^\pm } \) production asymmetry and any charge asymmetry in the detection efficiency of the kaon that accompanies the \({D} \) meson.

Equations (3) and (4) are sensitive to \(\gamma \) through the interference terms, and the magnitude of the interference is determined by the size of \(r_{B} ^{{D} {K}}\), which has been measured to be \(\approx 0.1\) [21]. The parameters \(F_i\) are defined as

$$\begin{aligned} F_i\equiv \frac{\int _i\mathop {}\!\textrm{d}\Phi \eta (\Phi )|{\mathcal {A}}_{{D} ^0} |^2}{\int \mathop {}\!\textrm{d}\Phi \eta (\Phi )|{\mathcal {A}}_{{D} ^0} |^2}, \end{aligned}$$
(5)

which are interpreted as the fractional yield of \({{D} ^0} \) decays in bin i measured in this analysis. The function \(\eta (\Phi )\) accounts for the detection efficiency, which in general depends on the location of the decay in phase space. The strong-phase information is encoded in the parameters \(c_i\) and \(s_i\), where

$$\begin{aligned} c_i\equiv \frac{\int _i\mathop {}\!\textrm{d}\Phi |{\mathcal {A}}_{{D} ^0} ||{\mathcal {A}}_{{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} |\cos (\Delta \delta _{D})}{\sqrt{\int _i\mathop {}\!\textrm{d}\Phi |{\mathcal {A}}_{{D} ^0} |^2\int _i\mathop {}\!\textrm{d}\Phi |{\mathcal {A}}_{{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} |^2}}, \end{aligned}$$
(6)

which is the amplitude-averaged cosine of the strong-phase difference \(\Delta \delta _{D} = \delta _{D} (\Phi ) - \delta _{D} ({\bar{\Phi }})\) of the \({D} \) decay. The expression for the amplitude-averaged sine of the strong-phase difference \(s_i\) is analogous. It follows that \({\bar{F}}_i = F_{-i}\), \(c_{-i} = c_i\) and \(s_{-i} = -s_i\), where \({\bar{F}}_i\) are the corresponding \({{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \) fractional bin yields. The values of \(c_i\) and \(s_i\) are currently taken from an amplitude model [17]. In the future it is expected that direct measurements of \(c_i\) and \(s_i\) will become available from a sample of correlated \({D} {\hspace{0.0pt}\overline{\hspace{0.0pt}D}} \) decays collected by the BESIII experiment at charm threshold [18]. The anticipated size of this data sample at charm threshold leads to the choice of \({\mathcal {N}} = 8\) for the number of bins.

Analogous expressions to Eqs. (2), (3) and (4) can be written for the decay mode \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \), with the \({D} {K} \) superscripts replaced by \({D} {\uppi } \). Since the decay topology is identical to that of \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \), the phase-space acceptance \(\eta (\Phi )\) is expected to be very similar between the two \({{B} ^\pm } \)-decay modes, and studies using simulation samples show that any differences are negligible within the current precision. Thus, the \(F_i\) parameters can be considered as common between \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) and \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) decays. The mode \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) has a branching fraction that is an order of magnitude larger than that of the \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) mode, but the interference effects, governed by the parameter \(r_{B} ^{{D} {\uppi }}\approx 0.005\) [21], are much smaller. Therefore, this decay has a significantly lower sensitivity to \(\gamma \), but is a suitable mode for determining the \(F_i\) parameters. By including the \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) channel as a signal mode, the \(F_i\) can be treated as free parameters in the analysis, thus the form of the acceptance function \(\eta (\Phi )\) is not needed.

From the definition of Eq. (5), it follows that \(\sum _iF_i = 1\). Therefore, only \(2{\mathcal {N}} - 1\) of the \(F_i\) parameters are independent. To accommodate this constraint in the analysis, the \(F_i\) parameters are fitted with the alternative parameterisation in terms of the recursive fractions \(R_i\),

$$\begin{aligned} F_i \equiv {\left\{ \begin{array}{ll} R_i, &{} i = -{\mathcal {N}} \\ R_i\prod _{j< i}(1 - R_j), &{} -{\mathcal {N}}< i< +{\mathcal {N}} \\ \prod _{j < i}(1 - R_j), &{}i = +{\mathcal {N}}. \end{array}\right. } \end{aligned}$$
(7)

Measuring the yields of \({{B} ^\pm } \) decays in each bin of phase space allows the eight \(C\!P\)-violating observables \(x_\pm ^{{D} {K}}\), \(y_\pm ^{{D} {K}}\), \(x_\pm ^{{D} {\uppi }}\) and \(y_\pm ^{{D} {\uppi }}\) to be determined. These \(C\!P\)-violating observables can be interpreted in terms of the five underlying physics parameters \(\gamma \), \(\delta _{B} ^{{D} {K}}\), \(r_{B} ^{{D} {K}}\), \(\delta _{B} ^{{D} {\uppi }}\) and \(r_{B} ^{{D} {\uppi }}\). Pseudoexperiments with all eight CP-violating observables as free parameters exhibit unstable fit results for \(r_{B} ^{{D} {\uppi }} < 0.03\), due to large correlations between the \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) \(C\!P\)-violating observables and the \(F_i\) parameters [6]. Since \(\gamma \) is a common parameter between the \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) and \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) analyses, the \(C\!P\)-violating observables for the \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) mode can be re-parameterised as

$$\begin{aligned} x_\xi ^{{D} {\uppi }}= & {} \Re \big (\xi ^{{D} {\uppi }}\big ), \quad y_\xi ^{{D} {\uppi }} = \Im \big (\xi ^{{D} {\uppi }}\big ), \quad \nonumber \\ \xi ^{{D} {\uppi }}= & {} \frac{r_{B} ^{{D} {\uppi }}}{r_{B} ^{{D} {K}}}\exp \Big (i\big (\delta _{B} ^{{D} {\uppi }} - \delta _{B} ^{{D} {K}}\big )\Big ). \end{aligned}$$
(8)

In summary, a measurement of the yields of \({{B} ^\pm } \rightarrow [{{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} h^\pm \) decays in bins of phase space allows the six \(C\!P\)-violating observables \(x_\pm ^{{D} {K}}\), \(y_\pm ^{{D} {K}}\), \(x_\xi ^{{D} {\uppi }}\) and \(y_\xi ^{{D} {\uppi }}\) to be determined, along with the experiment-specific parameters \(F_i\), expressed in terms of \(R_i\), and the four normalisation parameters \(h_{{B} ^\pm } ^{{D} h}\).

Additionally, \(\gamma \) can be further constrained by measuring phase-space integrated \(C\!P\)-violating observables [22, 23]. These observables bring information that is independent of the binned analysis, which is only sensitive to the relative variation in yields across phase space. In the case of \({D} \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} \) decays the phase-space integrated observables are the charge asymmetries

$$\begin{aligned} A^{KK\pi \pi }_h \equiv \frac{\Gamma ({{{B} ^-}} \rightarrow {D} h^-) - \Gamma ({{{B} ^+}} \rightarrow {D} h^+)}{\Gamma ({{{B} ^-}} \rightarrow {D} h^-) + \Gamma ({{{B} ^+}} \rightarrow {D} h^+)}, \end{aligned}$$
(9)

for \(h = \pi \) or K, and the double ratio

$$\begin{aligned} R^{KK\pi \pi }_{{C\!P}}\equiv & {} \frac{R_{{K} {K} {\uppi } {\uppi }}}{R_{K{\uppi } {\uppi } {\uppi }}}, \nonumber \\ R_f\equiv & {} \frac{\Gamma ({{{B} ^-}} \rightarrow [f]_{D} {{K} ^-}) + \Gamma ({{{B} ^+}} \rightarrow [f]_{D} {{K} ^+})}{\Gamma ({{{B} ^-}} \rightarrow [f]_{D} {{\uppi } ^-}) + \Gamma ({{{B} ^+}} \rightarrow [f]_{D} {{\uppi } ^+})}, \nonumber \\ \end{aligned}$$
(10)

where in the case of \(R_{K{\uppi } {\uppi } {\uppi }}\) the kaon from the D-meson decay has the same charge as the pion or kaon from the B-meson decay. The value of \(R_{K{\uppi } {\uppi } {\uppi }}\) can be determined from the phase-space integrated yields obtained in Ref. [19]. By integrating Eqs. (3) and (4) over all bins i, it can be shown that these phase-space integrated \(C\!P\)-violating observables may be expressed in terms of the underlying physics parameters,

$$\begin{aligned} A^{KK\pi \pi }_h&= \frac{2r_{B} ^{{D} h}\kappa \sin (\delta _{B} ^{{D} h})\sin (\gamma )}{1 + (r_{B} ^{{D} h})^2 + 2r_{B} ^{{D} h}\kappa \cos (\delta _{B} ^{{D} h})\cos (\gamma )}, \end{aligned}$$
(11)
$$\begin{aligned} R^{KK\pi \pi }_{{C\!P}}&= 1 + (r_{B} ^{{D} {K}})^2 + 2r_{B} ^{{D} {K}}\kappa \cos (\delta _{B} ^{{D} {K}})\cos (\gamma ), \end{aligned}$$
(12)

where \(\kappa = 2F^{KK\pi \pi }_+ - 1\) is the dilution factor when integrating over all of phase space, with \(F^{KK\pi \pi }_+\) being the \(C\!P\)-even fraction of the decay. In Eqs. (11) and (12), the small effects of charm mixing have been neglected, but these may readily be included [24]. Analogous observables \(A^{\pi \pi \pi \pi }_h\) and \(R^{\pi \pi \pi \pi }_{{C\!P}}\) exist for the decay \(D \rightarrow {{\uppi } ^+} {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^-} \).

3 Binning scheme

In the \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \), \({D} \rightarrow {{K} ^0_{\textrm{S}}} h^+h^-\) (\(h={\uppi },{K} \)) analysis presented in Ref. [6], an optimal binning scheme, which can be visualised in a two-dimensional Dalitz plot, is used to maximise the sensitivity to \(\gamma \). The binning scheme for the \({{D} ^0} \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} \) decay is defined analogously, but cannot be easily visualised, as the phase space of four-body decays is five-dimensional.

When defining a binning scheme for the decay \({D} \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} \), there are two main requirements. Firstly, it should minimise the dilution of the strong phases when averaged over each bin. Secondly, it should maximise the interference effects and thus the sensitivity to \(\gamma \). The scheme is constructed with the guidance of the amplitude model presented in Ref. [17].

The position of a \({D} \) decay in phase space is specified by the four-momenta of the decay products. These four-momenta allow the corresponding decay amplitudes \({\mathcal {A}}_{{D} ^0} \) and \({\mathcal {A}}_{{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \) of the \({{D} ^0} \) and \({{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \) decays, respectively, to be determined from the amplitude model. From the \({{D} ^0} \) and \({{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \) decay amplitudes, two convenient parameters,

$$\begin{aligned} \Delta \delta _{D} \equiv \arg \Big (\frac{{\mathcal {A}}_{{D} ^0}}{{\mathcal {A}}_{{{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0}}}\Big ), \quad r_{D} \equiv \Big |\frac{{\mathcal {A}}_{{D} ^0}}{{\mathcal {A}}_{{{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0}}}\Big |, \end{aligned}$$
(13)

are defined. These are the strong-phase difference and the magnitude of the ratio between the \({{D} ^0} \) and \({{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \) amplitudes according to the model, respectively. Effectively, by considering \(\Delta \delta _{D} \) and \(r_{D} \) as parameters, each \({D} \) decay in the five-dimensional phase space is projected onto a two-dimensional surface, where a binning scheme can be defined.

Fig. 1
figure 1

Left: Optimised \(2\times 8\) binning scheme in \(\Delta \delta _D\)-\(\ln (r_D)\) space. Right: The associated \(c_i\) and \(s_i\) parameters calculated using the amplitude model (right). The numbers indicate the bin numbers

The binning is first performed in the \(\Delta \delta _{D} \) coordinate, which spans the range \([-\pi , \pi ]\) and is divided into \({\mathcal {N}}\) bins with boundaries that are symmetric around \(\Delta \delta _{D} =0\). Assigning a decay to a particular bin ensures it is grouped with other decays with a similar strong-phase difference, which maximises the sensitivity to \(\gamma \). Bin i is then divided in two, with labels i and \(-i\) according to the value of \(r_{{D}}\), giving \(2 \times {{\mathcal {N}}}\) bins in total. This division is performed in a manner to enhance the difference between \(F_{+i}\) and \(F_{-i}\), which maximises the magnitude of the interference terms in Eqs. (3) and (4). Candidates with \(\ln r_{D} < 0\) (\(> 0\)) are assigned to bin \(i > 0\) (\(< 0\)), and the bin numbering starts at \(i = {\mathcal {N}}\) (\(i = 1\)) near \(\Delta \delta _{D} = -\pi \), with decreasing (increasing) bin numbers.

Following the procedure described in Ref. [25], the binning scheme is optimised by adjusting the bin boundaries in \(\Delta \delta _D\) to maximise the Q-value, defined by \(Q^2 \equiv (Q_+^2 + Q_-^2)/2\), where

$$\begin{aligned} Q_\pm ^2 \equiv 1 - \left( 1 - \sum _i\frac{F_iF_{-i}\bigg (1 - c_i^2 - s_i^2\bigg )}{N_{i}^\pm }\right) \left( \sum _iF_i\right) ^{-1}. \nonumber \\ \end{aligned}$$
(14)

Here \(N_i^\pm \), the \(B^\pm \rightarrow DK^\pm \) decay yield in bin i, as predicted by Eqs. (3) and (4), is calculated with the normalisation coefficients \(h_{B^\pm }\) set to unity. The parameters \(Q_{\pm }\) give the statistical sensitivity of \(x_\pm \) and \(y_\pm \) from a binned fit, divided by that of an unbinned fit. With an infinite number of bins, \(Q\rightarrow 1\). This metric neglects any perturbation to the sensitivity that may arise from the presence of background events.

By allowing for only even values of \({\mathcal {N}}\), the bin boundaries at \(\Delta \delta _{D} = 0\) and \(\pm \pi \) may be fixed. Then each pair of bin boundaries on either side of \(\Delta \delta _{D} = 0\) are simultaneously adjusted until the Q-value is maximal. In each iteration \(F_i\), \(c_i\) and \(s_i\) are calculated from the amplitude model using Eqs. (5) and (6), assuming a uniform acceptance. The five-dimensional integral over phase space is performed with Monte Carlo integration. Large samples of \({D} \)-decays are generated for the integration, such that the uncertainty due to the finite sample size is negligible. The values of \(N_i^\pm \) are determined from Eqs. (3) and (4), and take as input \(\gamma = 75^\circ \), \(\delta ^{DK}_{B} = 130^\circ \) and \(r^{DK}_B = 0.1\), which lie close to the known values of these parameters [21]. Note, however, that Q has a very weak dependence on the values of the phases assumed in these expressions, so this choice does not bias the analysis.

Table 1 Values of \(c_i\), \(s_i\), \(F_i\) and \(V_i\) for the optimised \(2\times 8\) binning scheme, as calculated from the amplitude model. The number of digits quoted reflects the uncertainty due to the sample size used in the Monte Carlo integration

Figure 1, on the left, shows the binning scheme resulting from this procedure for \(2 \times 8\) bins, and the values of \(c_i\) and \(s_i\) predicted by the amplitude model are shown in Table 1, along with the predicted values of \(F_i\) and the fractional bin volume \(V_i\). In the calculation, the region of phase space where the invariant mass of the \({{\uppi } ^+} {{\uppi } ^-} \) lies close to the \({{K} ^0_{\textrm{S}}} \) mass is excluded. This requirement, which removes around \(5\%\) of signal decays, is imposed to match a selection requirement to remove background that is described in Sect. 5.

The binning scheme shown on the left in Fig. 1 has \(Q = 0.90\), which indicates that only \(10\%\) of the statistical sensitivity is lost through the binning of phase space. The procedure for assigning bin numbers for this scheme according to the four-momenta of the \({D} \) decay is provided in Ref. [26]. An alternative optimised binning scheme with \({\mathcal {N}}=4\) is presented in Appendix A.

The determination of the \({C\!P} \)-violating parameters requires as input the values of \(c_i\) and \(s_i\) in each bin. Although the bins have been defined using the amplitude model, it will be possible to use direct measurements of these parameters made at charm threshold, when they become available. The choice of binning scheme does not bias the determination of the \({C\!P} \)-violating parameters, even if the direct measurements were to indicate that the amplitude model gives an imperfect description of the strong-phase variation. In this case, there would be a reduction in statistical sensitivity compared to current expectations, but the result obtained for \(\gamma \) would have no model-dependent uncertainty.

4 The LHCb detector and data set

This analysis uses data collected by the LHCb experiment in pp collisions at \(\sqrt{s} = 7\text {\,Te\hspace{-1.00006pt}V} \), \(8\text {\,Te\hspace{-1.00006pt}V} \) and \(13\text {\,Te\hspace{-1.00006pt}V} \). The data sets correspond to integrated luminosities of \(1\text {\,fb} ^{-1} \), \(2\text {\,fb} ^{-1} \) and \(6\text {\,fb} ^{-1} \), respectively.

The LHCb detector [27, 28] is a single-arm forward spectrometer covering the pseudorapidity range \(2<\eta <5\), designed for the study of particles containing \(\textrm{b} \) or \(\textrm{c} \) quarks. The detector includes a high-precision tracking system consisting of a silicon-strip vertex detector surrounding the pp interaction region, a large-area silicon-strip detector located upstream of a dipole magnet with a bending power of about \(4{\mathrm {\,Tm}}\), and three stations of silicon-strip detectors and straw drift tubes placed downstream of the magnet. The tracking system provides a measurement of the momentum, \(p\), of charged particles with a relative uncertainty that varies from 0.5% at low momentum to 1.0% at 200 \(\text {\,Ge\hspace{-1.00006pt}V\!/}c\). The minimum distance of a track to a primary pp collision vertex (PV), the impact parameter (IP), is measured with a resolution of \((15+29/p_{\textrm{T}})\,\upmu \text {m} \), where \(p_{\textrm{T}}\) is the component of the momentum transverse to the beam, in \(\text {\,Ge\hspace{-1.00006pt}V\!/}c\). Different types of charged hadrons are distinguished using information from two ring-imaging Cherenkov detectors. Photons, electrons and hadrons are identified by a calorimeter system consisting of scintillating-pad and preshower detectors, an electromagnetic and a hadronic calorimeter. Muons are identified by a system composed of alternating layers of iron and multiwire proportional chambers. The online event selection is performed by a trigger, which consists of a hardware stage, based on information from the calorimeter and muon systems, followed by a software stage, which applies a full event reconstruction.

Simulation is required to model the effects of the detector acceptance, and for the study of possible background processes. In the simulation, pp collisions are generated using Pythia [29, 30] with a specific LHCb configuration [31]. Decays of unstable particles are described by EvtGen [32], in which final-state radiation is generated using Photos [33]. The \({{D} ^0} \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} \) decay is simulated using the amplitude model from Ref. [17]. The interaction of the generated particles with the detector, and its response, are implemented using the Geant4 toolkit [34, 35] as described in Ref. [36]. The underlying pp interaction is reused multiple times, with an independently generated signal decay for each event [37].

5 Candidate selection

A \({{B} ^\pm } \) candidate is reconstructed by combining five charged tracks. Four of the charged tracks are required to have an invariant mass within \(25\text {\,Me\hspace{-1.00006pt}V\!/}c^2 \) of the \({{D} ^0} \) meson mass [38]. This requirement, which corresponds to around two and a half times the resolution of the mass peak, removes processes that have either a missing or misidentified particle. Candidates where the opening angle between any pair of tracks from the \({D} \)-decay products is smaller than \(0.03^\circ \) are discarded, as these are likely to correspond to a single charged particle that is duplicated in the reconstruction.

To suppress charmless background, which arises from \({{B} ^\pm } \) meson decays where there is no intermediate charm meson, the distance between the \({D} \) and \({{B} ^\pm } \) decay vertices is required to be greater than twice its resolution for the binned measurement. This criterion eliminates \(95\%\) of this category of decays. In the phase-space integrated measurement, which is found to be more sensitive to this source of contamination, the decay distance requirement is tightened to four times its resolution, suppressing the background by a further order of magnitude.

Separation of \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) and \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) decays is achieved by imposing mutually exclusive particle identification (PID) requirements on the companion track, which is the \({{K} ^\pm } \) or \({{\uppi } ^\pm } \) meson of the \({{B} ^\pm } \rightarrow {D} h^\pm \) decay. Companion tracks that have associated activity in the muon detector are removed; this requirement reduces background from semi-leptonic b-hadron decays involving a muon which is misidentified as a companion kaon or pion. Background from semi-leptonic b-decays involving an electron is found to be negligible.

Background from \({D} \) decays where a \({{\uppi } ^+} {{\uppi } ^-} \) pair originates from a \({{K} ^0_{\textrm{S}}} \) meson is suppressed by excluding regions containing \({{\uppi } ^+} {{\uppi } ^-} \) pairs with invariant mass inside the interval \([477, 507]\text {\,Me\hspace{-1.00006pt}V\!/}c^2 \) from the binning scheme. Additionally, PID requirements are imposed on the kaon from the \({D} \) candidate with opposite sign to the companion track. This selection requirement suppresses \({D} \rightarrow {{K} ^\mp } {{\uppi } ^\pm } {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^0} \) background where the \({{\uppi } ^0} \) meson is not reconstructed and the kaon is misidentified.

Fig. 2
figure 2

Invariant-mass distributions for the (left) \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) and (right) \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) selections, for the \({D} \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} \) decay. The data are shown as black points and the blue curve is the fit result. The square brackets in the legend denote particles that are not reconstructed

Combinatorial background is suppressed using a boosted decision tree (BDT) algorithm [39, 40] implemented in the TMVA toolkit [41, 42]. Simulated signal events are used as the signal training sample, while candidates with invariant mass in the upper \({{B} ^\pm } \) sideband between 5800–7000\(\text {\,Me\hspace{-1.00006pt}V\!/}c^2\) form the background training sample. The input variables of the BDT include the momenta and IPs of the \({{B} ^\pm } \), \({D} \) and companion-track candidates. The parameters are described in detail in Ref. [43]. The optimal working point of the BDT is determined by performing pseudoexperiments to determine the cut value that provides the best sensitivity to \(\gamma \). Candidates with a BDT score below this cut value are discarded.

To improve the resolution of the momenta of the \({D} \)-decay products and the invariant mass of the \({{B} ^\pm } \) candidate, a kinematic fit is performed in which the \({D} \) meson candidate is constrained to its known mass [38], and the \({{B} ^\pm } \) candidate is constrained to originate from its associated PV. This is defined as the PV with the smallest impact parameter with respect to the \({{B} ^\pm } \) candidate.

6 Invariant-mass fits

An unbinned, extended maximum-likelihood fit is performed simultaneously to the invariant-mass spectrum of the \({{B} ^\pm } \rightarrow [{{K} ^-} {{K} ^+} {{\uppi } ^+} {{\uppi } ^-} ]_{D} h^\pm \) and \({{B} ^\pm } \rightarrow [{{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^+} {{\uppi } ^-} ]_{D} h^\pm \) candidates in the range from \(5080\text {\,Me\hspace{-1.00006pt}V\!/}c^2 \) to \(5700\text {\,Me\hspace{-1.00006pt}V\!/}c^2 \). The fit is first performed on the \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) and \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) candidates, integrated over all phase-space bins, which is referred to as the global fit. The global fit is used to determine the parameters of the functions that describe the signal and background invariant-mass distributions. The \({{B} ^\pm } \rightarrow [{{K} ^-} {{K} ^+} {{\uppi } ^+} {{\uppi } ^-} ]_{D} h^\pm \) invariant-mass distributions are shown in Fig. 2.

The yield of \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) is varied separately for the two \({D} \) decays, while the yield of \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) is parameterised as a ratio relative to the \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) yield. This ratio is a common fit parameter for the two \({D} \) decays, as are the parameters that describe the signal shape.

The peak at around \(5280\,\text {\,Me\hspace{-1.00006pt}V\!/}c^2 \) corresponds to \({{B} ^\pm } \rightarrow [{{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} h^\pm \) candidates that are correctly reconstructed. The signal invariant-mass shape is parameterised as

$$\begin{aligned} \begin{aligned} f_{\textrm{signal}}(m|m_{B}, \sigma , \alpha _L, \alpha _R, \beta , k)&= k\times f_{\textrm{MG}}(m|m_{B}, \sigma , \alpha _L, \alpha _R, \beta ) \\&\quad + (1 - k)\times f_{\textrm{G}}(m|m_{B}, \sigma ), \end{aligned}\nonumber \\ \end{aligned}$$
(15)

where \(f_{\textrm{G}}\) is a Gaussian function and \(f_{\textrm{MG}}\) is a modified Gaussian function,

$$\begin{aligned}{} & {} \!\!\!f_{\textrm{MG}}(m|m_{B}, \sigma , \alpha _L, \alpha _R, \beta )\propto \nonumber \\{} & {} \quad {\left\{ \begin{array}{ll} \exp \Big (-\frac{\Delta m^2(1 + \beta \Delta m^2)}{2\sigma ^2 + \alpha _L\Delta m^2}\Big ), \quad \Delta m = m - m_{B} < 0, \\ \exp \Big (-\frac{\Delta m^2(1 + \beta \Delta m^2)}{2\sigma ^2 + \alpha _R\Delta m^2}\Big ), \quad \Delta m = m - m_{B} > 0. \end{array}\right. } \end{aligned}$$
(16)

The function \(f_{\textrm{MG}}\) has approximately Gaussian behaviour when \(\Delta m^2\ll \sigma ^2/\alpha _{L, R}\) or \(\Delta m^2\gg \beta ^{-1}\), but it includes tails to better model the experimental resolution. The tail parameters \(\alpha _{L, R}\) and \(\beta \), and the fraction k, are determined in a fit to simulated events, but the peak position \(m_{B} \) and the width \(\sigma \) are determined in the fit to data. The mass \(m_{B} \) is common between the \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) and \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) channels, but \(\sigma \) is different because the \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) width is narrower due to the lower energy release in the decay.

At masses above the \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) peak there is a non-negligible contribution from \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) decays where the companion is misidentified as a kaon. The rate of this cross-feed background is fixed from the relative PID efficiencies, which are determined in calibration data that are weighted to match the momentum and pseudorapidity distributions of the companion track of the signal. The exact shape is determined using a data-driven method by swapping the mass hypothesis of the companion track in the \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) peak. Similarly, the shape of \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) candidates misidentified as \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) candidates is also accounted for, but the impact of this background is minimal due to its smaller branching fraction.

Table 2 Yields of \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) and \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) candidates, partially reconstructed background, \({D} \rightarrow {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^0} \) background, combinatorial background and charmless background in the region \(m_{B} \in [5249, 5309]\text {\,Me\hspace{-1.00006pt}V\!/}c^2 \), where the charm meson decays via \({D} \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} \) and \({D} \rightarrow {{\uppi } ^+} {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^-} \). Charmless background in the \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) decay modes is known to be negligible, and is therefore not modelled

Candidates with masses below that of the signal peak are background from \({B} \)-meson decays where a neutral particle or charged pion is not reconstructed. In this analysis, the model describing this partially reconstructed background and its associated parameters are taken from Ref. [6], with the exception of the contamination from \({{B} ^0_{\textrm{s}}} \rightarrow {{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} {{K} ^-} {{\uppi } ^+} \) and \({{\hspace{0.0pt}\overline{\hspace{0.0pt}B}} {}^0_{\textrm{s}}} \rightarrow {{D} ^0} {{K} ^+} {{\uppi } ^-} \) decays with a missing pion. The total yield of this latter background, which is fixed in the fit, has been updated according to results in Ref. [7].

Additionally, the decay \({D} \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} \) is contaminated by \({D} \rightarrow {{K} ^\mp } {{\uppi } ^\pm } {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^0} \) decays, where a charged pion is misidentified as a kaon and the neutral pion is not reconstructed. This background is present below the signal peak, but it has a large tail towards the upper end of the invariant mass spectrum of the \({{B} ^\pm } \) candidates as well. The shape of this background is fixed using a simulation sample, while its yield is a free parameter. The ratio between this background and signal is common between the \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) and \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) modes.

The contamination of charmless decays in the \({D} \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} \) mode is also different from Ref. [6]. In particular, the \({{B} ^\pm } \rightarrow [{{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} {{K} ^\pm } \) sample has a significant contribution from the mode \({{B} ^\pm } \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} {{K} ^\pm } \). Both the magnitude and invariant-mass shape of this contribution are fixed in the invariant-mass fit from studies of the lower sideband of the \({D} \) invariant mass. Analogous studies show that there is no significant contamination from charmless decays in the \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) selection.

The signal yields, obtained from the invariant-mass fit, integrated over all phase-space bins, are given in Table 2. The yields are scaled from the full fit region to the signal region \(m_{B} \in [5249, 5309]\text {\,Me\hspace{-1.00006pt}V\!/}c^2 \). The uncertainties on the \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) yields are reduced due to the common ratio determined from both the \({D} \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} \) and \({D} \rightarrow {{\uppi } ^+} {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^-} \) decay modes.

After the global invariant-mass fit, a second fit is performed where the \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) and \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) candidates are split by charge and sorted into bins of phase space, which makes a total of \(2\times 2\times 16 = 64\) categories. The lower fit boundary is increased to \(5150\text {\,Me\hspace{-1.00006pt}V\!/}c^2 \) to remove most of the partially reconstructed background. The shape parameters and relative yields of the different background components are fixed from the global fit. The signal yields in each bin are parameterised in terms of the \(C\!P\)-violating observables, which are free parameters in the fit. The \(F_i\) parameters are also free parameters, while the strong-phase parameters \(c_i\) and \(s_i\) are fixed according to the LHCb amplitude model.

In each bin, the yield of combinatorial background and partially reconstructed background are free parameters, with the exception of the \({{B} ^0_{\textrm{s}}} \rightarrow {{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} {{K} ^-} {{\uppi } ^+} \) contamination (and charge-conjugated case), which is treated separately because the charm meson has the flavour opposite to the signal decay and the other partially reconstructed background contributions. In \({{{B} ^-}} \) (\({{{B} ^+}} \)) decays, the fractional bin yield of the \({{B} ^0_{\textrm{s}}} \) (\({\hspace{0.0pt}\overline{\hspace{0.0pt}B}} {}^0_{\textrm{s}} \)) background is therefore set equal to \(F_{-i}\) (\(F_i\)). In simulation, the \({D} \rightarrow {{K} ^\mp } {{\uppi } ^\pm } {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^0} \) decays are uniformly distributed in the \({D} \rightarrow {{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} \) phase space. Therefore, the distribution of \({D} \rightarrow {{K} ^\mp } {{\uppi } ^\pm } {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^0} \) decays between phase space bins is assumed to be proportional to the bin volume, given in Table 1. The distribution of the charmless background between phase-space bins is determined from the lower \({D} \)-mass sideband.

Fit biases and instabilities in the fit are studied by performing pseudoexperiments. The pull distributions of \(x^{DK}_\pm \) and \(y^{DK}_\pm \) are found to be consistent with a normal Gaussian distribution. The results for \(x^{D\pi }_\xi \) and \(y^{D\pi }_\xi \) show biases of up to \(7\%\) and widths that show up to \(17\%\) overcoverage. Corrections are applied to the measured values in the data to account for these effects.

Fig. 3
figure 3

Graphical representation of the \(C\!P\)-violating observables for the (left) \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) and (right) \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) decay modes, assuming amplitude-model predictions of \(c_i\) and \(s_i\). The 1\(\sigma \) and 2\(\sigma \) contours shown represent the statistical uncertainties and correspond to \(68.3\%\) (\(\Delta \chi ^2 = 2.30\)) and \(95.5\%\) (\(\Delta \chi ^2 = 6.18\)) confidence intervals

Fig. 4
figure 4

Fractional bin asymmetries for the (left) \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) and (right) \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) decays. The data are overlaid with the fit result and the prediction without \(C\!P\) violation

The fitted \(C\!P\)-violating observables are listed in Table 3 and plotted in Fig. 3, along with the likelihood contours, which only include statistical uncertainties. The lengths of the two vectors from the origin to \((x_\pm ^{{D} {K}}, y_\pm ^{{D} {K}})\) determine \(r_{B} ^{{D} {K}}\), while the angle between them is \(2\gamma \), according to the expressions in Eq. (2). In the absence of \(C\!P\) violation, the two vectors would be identical. The vector from the origin to \((x_\xi ^{{D} {\uppi }}, y_\xi ^{{D} {\uppi }})\) indicates the relative size and angle between \(r_{B} \) and \(\delta _{B} \) of the \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) and \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) decays, according to Eq. (8). Since this contour overlaps with the origin, the sensitivity to \(C\!P\) violation is much smaller in the \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) mode.

Table 3 Results of the binned fit. The first uncertainty is statistical, the second is systematic, and the third is associated with the model dependence of the strong-phase parameters

The \(C\!P\)-violation effects can be illustrated directly by considering the asymmetries in each bin. This information is obtained from an alternative fit where the signal yields, instead of the \(C\!P\)-violating observables, are determined. The bin asymmetries, \((N_i^- - N_{-i}^+)/(N_i^- + N_{-i}^+)\), calculated from these yields are shown in Fig. 4. The bin yields are normalised separately for \({{{B} ^-}} \) and \({{{B} ^+}} \), so that only bin-to-bin variations are apparent. Some of those for \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) are significant, and exhibit a non-trivial distribution, which is driven by the variation in the strong-phase difference between \({{D} ^0} \) and \({{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \) decays. In the \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) mode, there is a lower sensitivity to \(C\!P\) violation and therefore no such behaviour is seen. The hypothesis that the fit model is correct leads to p-values of 0.95 and 0.05 for the left and right histograms in Fig. 4 respectively, based on statistical uncertainties only.

The total bin yields \(N_i^- + N_{-i}^+\) are also shown in Fig. 5, where the sum over all bins is normalised to unity. The projections from the fit results are also plotted, and reasonable agreement is found. Pseudoexperiments indicate that the p-values of the two histograms in Fig. 5 are highly correlated, and their combined p-value is 0.04, accounting for statistical uncertainties only.

Fig. 5
figure 5

Total bin yields for the (left) \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) and (right) \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) decays. The data are overlaid with the fit projections

Fig. 6
figure 6

Invariant-mass distributions and fit projections of (top) \({{B} ^\pm } \rightarrow [{{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} {{K} ^\pm } \) and (bottom) \({{B} ^\pm } \rightarrow [{{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} {{\uppi } ^\pm } \) candidates, for (left) \({{{B} ^+}} \) and (right) \({{{B} ^-}} \) decays. The data are shown as black points and the blue curve is the fit result

Fig. 7
figure 7

Invariant-mass distributions and fit projections of (top) \({{B} ^\pm } \rightarrow [{{\uppi } ^+} {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} {{K} ^\pm } \) and (bottom) \({{B} ^\pm } \rightarrow [{{\uppi } ^+} {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} {{\uppi } ^\pm } \) candidates, for (left) \({{{B} ^+}} \) and (right) \({{{B} ^-}} \) decays. The data are shown as black points and the blue curve is the fit result

To determine the phase-space integrated \(C\!P\)-violating observables defined in Eqs. (9) and  (10), an analogous fit is performed without phase-space binning. The yields of the \({{B} ^\pm } \rightarrow [{{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} h^\pm \) and \({{B} ^\pm } \rightarrow [{{\uppi } ^+} {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} h^\pm \) modes, split by charge, are expressed in terms of the \(C\!P\)-violating observables and fitted simultaneously. The shape parameters of the signal and background contributions are common fit parameters between the two D-decay channels. The fits to the invariant-mass distributions for the \({{B} ^\pm } \) candidates are shown in Figs. 6 and 7, split by \({B} \) decay, \({D} \) decay and charge, and the resulting \(C\!P\)-violating observables are listed in Table 4. The measured values of the observables of the \({{B} ^\pm } \rightarrow [{{\uppi } ^+} {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} h^\pm \) mode are consistent with those reported in Ref. [4].

The results in Table 4 are corrected for production asymmetries of the \({{B} ^\pm } \) mesons and detection asymmetries of the companion hadron. The production asymmetry was measured in Ref. [7] to be \((0.028 \pm 0.068)\%\). In Ref. [44], the difference in detection asymmetries between kaons and pions was determined to be \((-0.96 \pm 0.13)\%\), in addition to a hardware-trigger asymmetry of \((-0.17 \pm 0.08)\%\). The detection asymmetry of pions was found to be \((0.064 \pm 0.018)\%\) [44].

Table 4 Results of the phase-space integrated measurements. The first uncertainty is statistical and the second systematic
Table 5 Uncertainties on the results of the binned analysis

7 Systematic uncertainties

The systematic uncertainties in the binned measurement are summarised in Table 5. The uncertainties arise both from contributions that are internal to the analysis, and also from external knowledge of the \(c_i\) and \(s_i\) parameters.

The uncertainty associated with the fixed invariant-mass shapes are propagated to the \(C\!P\)-violating observables by repeating the two-stage fit procedure with different choices of shape. For each iteration, the shape parameters that are fixed in the global fit are changed to new values obtained with a resampling technique and the global fit is rerun. The other shape parameters that are determined from this fit are then input to the binned fit, which is otherwise unchanged from the baseline configuration. The standard deviations of the resulting distributions of the \(C\!P\)-violating observables are assigned as the systematic uncertainty due to fixed mass shapes. Furthermore, to assess the impact of any bin-dependence of the mass shapes, the mass shapes are determined separately in each bin and pseudoexperiments are generated with individual mass shapes in each bin. The shifts in the central values are taken as the corresponding systematic uncertainty.

The uncertainties on the PID efficiencies are propagated to \(x^{{D} {K}}_\pm \), \(y^{{D} {K}}_\pm \), \(x^{{D} {\uppi }}_\xi \) and \(y^{{D} {\uppi }}_\xi \) by repeating the fit to the \(C\!P\)-violating observables, each time varying the parameters within their uncertainties. The same procedure is followed to assign the uncertainty associated with the relative contributions of the different components of the low-mass partially reconstructed background. Similarly, for the charmless background, the yields are varied within their uncertainties. The standard deviations of the fitted \(C\!P\)-violating observables are taken as the systematic uncertainty.

The partially reconstructed background at low mass is subject to \(C\!P\) violation that means the bin distribution of these events differs between the \({{{B} ^+}} \) and \({{{B} ^-}} \) samples. To investigate this effect, pseudoexperiments are generated containing \(C\!P\) violation for these decays following the procedure described in Ref. [6], which are then fitted using the baseline model. The observed shifts in the central values are taken as the systematic uncertainty from this source.

There are several known sources of background present in the signal region that are not accounted for in the invariant-mass fit. These are semi-leptonic b-hadron decays that survive the muon veto; \({{B} ^\pm } \rightarrow {D} h^\pm \) decays where the \({D} \) meson decays semi-leptonically; \({{B} ^\pm } \rightarrow {D} h^\pm \), \({D} \rightarrow {{K} ^\mp } {{\uppi } ^\pm } {{\uppi } ^+} {{\uppi } ^-} \) decays, where the kaon is misidentified as a pion, and two of the pions are misidentified as kaons; and decays of \({{\Lambda } ^0_{\textrm{b}}} \rightarrow {\textrm{p}} {{D} ^0} {{\uppi } ^-} \) (and charge conjugated case), where the proton is misidentified as the companion hadron and the pion is not reconstructed. To evaluate the potential bias arising from these neglected contributions, pseudoexperiments are generated with each component included, which are then fitted with the baseline model. The shifts in the resulting \(C\!P\)-violating observables are taken as the systematic uncertainty.

The distribution of the \({D} \rightarrow {K} ^\mp {\uppi } ^\pm {\uppi } ^+{\uppi } ^-{{\uppi } ^0} \) background over phase space is not well known. The impact of this lack of knowledge is assessed by changing the distribution from that of the baseline model to one in which the population in each bin is proportional to the \(F_i\) parameters. The shifts in the \(C\!P\)-violating observables are assigned as the systematic uncertainty.

Finally, systematic uncertainty due to fit biases is included, which is set to be equal to the size of the bias for each \(C\!P\)-violating observable. Adding these in quadrature to the contributions discussed above gives a total internal systematic uncertainty for each observable, but excluding the systematic uncertainty arising from \(c_i\) and \(s_i\). The total internal systematic uncertainty is found to be an order of magnitude smaller than the corresponding statistical uncertainty.

In the current analysis the values of the \(c_i\) and \(s_i\) parameters are taken from the amplitude model constructed with LHCb data and described in Ref. [17]. An alternative model fitted to data from the CLEO experiment [45] is used to generate pseudoexperiments which are then fitted using the \(c_i\) and \(s_i\) parameters from the LHCb model. The observed shifts in the \(C\!P\)-violating observables are taken as the uncertainties arising from the choice of model used to calculate the \(c_i\) and \(s_i\) parameters. These uncertainties are in several cases significantly larger than the LHCb systematic uncertainties. In the future, the values of \(c_i\) and \(s_i\) will be taken from measurements performed at charm threshold, which will affect both the central values of the observables and allow the corresponding uncertainties to be assigned in a model-independent manner.

Table 6 Uncertainties on the results of the phase-space integrated analysis
Fig. 8
figure 8

Interpretation of the binned and phase-space (‘PS’) integrated measurements in terms to the underlying physics parameters. The 1\(\sigma \) and 2\(\sigma \) contours are shown, which correspond to \(68.3\%\) (\(\Delta \chi ^2 = 2.30\)) and \(95.5\%\) (\(\Delta \chi ^2 = 6.18\)) confidence intervals, respectively. Also shown is the result from the analysis of other decay modes at LHCb (‘LHCb 2021’) [21]

Cross checks are performed with simulation that validate certain assumptions in the analysis, and for which no systematic uncertainties are therefore applied. These studies assess the difference in acceptance over D-meson phase space for \(B^\pm \rightarrow DK^\pm \) and \(B^\pm \rightarrow D\pi ^\pm \) decays, acceptance effects on the effective values of \(c_i\) and \(s_i\), the effects of bin migration, and the effect of neglecting \({{D} ^0} \)-\({{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \) mixing in the fit. In all cases there are negligible biases on the measured parameters, within the current statistical precision.

The binned measurement is largely insensitive to \({{D} ^0} \)-\({{\hspace{0.0pt}\overline{\hspace{0.0pt}D}} {}^0} \) mixing and bin migration because \(F_i\) are free parameters in the fit. Since the mixing and bin-migration effects are very similar between the \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) and \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) modes, the \({{B} ^\pm } \rightarrow {D} {{\uppi } ^\pm } \) mode provides a first order correction that is incorporated into the \(F_i\) parameters.

The systematic uncertainties on the phase-space integrated observables are shown in Table 6. They are evaluated using the same strategy as those in the binned analysis. In addition, there are systematic uncertainties due to the production and detection asymmetries, which are estimated by repeating the fit many times, each time varying the parameters randomly within their uncertainties [7], and taking the spreads of the resulting distributions as the assigned uncertainties. The total systematic uncertainties for the asymmetries are an order of magnitude smaller than the statistical uncertainties, and of a similar size for the ratio observables.

Correlation matrices for all the measured observables can be found in Appendix B.

8 Interpretation

The measured \(C\!P\)-violating observables in Table 3 are interpreted in terms of the underlying physics parameters \(\gamma \), \(\delta _{B} ^{{D} {K}}\), \(r_{B} ^{{D} {K}}\), \(\delta _{B} ^{{D} {\uppi }}\) and \(r_{B} ^{{D} {\uppi }}\) using a maximum likelihood fit, following the procedure described in Ref. [21].

The fit is first made to the results of the \(B^\pm \rightarrow [K^+K^-\pi ^+\pi ^-]_D h^\pm \) binned analysis alone. The 1 and 2\(\sigma \) contours in the \(\gamma \) vs. \(\delta _B^{DK}\) and the \(r_B^{DK}\) vs. \(\delta _B^{DK}\) planes are shown in Fig. 8. The numerical results are

$$\begin{aligned}{} & {} \gamma = (116^{+12}_{-14})^\circ , \\{} & {} \delta _{B} ^{{D} {K}} = (81^{+14}_{-13})^\circ , \\{} & {} r_{B} ^{{D} {K}} = 0.110^{+0.020}_{-0.020}, \\{} & {} \delta _{B} ^{{D} {\uppi }} = (298^{+62}_{-118})^\circ , \\{} & {} r_{B} ^{{D} {\uppi }} = 0.0041^{+0.0054}_{-0.0041}, \end{aligned}$$

where the uncertainties are the combined statistical and systematic uncertainties, and the results are completely dominated by the former. These model-dependent results may be compared to those from a recent measurement of \(\gamma \) and associated parameter derived from an ensemble of beauty and charm-meson decay studies performed by LHCb [21]. The \(3\sigma \) contours of the results for \(\gamma \) and \(\delta _B^{DK}\) from the \(B^\pm \rightarrow [K^+K^-\pi ^+\pi ^-]_D h^\pm \) analysis encompass the central values coming from the other decay modes.

The fit is then made to the phase-space integrated \(C\!P\)-violating observables. Here it is necessary to know the \(C\!P\)-even fractions \(F_+\) for each decay, which have been measured by the BESIII collaboration. In the case of \({D} \rightarrow K^+K^-\pi ^+\pi ^-\) the measured value is \(F_+ = 0.73 \pm 0.04\) [46] and for \(D \rightarrow \pi ^+\pi ^-\pi ^+\pi ^-\) the value \(0.735 \pm 0.016\) is used [47]. Due to the trigonometric dependence, multiple solutions are obtained. The likelihood contours are shown in Fig. 8. It can be seen that these are compatible with measurements using other decay channels.

9 Summary and conclusions

The first measurement of \(C\!P\)-violating observables for the decay \({{B} ^\pm } \rightarrow [{{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} h^\pm \) is presented. The analysis is performed in bins of phase space of the \({D} \)-meson decay, which are chosen to optimise sensitivity to the angle \(\gamma \) of the CKM Unitary Triangle. The local asymmetries confirm the presence of \(C\!P\) violation effects that have also been observed in other \({{B} ^\pm } \rightarrow {D} {{K} ^\pm } \) decay modes. In addition, measurements of \(C\!P\)-violating observables integrated over phase space are performed for the decays \({{B} ^\pm } \rightarrow [{{K} ^+} {{K} ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} h^\pm \) and \({{B} ^\pm } \rightarrow [{{\uppi } ^+} {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} h^\pm \). All studies make use of the full data set collected by LHCb in 2011–2012 and 2015–2018, corresponding to an integrated luminosity of \(9\text {\,fb} ^{-1} \). The measurement of the phase-space integrated observables in the \({{B} ^\pm } \rightarrow [{{\uppi } ^+} {{\uppi } ^-} {{\uppi } ^+} {{\uppi } ^-} ]_{D} h^\pm \) mode supersedes those reported in Ref. [7].

The measurements of the \(C\!P\)-violating observables in the binned analysis require knowledge of the \({D} \)-meson strong-phase parameters \(c_i\) and \(s_i\). The values of these parameters are currently taken from an amplitude model [17]. In the future, direct measurements of these parameters at charm threshold [18] in combination with the measured yields in each bin, which are reported in Appendix A, will allow the \(C\!P\)-violating observables to be determined in a model-independent fashion.

The current measurements, together with those of the integrated charge asymmetries, may be interpreted in terms of \(\gamma \) and the other underlying physics parameters. When this is done for the results of the binned \(B^\pm \rightarrow [K^+K^-\pi ^+\pi ^-]_D h^\pm \) analysis, a model-dependent value of \(\gamma = (116^{+12}_{-14})^\circ \) is obtained. This result will evolve when the observables are re-evaluated using model-independent inputs. A model-independent determination of \(\gamma \) making use of both the binned and unbinned analysis of these four-body D-meson decay modes will be a valuable addition to the set of measurements of this important parameter already performed at LHCb. The potential of exploiting modes of this multiplicity for a model-independent measurement of \(\gamma \) has already been demonstrated with \({{B} ^\pm } \rightarrow [{{K} ^\mp } {{\uppi } ^\pm } {{\uppi } ^\mp } {{\uppi } ^\pm } ]_{D} h^\pm \) decays [19]. The precision of the four-body D-decay studies is limited by the sample size and is expected to improve significantly with future data from LHCb.