Measurement of the CKM angle $\gamma$ in the ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ channel using self-conjugate ${D} \rightarrow {{K} ^0_{\textrm{S}}} h^+ h^-$ decays

Abstract

A model-independent study of $C\!P$ violation in ${{B} ^0} \rightarrow {D} {{K} ^{*0}}$ decays is presented using data corresponding to an integrated luminosity of 9$\text {\,fb} ^{-1}$ collected by the LHCb experiment at centre-of-mass energies of $\sqrt{s}=7, \, 8$ and 13$\text {\,Te\hspace{-1.00006pt}V}$. The CKM angle $\gamma$ is determined by examining the distributions of signal decays in phase-space bins of the self-conjugate ${D} \rightarrow {{K} ^0_{\textrm{S}}} h^+ h^-$ decays, where $h = \pi , K$. Observables related to $C\!P$ violation are measured and the angle $\gamma$ is determined to be $\gamma =(49^{+ 22}_{-19})^\circ$. Measurements of the amplitude ratio and strong-phase difference between the favoured and suppressed ${B} ^0$ decays are also presented.

1 Introduction

In the Standard Model (SM), the Cabibbo–Kobayashi–Maskawa (CKM) matrix [1, 2] describes flavour-changing weak transitions of quarks. The phase difference between the CKM matrix elements for ${b} \rightarrow {u}$ and ${b} \rightarrow {c}$ quark transitions, defined as $\gamma \equiv \textrm{arg}\left( -V_{\textit{ud}}V_{\textit{ub}}^{*}/V_{\textit{cd}}V_{\textit{cb}}^{*}\right)$, is of particular interest because it is measurable in purely tree-level decays and has negligible theoretical uncertainty [3]. Therefore, the SM can be tested by comparing direct measurements of $\gamma$ with indirect determinations obtained by fitting the CKM unitarity triangle. The average value of the direct measurements is $\gamma _\mathrm{{direct}}=(66.2^{+3.4}_{-3.6})^\circ$ [4], which agrees at current precision with the indirectly determined value $\gamma _\mathrm{{indirect}}=(65.6^{+0.9}_{-2.7})^\circ$ [5] or $\gamma _\mathrm{{indirect}}=(65.8 \pm 2.2)^\circ$ [6] depending on the statistical approach used. A more stringent test requires improving the precision on both the direct and indirect determinations of $\gamma$.

The precision on $\gamma$ is dominated by the measured $C\!P$ violation in the interference between ${b} \rightarrow {u} \overline{{c}} {s}$ and ${b} \rightarrow {c} \overline{{u}} {s}$ quark transitions in ${B} ^\pm \rightarrow {D} K^\pm$ decays. Here, $D$ represents a superposition of ${D} ^0$ and ${\hspace{1.79993pt}\overline{\hspace{-1.79993pt}D}} {}^0$ mesons. However it is possible to gain complementary information from the ${{B} ^0} \rightarrow {D} {{K} ^*} (892)^{0}$ decay.¹ While this decay has a lower branching fraction compared to the ${B} ^\pm \rightarrow {D} K^\pm$ channel, the interference between the favoured and suppressed ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ decays is expected to be a factor of 3 larger since both amplitudes are colour suppressed, leading to a higher per-event sensitivity to $\gamma$. Feynman diagrams of the two possible ${{B} ^0}$ decays are shown in Fig. 1. The flavour of the $B$ meson at the point of decay is unambiguously provided by the charge of the kaon from the ${{K} ^*} (892)^{0} \rightarrow {{K} ^+} {{\pi } ^-}$ decay, and hence the analysis of this channel can proceed without considering time dependence. Interference between the two amplitudes is accessed through reconstruction of the ${D}$ meson in final states common to both ${D} ^0$ and ${\hspace{1.79993pt}\overline{\hspace{-1.79993pt}D}} {}^0$. For the analysis presented here the $D$ mesons are reconstructed in the self-conjugate ${D} \rightarrow {{K} ^0_{\textrm{S}}} h^+ h^-$ decay modes ($h = \pi , K$). The Belle [7, 8] and BaBar [9] collaborations have used the ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ channel to determine $\gamma$ with various final states of $D$ decay, including ${D} \rightarrow {{K} ^0_{\textrm{S}}} {{\pi } ^+} {{\pi } ^-}$. However, the most precise measurements using the ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ decay mode have been made by the LHCb experiment [10, 11].

Fig. 1

Feynman diagrams for the (left) suppressed and (right) favoured ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ decays

The work presented here uses data collected with the LHCb detector in proton-proton (pp) collisions at centre-of-mass energies of $\sqrt{s}=7, \, 8$ and 13$\text {\,Te\hspace{-1.00006pt}V}$ between 2011–2012 and 2015–2018, corresponding to an integrated luminosity of 9$\text {\,fb} ^{-1}$. The experimental procedure employed here closely follows that described in Ref. [11], where $C\!P$ violation observables that are related to $\gamma$ are determined through the distributions of ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ and ${{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0} \rightarrow {D} {{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}K}} {}^{*0}}$ decays in regions of the ${D} \rightarrow {{K} ^0_{\textrm{S}}} h^+ h^-$ decay phase space [12,13,14,15]. The extraction of $\gamma$ requires knowledge of the $D$ decay strong-phase parameters, which were directly determined by the BESIII [16,17,18] and CLEO [19] collaborations. Therefore, the measurement avoids using any $D$ decay amplitude model, thus is free of any systematic uncertainty attributed to such models.

The data set used for the work presented here is increased compared to Ref. [11]. In addition, a number of procedural improvements are made, such as adopting a more optimal division of ${D} \rightarrow {{K} ^0_{\textrm{S}}} {{\pi } ^+} {{\pi } ^-}$ phase space, and employing an improved strategy to handle the varying reconstruction efficiency over $D$ decay phase space. Furthermore, the strong-phase inputs are updated to reflect the most recent combination of results from CLEO and BESIII [16, 17].

2 Analysis overview

The amplitudes of the favoured and suppressed ${{B} ^0} \rightarrow {D} {{K} ^+} {{\pi } ^-}$ decays, where the ${{K} ^+} {{\pi } ^-}$ is not restricted to the ${K} ^{*0}$ resonance, can be written as

$$\begin{aligned} A({{B} ^0} \rightarrow {{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}D}} {}^0} {{K} ^+} {{\pi } ^-}) \equiv&\ A_c(p) e^{i\delta _c(p)}&\ \text {(favoured)}, \end{aligned}$$

(1)

$$\begin{aligned} A({{B} ^0} \rightarrow {{D} ^0} {{K} ^+} {{\pi } ^-}) \equiv&\ A_u(p) e^{i\left[ \delta _u(p) + \gamma \right] }&\ \text {(suppressed)}, \end{aligned}$$

(2)

where $A_{c(u)}$ and $\delta _{c(u)}$ are the magnitude and strong-phase of the decay corresponding to the ${b} \rightarrow {c} ({u})$ transitions, respectively, and p is the phase-space coordinate of the $D{{K} ^+} {{\pi } ^-}$ final state. The equivalent amplitudes for the $C\!P$ conjugate, ${{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0} \rightarrow {D} {{K} ^-} {{\pi } ^+}$, are given by transforming $\gamma \rightarrow -\gamma$. In this analysis, the amplitude ratio ($r_{{B} ^0}$) and strong-phase difference ($\delta _{{B} ^0}$) between the favoured and suppressed signal decays are measured alongside the angle $\gamma$. They are defined as

$$\begin{aligned} r_{{B} ^0} ^2&= \frac{\int _{{{K} ^{*0}}} \textrm{d}p\, A_u(p)^2}{\int _{{{K} ^{*0}}} \textrm{d}p\, A_c(p)^2}, \end{aligned}$$

(3)

$$\begin{aligned} \kappa e^{i\delta _{{B} ^0}}&= \frac{\int _{{{K} ^{*0}}} \textrm{d}p\, A_c(p) A_u(p) e^{i\left[ \delta _u(p) - \delta _c(p)\right] }}{\sqrt{\int _{{{K} ^{*0}}} \textrm{d}p\, A_c(p)^2} \sqrt{\int _{{{K} ^{*0}}} \textrm{d}p\, A_u(p)^2}}, \end{aligned}$$

(4)

where the integral is performed over the ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ region of the ${{B} ^0} \rightarrow {D} {{K} ^+} {{\pi } ^-}$ phase space. The coherence factor, $\kappa$, accounts for pollution from decays that are not ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$, and satisfies $0 \le \kappa \le 1$. The value of $\kappa =0.958^{+0.005}_{-0.046}$ is used as a direct input from the LHCb amplitude analysis of ${{B} ^0} \rightarrow {D} {{K} ^+} {{\pi } ^-}$ decays described in Ref. [20]. The kinematic selection of the ${K} ^{*0}$ candidates used in this work follows that of Ref. [20] to match the phase-space region in which $\kappa$ is evaluated.

The amplitudes for the ${{D} ^0} \rightarrow {{K} ^0_{\textrm{S}}} h^+ h^-$ and ${{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}D}} {}^0} \rightarrow {{K} ^0_{\textrm{S}}} h^+ h^-$ decays are written as $A_{D} (m_-^2, m_+^2) = |A_{D} (m_-^2, m_+^2) |e^{i\delta (m_-^2,\ m_+^2)}$ and $A_{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}D}}} (m_-^2, m_+^2) = |A_{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}D}}} (m_-^2, m_+^2) |e^{i\overline{\delta }(m_-^2, m_+^2)}$, respectively, where $m_\pm ^2 = m^2({{K} ^0_{\textrm{S}}} h^\pm )$ are the Dalitz plot coordinates. The $D$ decay phase space is divided into independent regions. A scheme is used with $2\mathcal {N}$ bins labelled from $i=-\mathcal {N}$ to $i=\mathcal {N}$ (excluding 0). The division is symmetrical about the line $m_-^2 = m_+^2$, and a bin where $m_-^2 > m_+^2$ ($m_-^2 < m_+^2$) is referred to as the $i^{th}$ ($-i^{th}$) bin. The ‘optimal’ [19] (‘2-bin’) scheme with $\mathcal {N}=8$ ($\mathcal {N}=2$) bins is used for the ${D} \rightarrow {{K} ^0_{\textrm{S}}} {{\pi } ^+} {{\pi } ^-}$ (${D} \rightarrow {{K} ^0_{\textrm{S}}} {{K} ^+} {{K} ^-}$) mode, and is displayed in Fig. 2.

Fig. 2

Dalitz plot binning schemes used for (left) ${D} \rightarrow {{K} ^0_{\textrm{S}}} {{\pi } ^+} {{\pi } ^-}$ and (right) ${D} \rightarrow {{K} ^0_{\textrm{S}}} {{K} ^+} {{K} ^-}$ decays

The total amplitude of the ${{B} ^0} \rightarrow ({{K} ^0_{\textrm{S}}} h^+ h^-)_D{{K} ^+} {{\pi } ^-}$ decay is given by

$$\begin{aligned}&A({{B} ^0} \rightarrow ({{K} ^0_{\textrm{S}}} h^+ h^-)_D{{K} ^+} {{\pi } ^-} ) =A_c(p) e^{i\delta _c(p)} A_{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}D}}} (m_-^2, m_+^2) \nonumber \\&\quad + A_u(p) e^{i\left[ \delta _u(p) + \gamma \right] } A_{D} (m_-^2, m_+^2) , \end{aligned}$$

(5)

where that of the $C\!P$ conjugate ${\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0$ decay is found by transforming $\gamma \rightarrow -\gamma$ and $m_- \leftrightarrow m_+$. Squaring the total amplitude and integrating over the ${K} ^{*0}$ phase-space region gives

$$\begin{aligned}&\Gamma ({{B} ^0} \rightarrow ({{K} ^0_{\textrm{S}}} h^+ h^-)_{D} {{K} ^{*0}}) \propto \ |A_{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}D}}} |^2 + r_{{B} ^0} ^2 |A_{D} |^2 \nonumber \\&\quad + 2\kappa r_{{B} ^0} |A_{D} | |A_{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}D}}} | \Bigg [\cos (\delta _{{B} ^0} + \gamma )\cos \delta _{D} \nonumber \\&\quad - \sin (\delta _{{B} ^0} + \gamma )\sin \delta _{D} \Bigg ], \end{aligned}$$

(6)

where the Dalitz plot coordinates of the $D$ decay strong-phase difference, defined as $\delta _D(m_-^2,\ m_+^2) = \delta (m_-^2,\ m_+^2) - \overline{\delta }(m_-^2,\ m_+^2)$, and magnitudes have been omitted for brevity. The expression for the decay rate integrated over a Dalitz plot bin is given by

$$\begin{aligned}&\Gamma _{i}({{B} ^0} \rightarrow ({{K} ^0_{\textrm{S}}} h^+ h^-)_{D} {{K} ^{*0}}) \propto \ K_{-i} + r_{{B} ^0} ^2 K_{i} \nonumber \\&\quad + 2\kappa r_{{B} ^0} \sqrt{K_i K_{-i}}[\cos (\delta _{{B} ^0} + \gamma ) c_{i} - \sin (\delta _{{B} ^0} + \gamma ) s_{i}], \end{aligned}$$

(7)

where the $D$ decay magnitude and strong-phase difference have been replaced by integrals over Dalitz plot bins

$$\begin{aligned} K_i&= \int _i \textrm{d}m_-^2 \textrm{d}m_+^2 \, |A_{D} (m_-^2, m_+^2) | ^2, \end{aligned}$$

(8)

$$\begin{aligned} c_i&= \frac{1}{\sqrt{K_i K_{-i}}}\nonumber \\&\quad \times \int _{i} \textrm{d}m_-^2 \textrm{d}m_+^2 \, |A_{D} (m_-^2, m_+^2) | |A_{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}D}}} (m_-^2, m_+^2) | \cos \delta _D(m_-^2,m_+^2),\end{aligned}$$

(9)

$$\begin{aligned} s_i&= \frac{1}{\sqrt{K_i K_{-i}}}\nonumber \\&\quad \times \int _{i} \textrm{d}m_-^2 \textrm{d}m_+^2 \, |A_{D} (m_-^2, m_+^2) | |A_{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}D}}} (m_-^2, m_+^2) | \sin \delta _D(m_-^2,m_+^2). \end{aligned}$$

(10)

Swapping the coordinates $m_- \leftrightarrow m_+$ is equivalent to a bin transformation $i \leftrightarrow -i$, and results in the relations $c_i = c_{-i}$ and $s_i = - s_{-i}$.

Experimentally, candidate yields are determined instead of the decay rates. Detector, reconstruction and selection related efficiencies are accounted for by using a set of parameters referred to as $F_i$ that are determined in each bin. They are defined as

$$\begin{aligned} F_{i} \equiv \frac{\int _{i} \textrm{d}m_-^2 \textrm{d}m_+^2 \, |A_{D} (m_-^2, m_+^2) | ^2 \eta (m_-^2, m_+^2)}{\sum _j \int _j \textrm{d}m_-^2 \textrm{d}m_+^2 \, |A_{D} (m_-^2, m_+^2) | ^2\eta (m_-^2, m_+^2)}, \end{aligned}$$

(11)

where $\eta (m_-^2, m_+^2)$ is the efficiency profile which varies over the $D$ decay phase space. The $F_i$ are the efficiency-modulated $K_i$ parameters, and are dependent on the experimental resolution and selection efficiency. A similar efficiency adjustment is not included in the $c_i$ and $s_i$ parameters because the effect is small, however a systematic uncertainty is included to account for this assumption. The $F_i$ parameters have been determined using ${B} ^{\pm }\rightarrow {D} \pi ^\pm$ decays [21]. As these parameters are selection dependent, they are only valid for use in this analysis under the assumption that the relative variation in $\eta (m_-^2,m_+^2)$ between $D$ meson decays in ${B} ^{\pm }\rightarrow {D} \pi ^\pm$ and ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ is the same. Differences in the efficiency profiles are minimised by employing a similar selection between the ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ and ${B} ^{\pm }\rightarrow {D} \pi ^\pm$ decays and small residual differences are determined using simulation samples and used to assign systematic uncertainties on the $C\!P$ violation observables. The yields of ${B} ^0$ and ${\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0$ decays in a Dalitz plot bin are given by

$$\begin{aligned} N_i({{B} ^0}) = h^{{B} ^0} \left[ F_{-i} + (x_+ ^2 + y_+ ^2)F_{i} + 2\kappa \sqrt{F_{i}F_{-i}} (x_+ c_{i} - y_+ s_{i})\right] , \end{aligned}$$

(12)

$$\begin{aligned} N_i({{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0}) = h^{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0} \left[ F_{i} + (x_- ^2 + y_- ^2)F_{-i} + 2\kappa \sqrt{F_{i}F_{-i}} (x_- c_{i} + y_- s_{i})\right] , \end{aligned}$$

(13)

where the $C\!P$ violation observables [22], $x_\pm$ and $y_\pm$, are related to the physics parameters by

$$\begin{aligned} x_\pm \equiv r_{{B} ^0} \cos (\delta _{{B} ^0} \pm \gamma ), \end{aligned}$$

(14)

$$\begin{aligned} y_\pm \equiv r_{{B} ^0} \sin (\delta _{{B} ^0} \pm \gamma ). \end{aligned}$$

(15)

These observables have improved statistical behaviour in comparison to determining $\gamma$, $r_{{B} ^0}$ and $\delta _{{B} ^0}$ directly. The two normalisation constants in Eqs. (12) and (13), $h^{{B} ^0}$ and $h^{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0}$, are the observed total yields of the ${B} ^0$ and ${\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0$ decay modes. The use of two separate normalization constants is intentional, as nearly all detector and production asymmetries are absorbed into these parameters leaving the measurement insensitive to these effects. Equations (12) and (13) are used to fit the data and determine the $C\!P$ violation observables. In the fit, the external input parameters $\kappa$ [20], $F_i$ [21], $c_i$ and $s_i$ [16, 17] are fixed to their measured central values.

The ${{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^0} {{K} ^{*0}}$ decay, which has identical final state particles is reconstructed alongside the signal channel. In principle, the method described in this section could also be applied to ${{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {D} {{K} ^{*0}}$ decays. However, the sensitivity to $\gamma$ is significantly lower due to reduced interference between the two final state paths. The values of the CKM elements [23] can be used to predict that the ratio of suppressed to favoured amplitudes, $r_{{{B} ^0_{s}}}\sim 0.02$, is over a factor of 10 less than in ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ decays [24]. In this analysis it is assumed that the $C\!P$ violation in the ${{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {D} {{K} ^{*0}}$ decay is zero and it is not treated as a signal decay mode. Thus in the remainder of the paper this decay is referred to as the ${{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^0} {{K} ^{*0}}}$ background with a flavour specific D meson.

3 Detector and simulation

The LHCb detector [25, 26] is a single-arm forward spectrometer covering the pseudorapidity range $2<\eta <5$, designed for the study of particles containing $b$ or $c$ quarks. The detector includes a high-precision tracking system consisting of a silicon-strip vertex detector surrounding the pp interaction region, a large-area silicon-strip detector located upstream of a dipole magnet with a bending power of about $4{\mathrm {\,Tm}}$, and three stations of silicon-strip detectors and straw drift tubes placed downstream of the magnet. The tracking system provides a measurement of the momentum, $p$, of charged particles with a relative uncertainty that varies from 0.5% at low momentum to 1.0% at 200$\text {\,Ge\hspace{-1.00006pt}V\!/}c$. The minimum distance of a track to a primary pp collision vertex (PV), the impact parameter (IP), is measured with a resolution of $(15+29/p_{\textrm{T}})\,\upmu \text {m}$, where $p_{\textrm{T}}$ is the component of the momentum transverse to the beam, in $\text {\,Ge\hspace{-1.00006pt}V\!/}c$. Different types of charged hadrons are distinguished using information from two ring-imaging Cherenkov detectors. Photons, electrons and hadrons are identified by a calorimeter system consisting of scintillating-pad and preshower detectors, an electromagnetic and a hadronic calorimeter. Muons are identified by a system composed of alternating layers of iron and multiwire proportional chambers.

The online event selection is performed by a trigger, which consists of a hardware stage, based on information from the calorimeter and muon systems, followed by a software stage, which applies a full event reconstruction. The events that are selected for the analysis either have final-state tracks of the signal decay that are subsequently associated with an energy deposit in the calorimeter system that satisfies the hardware stage trigger, or are selected because one of the other particles in the event, not reconstructed as part of the signal candidate, fulfills any hardware stage trigger requirement. At the software stage, it is required that at least one particle should have high $p_{\textrm{T}}$ and high $\chi ^2_{\text {IP}}$, where $\chi ^2_{\text {IP}}$ is defined as the difference in the primary vertex fit $\chi ^2$ with and without the inclusion of that particle. A multivariate algorithm [27] is used to select secondary vertices consistent with being a two-, three-, or four-track b-hadron decay.

Simulated data are required to determine the invariant-mass shapes of signal and background components, and to compute relative selection efficiencies. In the simulation, pp collisions are generated using Pythia [28, 29] with a specific LHCb configuration [30]. Decays of unstable particles are described by EvtGen [31], in which final-state radiation is generated using Photos [32]. The decays ${D} \rightarrow {{K} ^0_{\textrm{S}}} {{\pi } ^+} {{\pi } ^-}$ and ${D} \rightarrow {{K} ^0_{\textrm{S}}} {{K} ^+} {{K} ^-}$ are generated uniformly over phase space. The interaction of the generated particles with the detector, and its response, are implemented using the Geant4 toolkit [33, 34] as described in Ref. [35].

Fig. 3

Dalitz plots of selected candidates for (left) ${\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0$ and (right) ${B} ^0$ decays followed by the (upper) ${D} \rightarrow {{K} ^0_{\textrm{S}}} {{\pi } ^+} {{\pi } ^-}$ and (lower) ${D} \rightarrow {{K} ^0_{\textrm{S}}} {{K} ^+} {{K} ^-}$ decay. Candidates that have an invariant mass within a 30 $\text {\,Me\hspace{-1.00006pt}V\!/}c^2$ region either side of the ${B} ^0$ mass are displayed. The kinematic boundaries are plotted as continuous red solid curves

4 Candidate selection

All tracks and decay vertices are required to be of good quality, and the reconstructed mass of the ${K} ^0_{\textrm{S}}$, $D$ and ${K} ^{*0}$ candidates must be close to their known values [23]. The ${K} ^0_{\textrm{S}}$ candidates are formed from two oppositely charged pions, where the tracks are reconstructed using hits in the vertex detector and other downstream tracking stations, or only the latter. These track types are referred to as long and downstream, respectively, and are treated separately since the former leads to better mass, momentum and vertex resolution on the ${{K} ^0_{\textrm{S}}}$ candidate and higher reconstruction efficiency. A $D$ meson candidate is formed by combining a ${K} ^0_{\textrm{S}}$ candidate with two oppositely charged pions. Particle identification (PID) requirements are placed on the particles, to reduce background from ${D} \rightarrow {{K} ^0_{\textrm{S}}} {{K} ^+} {{\pi } ^-}$ decays, semileptonic $D$ decays, and hadronic decays in flight to leptons. A requirement is placed on the displacement of the $D$ meson vertex from the $B$ meson vertex to reduce background from $B$ decays to the final state particles without the intermediate $D$ meson. The $D$ meson candidate is then combined with a ${K} ^{*0}$ candidate, which is formed by combining a pion and kaon, with strict PID requirements to suppress ${{B} ^0} \rightarrow {D} {{\pi } ^+} {{\pi } ^-}$ backgrounds and thus allow for correct identification of the $B$-meson flavour. A criterion is applied on the ${K} ^{*0}$ helicity angle, $\theta ^*$, defined as the angle between the kaon from the ${K} ^{*0}$ decay and the opposite of the $B$ momentum in the ${K} ^{*0}$ rest frame, to exploit differences in the angular distributions of the signal and background candidates. In signal decays, a $B$ meson decays to a vector and pseudo-scalar final state, so the corresponding distribution of $|\cos \theta ^*|$ peaks at 1, whereas it is flat for background candidates formed from random combinations of tracks, referred to as combinatorial background. Therefore, candidates are rejected if the value of $|\cos \theta ^*|$ is below a threshold that is chosen to match that applied in Ref. [20].

A kinematic fit is performed to improve the resolution of the invariant-mass of the ${B} ^0$ candidates and Dalitz plot coordinates. In this fit, the masses of the $D$ and ${K} ^0_{\textrm{S}}$ candidates are constrained to their known values [23], and the momentum of the ${B} ^0$ meson is required to be parallel to the vector linking the ${B} ^0$ decay vertex and the associated PV, which is defined as the PV leading to the smallest IP of the ${B} ^0$ candidate.

A boosted decision tree (BDT) classifier [36, 37] is employed to reduce combinatorial background. It is trained on ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ decays with ${D} \rightarrow {{K} ^0_{\textrm{S}}} {{\pi } ^+} {{\pi } ^-}$ separately for candidates with long and downstream ${K} ^0_{\textrm{S}}$ track types, and is applied to both $D$ decay modes. Signal is represented by simulated decays, and combinatorial background is represented by ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ candidates in data with an invariant mass between 5800 and 6200$\text {\,Me\hspace{-1.00006pt}V\!/}c^2$. The set of input variables are predominantly based on the decay topology and kinematics. They are taken from the BDT classifier applied in the analysis of $B^\pm \rightarrow {D} h^\pm$ decays outlined in Ref. [21]. Since there is an extra track in ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ decays, the $p$, $p_{\textrm{T}}$ and $\chi ^2_{\text {IP}}$ of the pion from the ${K} ^{*0}$ decay are also included. The optimal BDT classifier selection criterion is chosen to minimise the statistical uncertainty on $\gamma$ and is determined with pseudoexperiments.

Figure 3 displays Dalitz plot distributions of fully selected ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ candidates that have an invariant mass within $\pm 30$ $\text {\,Me\hspace{-1.00006pt}V\!/}c^2$ of the ${B} ^0$ mass [23], where the signal purity is approximately 60%. They are displayed in four categories given by the $D$ decay and $B$-meson flavour, and candidates from both ${K} ^0_{\textrm{S}}$ track types are combined for visualisation purposes only.

5 Fit to determine the CP violation observables

A two-stage fit strategy is adopted to determine the $C\!P$ violation observables. The same model is used for both stages in an unbinned maximum likelihood fit to the invariant-mass distribution of ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ candidates in the region 5200–5800$\text {\,Me\hspace{-1.00006pt}V\!/}c^2$. The lower end of the fit range is chosen to remove background from $C\!P$ violating ${{B} ^0} \rightarrow {{D} ^*} {{K} ^*}$ decays. The first stage, referred to as the global fit, is used to understand the background composition and parameterize the invariant-mass distribution. The candidates in this fit are divided into four groups, given by the $D$ decay mode and the${K} ^0_{\textrm{S}}$ track type. In the second stage the data are simultaneously fitted across 80 categories given by the $D$ decay mode, ${K} ^0_{\textrm{S}}$ track type, $B$-meson flavour and Dalitz plot bin.

Due to the similarities in the final state, the signal and ${{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^0} {{K} ^{*0}}}$ decays have a similar invariant mass shape. Both are modelled by a function with a Gaussian core and asymmetric tails,

$$\begin{aligned} f(m) = {\left\{ \begin{array}{ll} \exp (-\delta m^2 (\frac{1+\beta \delta m^2}{f_L})), ~~ \delta m<0 \\ \exp (-\delta m^2 (\frac{1+\beta \delta m^2}{f_R})), ~~ \delta m>0 \\ \end{array}\right. } \text {where \ } \begin{gathered} \delta m = m-\mu , \\ f_L = 2\sigma _L^2 + \alpha _L \delta m^2, \\ f_R = 2(\frac{\sigma _L}{r})^2 + \alpha _R \delta m^2 \\ \end{gathered}\nonumber \\ \end{aligned}$$

(16)

where $\mu$ is the mean, $\beta$ is the asymmetry, $\sigma _{L,R}$ and $\alpha _{L,R}$ describe the left and right widths and tails, respectively. The $\beta$, $\alpha$ and width ratio, $r=\frac{\sigma _L}{\sigma _R}$, parameters are fixed to values determined from simulation. The mean of the distribution representing ${{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^0} {{K} ^{*0}}}$ candidates is a free parameter shared between the categories, whilst that of signal is constrained using the known mass difference, $m({{B} ^0_{s}}) - m({{B} ^0}) = (87.42 \pm 0.16)$ $\text {\,Me\hspace{-1.00006pt}V\!/}c^2$ [23]. Finally, the width is shared between signal and ${{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^0} {{K} ^{*0}}}$ decays for both $D$ decay modes but different for long and downstream ${K} ^0_{\textrm{S}}$ track categories.

The dominant physics background near the signal is from ${{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^{*0}} {{K} ^{*0}}$ candidates with the ${{D} ^{*0}}$ decaying to a ${{D} ^0}$ and an unreconstructed $\gamma$ or ${\pi } ^0$. The mass model of this background is described by four components depending on which particle is missed and whether the helicity state of the ${D} ^{*0}$ is 0 or $\pm 1$ (the distributions of the $\pm 1$ states are indistinguishable). The parameters describing the shape of each component are fixed to the values determined in simulation. It is not possible to determine the relative fractions of these four components reliably using data collected with the self-conjugate ${D} \rightarrow {{K} ^0_{\textrm{S}}} h^+ h^-$ modes, because the invariant-mass region below 5200 $\text {\,Me\hspace{-1.00006pt}V\!/}c^2$ is dominated by a mix of ${{{B} ^0} \rightarrow {D} ^{*}{{K} ^{*0}}}$ and ${{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^{*0}} {{K} ^{*0}}}$ decays and their distributions significantly overlap. However, in Cabibbo-favoured $D$ meson decays the low invariant-mass region is dominated by either ${\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}$ or ${B} ^0$ decays. This advantage is used by fitting the invariant mass distribution of candidates reconstructed as ${{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^0} (\rightarrow K^-\pi ^+){{K} ^{*0}}$ decays to determine the relative fractions of each partially reconstructed ${{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^{*0}} {{K} ^{*0}}$ component. The selection of candidates and the mass fit parameterisation follows that described in Ref. [10], but the data set is increased to include that collected in 2017 and 2018. Given the studies in Ref. [38], contamination from ${{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^{*0}} K \pi$ decays that do not include the ${K} ^{*0}$ resonance is small and this background will be subsumed into either the ${{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^{*0}} {{K} ^{*0}}}$ shapes or the combinatorial background. A small amount of ${{{B} ^0} \rightarrow {D} ^{*}{{K} ^{*0}}}$ decays leaks into the fit range. Their invariant-mass shape and yield ratio are determined in a similar way to that for the ${{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^{*0}} {{K} ^{*0}}}$ background by studying candidates reconstructed as ${{B} ^0} \rightarrow {D} (\rightarrow K\pi ){{K} ^{*0}}$.

Backgrounds from ${{B} ^\pm \rightarrow {D} K^\pm }$ decays plus a random pion, and misidentified ${{{B} ^0} \rightarrow {D} {{\pi } ^+} {{\pi } ^-}}$ decays are represented by shapes determined using simulation samples. The relative yield of both are fixed with respect to that of the ${{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^0} {{K} ^{*0}}}$ candidates. The ratio of ${{B} ^\pm \rightarrow {D} K^\pm }$ decays is determined using branching fractions, fragmentation fractions [39, 40] and selection efficiencies in simulation, where differences from data are determined to be negligible. The ratio for misidentified ${{{B} ^0} \rightarrow {D} {{\pi } ^+} {{\pi } ^-}}$ decays is determined from the results of fits to ${{B} ^0} \rightarrow ({D} \rightarrow K\pi ){{K} ^{*0}}$ decays. Finally, the combinatorial background is described by an exponential function, where the yield and slope are freely varying parameters in each category.

Fig. 4

Invariant-mass distributions of ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ candidates with (upper) ${D} \rightarrow {{K} ^0_{\textrm{S}}} {{\pi } ^+} {{\pi } ^-}$ and (lower) ${D} \rightarrow {{K} ^0_{\textrm{S}}} {{K} ^+} {{K} ^-}$ decays, separated by the (left) long and (right) downstream ${K} ^0_{\textrm{S}}$ track type. The data are overlaid with the global fit projection

The projections of the global fits are displayed in Fig. 4. Table 1 details the yields of each component in a 30$\text {\,Me\hspace{-1.00006pt}V\!/}c^2$ region either side of the ${B} ^0$ mass [23]. The total signal yield and purity are $434 \pm 32$ and $(57 \pm 5)\%$, respectively. The dominant backgrounds in the signal region are from combinatorial candidates and ${{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^{*0}} {{K} ^{*0}}}$ decays. Other sources are negligible in comparison.

Table 1 Yield of each component in a 30$\text {\,Me\hspace{-1.00006pt}V\!/}c^2$ region either side of the ${B} ^0$ mass as determined by the global fit in four categories. Yields are either determined directly or through a combination of fit parameters. The uncertainties are determined through propagation and further modulated by integration within the region. Some backgrounds have negligible yields in the aforementioned invariant-mass region

Simulation is used to verify that the component shapes do not vary across the Dalitz plot. Therefore, the same model is applied in the fit to extract the $C\!P$ violation observables as for the global fit. The yield of each component, excluding combinatorial background, in a Dalitz plot bin is parameterised by the integrated yield multiplied by the expected fraction of candidates in that bin. For example, the distribution of signal candidates is described by Eqs. (12) and (13) where $h_{{B} ^0}$ and $h_{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0}$ are freely varying parameters.

In the fit, the $C\!P$ violation observables are free parameters shared across all fit categories and the $F_i$ [21], $c_{i}$, $s_i$ [16, 17] and $\kappa$ [20] parameters are fixed.

The integrated yields of ${{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^0} {{K} ^{*0}}}$ decays are freely varying parameters in four categories given by the $D$ decay mode and ${K} ^0_{\textrm{S}}$ track type, whilst those of the remaining physics backgrounds are fixed to the results of the global fit. For each of the background components, excluding combinatorial background, the fractional yield in a Dalitz plot bin is fixed. The effect of $C\!P$ violation in interference between the final state paths in ${{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^0} {{K} ^{*0}}}$ and ${{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^{*0}} {{K} ^{*0}}}$ decays is expected to be small because $r_{{B} ^0_{s}} \sim 0.02$. Therefore, ${\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}$ particles are assumed to decay exclusively to ${D} ^0$ mesons, thus the fraction of these candidates in a Dalitz plot bin is given by $F_i$. The level of $C\!P$ violation in ${{{B} ^0} \rightarrow {D} ^{*}{{K} ^{*0}}}$ decays is likely at a similar level to the signal, but assigned as zero in the fit due to the very small yield of this decay in the fit range. Therefore, the fractional yield of this component in a Dalitz plot bin is $F_{-i}$. A systematic uncertainty is assigned for this assumption as discussed in Sect. 6. For the ${{B} ^0} \rightarrow {D} {{\pi } ^+} {{\pi } ^-}$ candidates, the $D$ meson is assumed to be an equal mixture of ${D} ^0$ and ${\hspace{1.79993pt}\overline{\hspace{-1.79993pt}D}} {}^0$ mesons because either pion could be misidentified. Therefore, the fraction of these decays in a Dalitz plot bin is $0.5(F_i + F_{-i})$. The ${B} ^{\pm }\rightarrow {D} K^\pm$ background is $C\!P$ violating and its distribution over the Dalitz plot is therefore parameterised similarly to Eqs. (12) and (13) using values of the $C\!P$ violation observables determined from Ref. [24], with $\kappa =1$. Finally, the Dalitz plot distribution of combinatorial background is unknown, thus the corresponding yield in each bin is a free parameter.

Fig. 5

Left: two-dimensional 68.3% and 95.5% statistical confidence regions for the measured $(x_\pm , y_\pm )$ values, determined by scanning the profile likelihood function. The orange (blue) contours correspond to the observables related to ${B} ^0$ (${\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0$) decays. Right: raw asymmetry in each effective bin pair. It is determined using the fitted $C\!P$ violation observables (red histogram) and the results of an alternative fit where the signal yield in each Dalitz plot bin is a free parameter (black points, with statistical uncertainties that are capped to the physical limits where appropriate)

After correcting for small biases (the largest of which is 12% of the statistical uncertainty) and uncertainty undercoverage (the largest inflation was 3%) using pseudoexperiments, the $C\!P$ violation observables are measured to be $x_+ = 0.074 \pm 0.086$, $x_- = -0.215 \pm 0.086$, $y_+ =-0.336 \pm 0.105$ and $y_- = -0.012 \pm 0.128$, with the statistical correlation coefficients displayed in the Appendix. The left plot in Fig. 5 displays the 68.3% and 95.5% confidence regions for the $C\!P$ violation observables determined by scanning the profile likelihood function. The opening angle between the lines joining the points ($x_+$, $y_+$) and ($x_-$, $y_-$) with the origin corresponds to $2\gamma$. To understand the distribution of signal across the Dalitz plot the raw asymmetry, $N_i({{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0}) - N_{-i}({{B} ^0})/N_i({{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0}) + N_{-i}({{B} ^0})$, is calculated for each effective bin pair. An effective bin labelled i, is defined to compare the yield of ${\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0$ decays in a bin i with the yield of ${B} ^0$ decays in a bin $-i$. Figure 5 displays the asymmetries calculated using the binned yields from the default fit, and for illustrative purposes, those determined in an alternative fit where the signal yield in each region of the Dalitz plot is a free parameter. The good agreement between the yield of signal in each bin determined from the $C\!P$ violation observables and those determined from the alternative fit demonstrates that Eqs. (12) and (13) are an appropriate model for the data. It is possible to see regions of the Dalitz plot where the asymmetry does deviate from zero. However, $C\!P$ violation in this measurement is not yet established with the current precision.

6 Systematic uncertainties

A summary of the systematic uncertainties is displayed in Table 2. These are primarily evaluated with two methods: the fit to the data is repeated many times using a model with fixed parameters smeared according to their uncertainties and the root-mean-square (RMS) of the $C\!P$ violation observable distributions are taken to be the uncertainties, or many pseudodata sets tuned to the data are fitted using a model with an alternative configuration and the biases in the $C\!P$ violation observable distributions are taken to be the uncertainty.

Two systematic uncertainties are associated with the $D$ decay strong-phase inputs. The effect of their finite precision is determined by generating a set of $c_{i}$ and $s_i$ values smeared according to their uncertainties and correlations. The corresponding systematic uncertainties are 0.005, 0.004, 0.017 and 0.024 for $x_+$, $x_-$, $y_+$ and $y_-$, respectively. These are larger for $y_\pm$ because the $s_i$ values are known less precisely than those of $c_i$, but they remain significantly smaller than the statistical uncertainties.

An uncertainty arises because the effect of $\eta ({m_-^2,m_+^2})$ is not accounted for in the $D$ decay strong-phase parameters. Alternative $c_{i}$ and $s_i$ are calculated using an amplitude model [41] with a flat efficiency profile ($c_i^\mathrm{{flat}}, \ s_i^\mathrm{{flat}}$) and an efficiency profile determined using simulated signal decays ($c_i^\mathrm{{eff}}, \ s_i^\mathrm{{eff}}$). The subsequent systematic uncertainty is evaluated by fitting the data many times using a model with alternative $c_{i}$ and $s_i$ coefficients that are generated from a Gaussian with a width equal to the efficiency correction: $\delta c_i = c_i^\mathrm{{flat}} - c_i^\mathrm{{eff}}$ and $\delta s_i = s_i^\mathrm{{flat}} - s_i^\mathrm{{eff}}$.

Selection differences between ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ and ${B} ^{\pm }\rightarrow {D} \pi ^\pm$ candidates can alter the relative efficiencies in each Dalitz plot bin, introducing a bias on the $F_i$ parameters appropriate for these decay channels. The ratio of the squared $({{\pi } ^+} {{\pi } ^-})_{D}$ invariant-mass distribution in simulated ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ and ${B} ^{\pm }\rightarrow {D} \pi ^\pm$ decays are used to produce an alternative efficiency profile. This is subsequently applied to an amplitude model [41] to compute different $F_i$ values. The relative efficiency differences between signal and ${{{\hspace{1.79993pt}\overline{\hspace{-1.79993pt}B}} {}^0_{s}} \rightarrow {{D} ^0} {{K} ^{*0}}}$ decays are negligible. It is the dominant systematic uncertainty for the $x_\pm$ observables, but is significantly lower than the equivalent uncertainty determined in Ref. [11] where the efficiency profile from ${{B} ^0} \rightarrow {{D} ^*} ^-\mu ^+\nu _\mu$ decays was used.

Various systematic uncertainties related to the fit model are computed. The dominant contributions are the choice of signal shape and the effect of fixing the combinatorial background slope, signal mean and resolution to the global fit results, and are both evaluated using alternative models. In the former a different signal distribution is used, and in the latter the slopes in each of the four global fit categories are freely varying parameters that are shared between Dalitz plot bins. The remaining fit model systematics are those associated with the fixed background ratios, which are evaluated using sets of parameters smeared according to their uncertainties.

Table 2 Systematic uncertainties for the $C\!P$ violation observables. Statistical uncertainties are given for reference

In the fit model, $C\!P$ violation in partially reconstructed ${{{B} ^0} \rightarrow {D} ^{*}{{K} ^{*0}}}$ decays is neglected because there are few candidates in the fit range. The effect of this assumption is measured using an alternative model where these candidates have the same distribution in phase space as the signal decays. The amplitude ratio and strong-phase difference for these decays are unknown, but a similar interference as in ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ decays is expected since they have a similar amplitude ratio. Hence, the $C\!P$ violation observables determined by the nominal fit are used and the resulting uncertainty is small.

The systematic uncertainty associated with the limited knowledge of the coherence factor, $\kappa$, is determined to be small using an alternative model where its value is displaced by one standard deviation, $\kappa _\mathrm{{model}} = \kappa - \sigma (\kappa )$ [20]. Larger values of $\kappa$ are not included since its uncertainty is heavily asymmetric, and the lower uncertainty is found to dominate the spread of $C\!P$ violation observables.

In the selection, a requirement is placed on the displacement of the $D$ meson vertex from the $B$ meson vertex to reduce background from $B$ decays to the final state particles without the intermediate $D$ meson, which are referred to as charmless candidates. Studies of the $D$ meson invariant-mass sideband determine that the total charmless yield in the sample is $17 \pm 9$. A systematic uncertainty is assigned using an alternative model where charmless candidates are introduced in the signal region. The yields of these candidates are distributed uniformly over the Dalitz plot and given the small expected yields it is unnecessary to account for potential $C\!P$ violation in the charmless decays.

Fig. 6

Profile likelihood contours for (left) $\gamma$ versus $r_{{B} ^0}$ and (right) $\gamma$ versus $\delta _{{B} ^0}$ corresponding to 68.3% and 95.5% confidence regions

Measurements of the Dalitz plot coordinates are affected by the detector momentum resolution and can cause candidates to be assigned to the wrong bin. To first order, the $F_i$ values account for this, but the net migration between Dalitz plot bins can differ in ${B} ^{\pm }\rightarrow {D} \pi ^\pm$ and ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ decays since they exhibit different levels of $C\!P$ violation. The expected difference in the $F_i$ values in ${B} ^{\pm }\rightarrow {D} \pi ^\pm$ and ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ due to these second order effects is determined using the momentum resolution in simulation, the $C\!P$ violation observables of ${B} ^{\pm }\rightarrow {D} \pi ^\pm$ [24] and those of this analysis and the $D$ decay model from Ref. [41]. The expected differences are used to generate pseudoexperiments which are then fit with the nominal procedure to assign the systematic uncertainty due to momentum resolution.

The corrections applied to the $C\!P$ violation observables in Sect. 5 depend on the physics inputs used in the pseudodata studies. Therefore, a systematic uncertainty is assigned. The values of and correlations between $\gamma$, $r_{{B} ^0}$ and $\delta _{{B} ^0}$ from Ref. [24] are used to generate sets of alternative input $C\!P$ violation observables. The bias study is repeated many times to create a distribution of corrections, the RMS of which corresponds to the systematic uncertainty.

The total systematic uncertainties from all sources excluding those associated with the limited knowledge of the $c_i$ and $s_i$ coefficients is determined by summing all the contributions in quadrature. They are 0.011, 0.013, 0.009 and 0.011 for $x_+$, $x_-$, $y_+$ and $y_-$, respectively, and their correlations are given in the Appendix.

7 Interpretation

The $C\!P$ violation observables are determined to be

$$\begin{aligned} \begin{array}{rcrcrcrcr} x_+ &{} = &{} 0.074 &{} \pm &{} 0.086 &{} \pm &{} 0.005 &{} \pm &{} 0.011, \\ x_- &{} = &{} -0.215 &{} \pm &{} 0.086 &{} \pm &{} 0.004 &{} \pm &{} 0.013, \\ y_+ &{} = &{} -0.336 &{} \pm &{} 0.105 &{} \pm &{} 0.017 &{} \pm &{} 0.009, \\ y_- &{} = &{} -0.012 &{} \pm &{} 0.128 &{} \pm &{} 0.024 &{} \pm &{} 0.011, \\ \end{array} \end{aligned}$$

where the first uncertainty is statistical, the second is the systematic contribution from the $D$ decay strong-phase inputs and the third is from the experimental systematic uncertainties. The measured $C\!P$ violation observables are used in a maximum likelihood fit to determine the physics parameters $\gamma$, $r_{{B} ^0}$ and $\delta _{{B} ^0}$. The $C\!P$ violation observables are invariant under the transformation $\gamma \rightarrow \gamma +180^\circ$ and $\delta _{{B} ^0} \rightarrow \delta _{{B} ^0} +180^\circ$ which leads to two unambiguous solutions for the physics observables. In the region where $0<\gamma <180^\circ$ is satisfied, the best fit values are

$$\begin{aligned} \gamma= & {} (49^{+ 22}_{-19})^\circ , \\ r_{{B} ^0}= & {} 0.271^{+ 0.065}_{-0.066}, \\ \delta _{{B} ^0}= & {} (236^{+19}_{-21})^\circ , \end{aligned}$$

where the uncertainties are calculated using a frequentist method described in Ref. [24]. The corresponding 68.3% and 95.5% confidence regions in the $\gamma$ vs. $r_{{B} ^0}$ and $\gamma$ vs. $\delta _{{B} ^0}$ planes are displayed in Fig. 6.

In the most recent combination of LHCb results [24], the mean value of $\gamma$ determined using the ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ channel was $\gamma =(82.0^{+8.1}_{-8.8})^\circ$, which is higher than the average value using ${B} ^\pm$ decays, $\gamma =(61.7^{+4.4}_{-4.8})^\circ$. The value of $\gamma$ presented here is in good agreement with the current LHCb average, $\gamma = (65.4^{+3.8}_{-4.2})^\circ$ [24], and will reduce the difference between measurements performed using different $b$-mesons. Furthermore, it is compatible with the value measured in Ref. [11], $\gamma =(71 \pm 20)^\circ$, although there is not a substantial precision improvement despite using a larger data set. This is explained by noting that the uncertainty on $\gamma$ is inversely proportional to the value of $r_{{B} ^0}$, which had a higher central value in Ref. [11] than the current measurement. The value of $r_{{B} ^0}$ presented in this paper is consistent with previous determinations from LHCb [10], BaBar [9] and Belle [7, 8]. The precision of the $C\!P$ violation observables have significantly improved and therefore the results presented here will have a larger weight in future $\gamma$ combinations.

8 Summary

Proton-proton collision data corresponding to an integrated luminosity of 9$\text {\,fb} ^{-1}$ collected by the LHCb experiment at centre-of-mass energies of $\sqrt{s}=7, \, 8$ and 13$\text {\,Te\hspace{-1.00006pt}V}$ are used to perform a binned, model-independent $C\!P$ violation study of ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ decays to measure the CKM angle $\gamma$. Strong-phase information of ${D} \rightarrow {{K} ^0_{\textrm{S}}} h^+ h^-$ decays (where $h = \pi ,\ K$) from the CLEO [19] and BESIII [16,17,18] experiments is used as inputs. The measured value is $\gamma = (49^{+22}_{-19})^\circ$, where the uncertainty is statistically dominated and systematic contributions are an order of magnitude smaller. The $C\!P$ violation observables measured here are consistent with and supersede those presented in Ref. [11].

Appendix: Correlation matrices

Tables 3 and 4 display the correlation coefficients between the statistical and systematic uncertainties (excluding the strong-phase inputs) on the $C\!P$ violation observables, respectively.

A systematic uncertainty is assigned to account for the finite precision on the $D$ decay strong-phase inputs, $c_i$ and $s_i$ [16, 17]. It is given by the RMS of the distributions of $C\!P$ violation observables obtained from fitting the data many times using a model with $c_i$ and $s_i$ values that are smeared according to their uncertainties and correlations. This procedure is common between model-independent $\gamma$ measurements. Therefore, the alternative $c_i$ and $s_i$ used for this study are taken from an analysis of $B^\pm \rightarrow {D} h^\pm$ decays at LHCb [21], which allows correlation coefficients between $C\!P$ violation observables of both analysis to be computed. Thus, in combinations the correlation of this systematic uncertainty can be accounted for. These are displayed in Table 5.

Table 3 Statistical correlation matrix for the $C\!P$ violation observables

Table 4 Correlations between the $C\!P$ violation observables for systematic uncertainties excluding the strong-phase inputs

Table 5 Correlations in the $C\!P$ violation observables for the strong-phase related systematic uncertainties in ${{{B} ^0} \rightarrow {D} {{K} ^{*0}}}$ and $B^\pm \rightarrow {D} h^\pm$ [21]