1 Introduction

The discovery of a Higgs boson by the ATLAS [1] and CMS [2] experiments at the Large Hadron Collider (LHC) at CERN marked the beginning of a new era of experimental studies of the properties of this new particle. In the Standard Model (SM), the Higgs boson is a CP-even scalar particle, \(J^{CP}=0^{++}\).Footnote 1 Theories of physics beyond the SM (BSM) often require an extended Higgs sector featuring several neutral Higgs bosons. Such cases may include CP-mixing in the Higgs boson interactions, which could result in observable differences in the kinematics of final-state particles produced in their decays. A review of the phenomenology in the determination of Higgs boson spin and CP properties can be found in Ref. [3] and references therein.

Previous determinations of the Higgs boson spin and CP quantum numbers by the ATLAS and CMS Collaborations are reported in Refs. [4, 5]. Results on the same subject have also been published by the D0 and CDF Collaborations in Ref. [6]. All these studies indicate the compatibility of the spin and CP properties of the observed Higgs boson with the SM predictions. The ATLAS measurement excluded several alternative spin and parity hypotheses in favour of the quantum numbers predicted by the SM. In addition to the exclusion of several non-SM spin hypotheses, the CMS measurement probed the tensor structure of the Higgs boson decay to SM vector bosons in the spin-0 scenario. This paper complements the previous ATLAS study of the Higgs boson spin and parity. The new study takes advantage of improvements to the analysis strategy and to the modelling used to describe alternative spin hypotheses, and includes studies on CP-mixing for the spin-0 scenario. The improved theoretical framework is based on the Higgs boson characterisation model described in Refs. [3, 7].

The study of the spin and parity properties of the Higgs boson presented in this paper is based on the \(H \rightarrow \gamma \gamma \), \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) and \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) decay channels and their combination. The \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) analysis is described in detail in a separate publication [8]. These analyses are based on 4.5 and 20.3 fb\(^{-1}\) of pp collision data collected by the ATLAS experiment at centre-of-mass energies of 7 and 8 TeV, respectively. For the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) studies only the data collected at a centre-of-mass energy of 8 TeV are used. The SM hypothesis \(J^P=0^+\) is compared to alternative spin-0 models: a pseudoscalar boson \(J^P=0^-\) and a BSM scalar boson \(J^P=0^+_h\) [9, 10], which describes the interaction of the Higgs boson with the SM vector bosons with higher-dimension operators discussed in Sect. 3.1. Graviton-like tensor models with \(J^P=2^+\) with universal and non-universal couplings [3, 7] are also considered. In these tests of fixed spin and parity hypotheses it is assumed that the resonance decay involves only one CP eigenstate.

In addition to the fixed spin and parity hypothesis tests, the possible presence of BSM terms in the Lagrangian describing the HVV vertexFootnote 2 of the spin-0 resonance is also investigated. The HVV interaction is described in terms of an effective Lagrangian that contains the SM interaction and BSM CP-odd and CP-even terms [3, 7]. The relative fractions of the CP-odd and CP-even BSM contributions to the observed Higgs boson decays are constrained, and limits on the corresponding BSM tensor couplings are derived.

This paper is organised as follows. In Sect. 2 the ATLAS detector is described. In Sect. 3 the theoretical framework used to derive the spin and parity models, as well as the parameterisation used to describe the HVV coupling tensor structure, are discussed. In Sect. 4, the choice of Monte Carlo generators for the simulation of signal and backgrounds is described. The analyses of fixed spin and parity hypotheses for the three decay channels and their combination are presented in Sect. 5. Individual and combined studies of the tensor structure of the HVV interaction are presented in Sect. 6. Concluding remarks are given in Sect. 7.

2 The ATLAS detector

The ATLAS detector is described in detail in Ref. [11]. ATLAS is a multi-purpose detector with a forward-backward symmetric cylindrical geometry. It uses a right-handed coordinate system with its origin at the nominal interaction point (IP) in the centre of the detector and the z-axis along the beam pipe. The x-axis points from the IP to the centre of the LHC ring, and the y-axis points upward. Cylindrical coordinates \((r,\phi )\) are used in the transverse plane, \(\phi \) being the azimuthal angle around the beam pipe. The pseudorapidity is defined as \(\eta = -\ln \tan (\theta /2)\), where \(\theta \) is the polar angle.

At small radii from the beamline, the inner detector (ID), immersed in a 2 T magnetic field produced by a thin superconducting solenoid located in front of the calorimeter, is made up of fine-granularity pixel and microstrip detectors. These silicon-based detectors cover the range \(|\eta |<2.5\). A gas-filled straw-tube transition-radiation tracker (TRT) complements the silicon tracker at larger radii and also provides electron identification based on transition radiation. The electromagnetic (EM) calorimeter is a lead/liquid-argon sampling calorimeter with an accordion geometry. The EM calorimeter is divided into a barrel section covering \(|\eta | <1.475 \) and two end-cap sections covering \(1.375<|\eta | <3.2 \). For \(|\eta | <2.5 \) it is divided into three layers in depth, which are finely segmented in \(\eta \) and \(\phi \). An additional thin presampler layer, covering \(|\eta | <1.8 \), is used to correct for fluctuations in energy losses of particles before they reach the calorimeter. Hadronic calorimetry in the region \(|\eta | <1.7 \) uses steel absorbers and scintillator tiles as the active medium. Liquid argon with copper absorbers is used in the hadronic end-cap calorimeters, which cover the region \(1.5<|\eta | <3.2 \). A forward calorimeter using copper or tungsten absorbers with liquid argon completes the calorimeter coverage up to \(|\eta | =4.9 \). The muon spectrometer (MS) measures the deflection of muon trajectories with \(|\eta | <2.7 \), using three stations of precision drift tubes, with cathode strip chambers in the innermost layer for \(|\eta | > 2.0 \). The deflection is provided by a toroidal magnetic field with an integral of approximately 3 and 6 Tm in the central and end-cap regions of the ATLAS detector, respectively. The muon spectrometer is also instrumented with dedicated trigger chambers, the resistive-plate chambers in the barrel and thin-gap chambers in the end-cap, covering \(|\eta | <2.4 \).

3 Theoretical models

In this section, the theoretical framework for the measurements of the spin and parity of the resonance is discussed. An effective field theory (EFT) approach is adopted to describe the interaction between the resonance and the SM vector bosons, following the Higgs boson characterisation model described in Refs. [3, 7]. Three possible BSM scenarios for the spin and parity of the boson are considered:

  • the observed resonance is a spin-2 particle,

  • the observed resonance is a pure BSM spin-0 CP-even or CP-odd Higgs boson,

  • the observed resonance is a mixture of the SM spin-0 state and a BSM spin-0 CP-even or CP-odd state.

The third case would imply CP-violation in the Higgs sector. In the case of CP mixing, the Higgs boson would be a mass eigenstate, but not a CP eigenstate. In all cases, only one resonance with a mass of about 125 GeV is considered. It is also assumed that the total width of the resonance is small compared to the typical experimental resolution of the ATLAS detector (of the order of 1–2 GeV in the four-lepton and \(\gamma \gamma \) final states, as documented in Ref. [12]). Interference effects between the BSM signals and SM backgrounds are neglected.

Table 1 Parameters of the benchmark scenarios for spin-0 boson tensor couplings used in tests (see Eq. (1)) of the fixed spin and parity models

The EFT approach, used by the Higgs boson characterisation model, is only valid up to a certain energy scale, \(\Lambda \). The models described in Ref. [7] assume that the resonance structure corresponds to one new boson (\(X(J^P)\) with \(J^{P} = 0^{\pm }\) or \(2^+\)), assuming that any other BSM particle only exists at an energy scale larger than \(\Lambda \). The \(\Lambda \) scale is set to 1 TeV to account for the experimental results obtained at the LHC and previous collider experiments, which do not show any evidence of new physics at lower energy scales.

The case where the observed resonance has \(J^{P} = 1^{\pm }\) is not studied in this paper. The \(H \rightarrow \gamma \gamma \) decay is forbidden by the Landau–Yang theorem [13, 14] for a spin-1 particle. Moreover, the spin-1 hypothesis was already studied in the previous ATLAS publication [4] in the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) and \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) decays and excluded at a more than 99 % confidence level.

3.1 The spin-0 hypothesis

In the spin-0 hypothesis, models with fixed spin and parity, and models with mixed SM spin-0 and BSM spin-0 CP-even and CP-odd contributions are considered. In Ref. [7], the spin-0 particle interaction with pairs of W or Z bosons is given through the following interaction Lagrangian:

$$\begin{aligned} \mathcal L_0^{V}= & {} \left\{ \cos (\alpha ) \kappa _\mathrm{SM}\left[ \frac{1}{2}g_{HZZ} Z_{\mu } Z^ {\mu } + g_{HWW} W^{+}_{\mu } W^ {- \mu }\right] \right. \nonumber \\&\left. -\frac{1}{4}\frac{1}{\Lambda }\left[ \cos (\alpha ) \kappa _{HZZ} Z_{\mu \nu }Z^{\mu \nu }+ \sin (\alpha ) \kappa _{AZZ} Z_{\mu \nu } \tilde{Z}^{\mu \nu }\right] \right. \nonumber \\&\left. -\frac{1}{2} \frac{1}{\Lambda } \left[ \cos (\alpha ) \kappa _{HWW} W^+_{\mu \nu }W^{-\mu \nu }\right. \right. \nonumber \\&\left. \left. + \sin (\alpha ) \kappa _{AWW} W^+_{\mu \nu } \tilde{W}^{ - \mu \nu }\right] \right\} X_0. \end{aligned}$$
(1)

Here \(V^{\mu }\) represents the vector-boson field \({(V=Z,W^{\pm }})\), the \(V^{\mu \nu }\) are the reduced field tensors and the dual tensor is defined as \(\tilde{V}^{\mu \nu }= \frac{1}{2}\varepsilon ^{\mu \nu \rho \sigma }V_{\rho \sigma }\). The symbol \(\Lambda \) denotes the EFT energy scale. The symbols \(\kappa _\mathrm{SM}\), \(\kappa _{HVV}\) and \(\kappa _{AVV}\) denote the coupling constants corresponding to the interaction of the SM, BSM CP-even or BSM CP-odd spin-0 particle, represented by the \(X_0\) field, with ZZ or WW pairs. To ensure that the Lagrangian terms are Hermitian, these couplings are assumed to be real. The mixing angle \(\alpha \) allows for production of CP-mixed states and implies CP-violation for \(\alpha \ne 0\) and \(\alpha \ne \pi \), provided the corresponding coupling constants are non-vanishing. The SM couplings, \(g_{HVV}\), are proportional to the square of the vector boson masses: \(g_{HVV} \propto m^{2}_{V}\). Other higher-order operators described in Ref. [7], namely the derivative operators, are not included in Eq. (1) and have been neglected in this analysis since they induce modifications of the discriminant variables well below the sensitivity achievable with the available data sample.

As already mentioned, for the spin-0 studies the SM Higgs boson hypothesis is compared to two alternatives: the CP-odd \(J^P=0^-\) and the BSM CP-even \(J^P=0^+_h\) hypotheses. All three models are obtained by selecting the corresponding parts of the Lagrangian described in Eq. (1) while setting all other contributions to zero. The values of the couplings corresponding to the different spin-0 models are listed in Table 1.

The investigation of the tensor structure of the HVV interaction is based on the assumption that the observed particle has spin zero. Following the parameterisation defined in Eq. (1), scenarios are considered where only one CP-odd or one CP-even BSM contribution at a time is present in addition to the SM contribution. To quantify the presence of BSM contributions in \(H \rightarrow ZZ^{*}\) and \(H \rightarrow WW^{*}\) decays, the ratios of couplings \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\) and \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) are measured. Here \(\tilde{\kappa }_{AVV}\) and \(\tilde{\kappa }_{HVV}\) are defined as follows:

$$\begin{aligned} \tilde{\kappa }_{ AVV}=\frac{1}{4} \frac{ \mathrm{v} }{\Lambda } \kappa _{AVV}\quad \mathrm{and}\quad \tilde{\kappa }_{HVV}=\frac{1}{4} \frac{ \mathrm{v} }{\Lambda } \kappa _{HVV}, \end{aligned}$$
(2)

where \(\mathrm{v}\) is the vacuum expectation value [15] of the SM Higgs field.

The mixing parameters \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\) and \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) correspond to the ratios of tensor couplings \(g_4/g_1\) and \(g_2/g_1\) proposed in the anomalous coupling approach described in Refs. [9, 10]. To compare the results obtained in this analysis to other existing studies, the final results are also expressed in terms of the effective cross-section fractions \((f_{g2},\phi _{g2})\) and \((f_{g4},\phi _{g4})\) proposed in Refs. [3, 9, 10]. Further details of these conversions are given in Appendix A.

The BSM terms described in Eq. (1) are also expected to change the relative contributions of the vector-boson fusion (VBF) and vector-boson associated production (VH) processes with respect to the gluon-fusion (ggF) production process, which is predicted to be the main production mode for the SM Higgs boson at the LHC. For large values of the BSM couplings, at the LHC energies, the VBF production mode can have a cross section that is comparable to the ggF process [16]. This study uses only kinematic properties of particles from \(H \rightarrow V V^{*}\) decays to derive information on the CP nature of the Higgs boson. The use of the signal rate information for different production modes, in the context of the EFT analysis, may increase the sensitivity to the BSM couplings at the cost of a loss in generality. For example the ratio of the VBF and VH production modes with respect to the ggF one can be changed by a large amount for non-vanishing values of the BSM couplings. In the studies presented in this paper the predictions of the signal rates are not used to constrain the BSM couplings.

As described in Sect. 6.2, only events with no reconstructed jets (the 0-jet category) are used in the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) analysis for the studies of the tensor structure; hence this analysis has little sensitivity to the VBF production mode. The \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) analysis also has little sensitivity to this production mode since it is mainly based on variables related to the four-lepton kinematics. The Boosted Decision Tree (BDT) algorithm [17] used to discriminate signals from the \(ZZ^{*}\) background, described in Sects. 5.4 and 6.3, includes the transverse momentum of the four-lepton system and is trained on simulated samples of ggF-produced signals. An enhancement of the VBF production mode would improve the separation between background and signal since it predicts larger values of the transverse momentum spectrum for events produced via VBF than via ggF [3].

3.2 The spin-2 hypothesis

In the Higgs boson characterisation model [7], the description of the interaction of a spin-2 particle with fermions and vector bosons is described by the following Lagrangian:

$$\begin{aligned} \mathcal L_2 = - \frac{1}{\Lambda }\left[ \sum _V \kappa _V \mathcal{T}^V_{\mu \nu } X^{\mu \nu } + \sum _f \kappa _f \mathcal{T}^f_{\mu \nu } X^{\mu \nu } \right] . \end{aligned}$$
(3)

The spin-2 tensor field \(X^{\mu \nu }\) is chosen to interact with the energy-momentum tensors, \(\mathcal{T}^V_{\mu \nu }\) and \(\mathcal{T}^f_{\mu \nu }\), of any vector boson V and fermion f, as inspired by gravitation theories. The strength of each interaction is determined by the couplings \(\kappa _V\) and \(\kappa _f\). In the simplest formulation, all couplings are equal. This scenario is referred to as universal couplings (UC), while scenarios with different values of the couplings are referred to as non-universal couplings (non-UC). In the UC scenario, the production of a spin-2 particle in pp collisions is expected to be dominated by QCD processes, with negligible contributions from electroweak (EW) processes (i.e. from processes involving EW boson propagators). Simulation studies based on MadGraph5_aMC@NLO [16], which implements the Lagrangian described in Eq. (3), predict for the production cross section in the UC scenario \(\sigma _{\mathrm {EW}}/\sigma _{\mathrm {QCD}}\simeq 3 \times 10^{-4}\). These studies also show that EW production of the spin-2 resonance would occur mainly in association with a massive EW boson (WX, ZX). Present observations do not show a dominant VH production mechanism, hence suggesting that \(\sigma _{\mathrm {EW}}\) is significantly smaller than \(\sigma _{\mathrm {QCD}}\). This paper considers only QCD production for all the spin-2 benchmark scenarios.

The UC models predict a branching ratio of about 5 % to photon pairs and negligible branching ratios to massive EW gauge boson pairs, \(WW^*\) and \(ZZ^*\). This prediction is disfavoured by the experimental measurements [1820] and therefore the equality between all couplings \(\kappa \) cannot hold. In the benchmark scenarios studied in this paper, each of the couplings \(\kappa _W\), \(\kappa _Z\), and \(\kappa _\gamma \) is assumed to be independent of all the other couplings. In the following, the UC scenario only refers to \(\kappa _{q} = \kappa _{g}\), without implying the equality for the other \(\kappa \) values.

The simplest QCD production processes, \(gg\rightarrow X\) and \(q\bar{q} \rightarrow X\) (where q refers to light quarks), yield different polarisations for the spin-2 particle X, and hence different angular distributions of its decay products. These mechanisms are considered in the model of a graviton-like tensor with minimal couplings proposed in Refs. [9, 10], which has been studied experimentally in Ref. [4]. The EFT Lagrangian, however, also allows for more complex processes with emission of one or more additional partons. For instance, processes with one-parton emission, like \(qg\rightarrow qX\) and \(~\bar{q}g\rightarrow \bar{q}X\), can produce a spin-2 state through either a qqX or a ggX vertex. When two partons are emitted, as in \(gg\rightarrow q\bar{q}X\) or \(q\bar{q}\rightarrow q\bar{q}X\), the spin-2 production may occur through qqX or ggX vertices, respectively, such that the polarisation of X is not uniquely determined by the initial state. Moreover, the EFT also allows for four-leg vertices like qqgX. These additional diagrams effectively change the polarisation of the particle X, compared to what is assumed by the model in Refs. [9, 10]. As a consequence, the angular distributions of the decay products become harder to separate from those expected for a scalar resonance.

The QCD production of a spin-2 particle is driven by the values of the couplings \(\kappa _g,~\kappa _q\). Presently, there are no experimental constraints on the ratio \(\kappa _q/\kappa _g\) from observed decay modes, since the separation of jets initiated by gluons or by light quarks is experimentally difficult and has not yet been attempted in Higgs boson studies. The ratio \(\kappa _q/\kappa _g\) can thus be regarded as a free parameter. When \(\kappa _q\ne \kappa _g\), the spin-2 model predicts an enhancement of the tail of the distribution of the transverse momentum, \(p_{\text {T}} ^X\), of the spin-2 particle. Such a high-\(p_{\text {T}} ^X\) tail is not present for the \(\kappa _q = \kappa _g\) (UC) case. As stated before, however, the EFTs are valid only up to some energy scale, \(\Lambda \). At higher energies, new physics phenomena are expected to enter to regularise the anomalous ultra-violet behaviour.

In the present analysis, a selection \(p_{\text {T}} ^X<300\) GeV is applied when investigating non-UC scenarios, \(\kappa _q\ne \kappa _g\). In addition, for the non-UC scenarios, analyses using a tighter selection \(p_{\text {T}} ^X<125~\text {GeV}\) are also performed. This is a conservative choice for the \(p_{\text {T}} ^{X}\) selection, as the EFT must describe the physics at least up to the mass of the observed resonance. It has been verified that the choice of the \(p_{\text {T}} ^{X}\) selection does not affect the results for the UC scenario. Even assuming the \(p_{\text {T}} ^X<300\) GeV selection, some choices of \(\kappa _q/\kappa _g\) produce high-\(p_{\text {T}} ^X\) tails incompatible with the observed differential distribution reported in Refs. [21, 22]. For this reason the investigated range of the \(\kappa _q/\kappa _g\) ratio is limited to between zero and two. The spin-2 scenarios considered in this study are presented in Table 2. The \(\kappa _{q} = \kappa _{g}\) model is referred to hereafter as the UC scenario. The \(\kappa _q=0\) case implies a negligible coupling to light quarks, whereas the \(\kappa _q=2\kappa _g\) case is an alternative scenario with an enhanced coupling to quarks.

Table 2 Choices of the couplings to quarks \(\kappa _{q}\) and to gluons \(\kappa _{g}\) studied for the spin-2 benchmark scenarios. The values of the selection criteria applied to the transverse momentum \(p_{\text {T}} ^X\) of the spin-2 resonance are also shown. For the UC scenario no \(p_{\text {T}} ^X\) selection is applied

4 Data and simulated samples

The data presented in this paper were recorded by the ATLAS detector during the 2012 LHC run with proton–proton collisions at a centre-of-mass energy of 8 TeV, and correspond to an integrated luminosity of 20.3 fb\(^{-1}\). For the \(H \rightarrow \gamma \gamma \) and \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) channels, the data collected in 2011 at a centre-of-mass energy of 7 TeV corresponding to an integrated luminosity of 4.5 fb\(^{-1}\), are also used. Data quality requirements are applied to reject events recorded when the relevant detector components were not operating correctly. More than 90 % of the recorded luminosity is used in these studies. The trigger requirements used to collect the data analysed in this paper are the same as those described in previous publications [1820]. They are only briefly recalled in the following sections.

The Monte Carlo (MC) samples for the backgrounds and for the SM Higgs boson signal are the same as those used for the analyses described in Refs. [1820], whereas new non-SM signal samples have been simulated. An overview of the signal samples is given in Sect. 4.1.

The effects of the underlying event and of additional minimum-bias interactions occurring in the same or neighbouring bunch crossings, referred to as pile-up in the following, are modelled with Pythia 8 [23]. The ATLAS detector response is simulated [24] using either Geant 4 [25] alone or combined with a parameterised Geant 4-based calorimeter simulation [26].

4.1 SM Higgs boson and BSM signal samples

The SM Higgs boson ggF production for all analyses is modelled using the Powheg-Box [27] generator at next-to-leading order (NLO), interfaced to Pythia 8 for parton showering and hadronisation and to simulate multi-parton interactions. To improve the modelling of the SM Higgs boson \(p_{\text {T}}\), a reweighting procedure is applied. This procedure applies a weight depending on the \(p_{\text {T}}\) of the Higgs boson to each event. The weights are chosen in order to reproduce the prediction of the next-to-next-to-leading-order (NNLO) and next-to-next-to-leading-logarithms (NNLL) dynamic-scale calculation given by the hres2.1 program [28, 29].

For the \(H \rightarrow \gamma \gamma \) analysis, the signal samples are generated at several values of the Higgs boson mass \(m_H\) around 125 GeV. The samples are used to obtain a parameterisation of the signal yields and of the invariant mass distribution of the two-photon system as continuous functions of \(m_H\) (both inclusively and for each category in the analysis, as described in Sect. 5.2). The spin-2 samples are generated using the MadGraph5_aMC@NLO [16] program with LO accuracy for zero, one, and two additional partons, and with subsequent matching of the matrix-element calculation with a model of the parton shower, underlying event and hadronisation, using Pythia 6 [30].

In the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) analysis the signal samples representing the production and decay of Higgs bosons with spin-0 and different parities are generated as follows. The SM Higgs boson production via gluon fusion at the mass \(m_H=125.5\) GeV is simulated using the Powheg-Box generator. For the non-SM signals, the decays of the generated Higgs bosons are simulated, according to the Higgs boson parity assumptions, using the JHU [9, 10] MC generator at leading order (LO). The spin-2 samples are generated using the MadGraph5_aMC@NLO MC generator, as for the \(H \rightarrow \gamma \gamma \) analysis.

For the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) analysis, the SM Higgs boson signal is generated at \(m_H = 125\) GeV using the Powheg-Box Monte Carlo generator. The spin-0 BSM signal samples are generated using MadGraph5_aMC@NLO. The signal samples representing the production and decay of Higgs bosons with spin-2 are generated using the MadGraph5_aMC@NLO MC generator, as for the \(H \rightarrow \gamma \gamma \) analysis.

For studies of the tensor structure of the HVV decay, all simulated signal samples are obtained by using the matrix element (ME) reweighting method applied, as explained in the following, to a sample generated with non-zero values of the BSM couplings. The reweighting procedure is validated against samples produced at different values of the couplings, to ensure that the distributions of the CP-sensitive final-state observables and of their correlations are reproduced correctly. For the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) analysis, the MC production is only performed for one set of tensor couplings: \(g_1=1\), \(g_2=1+i\), \(g_4= 1+i\). All other configurations of couplings are obtained by reweighting this sample at generator level. The ratios of the corresponding squares of ME values calculated at LO are used as weights. To calculate these ME values, the JHUGenME [10] program is used. In the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) analysis, only one MC sample is generated, using MadGraph5_aMC@NLO with parameters \(\kappa _\mathrm{SM} =1\), \(\kappa _{AWW} = 2\), \(\kappa _{HWW} = 2\), \(\cos (\alpha ) = 0.3\), and all other samples are obtained from it by reweighting the events on the basis of the ME amplitudes.

In all the analyses presented in this paper, the mass of the Higgs boson is fixed to 125.4 GeV  [12].

4.2 Background samples

The MC simulated samples for the backgrounds, as well as for the determinations of the corresponding cross sections, are the same as those adopted in Refs. [1820]. In the \(H \rightarrow \gamma \gamma \) analysis, the background is dominated by prompt \(\gamma \gamma \) events, with smaller contributions from \(\gamma -\)jet events. For the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) analysis, the major background is the non-resonant \(ZZ^{*}\) process, with minor contributions from the \(t\bar{t} \) and Z+jets processes. For the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) analysis, the dominant backgrounds are non-resonant W boson pair (WW) production, \(t\bar{t} \) and single-top-quark production, and the \(Z/\gamma ^{*}\) process followed by the decay to \(\tau \tau \) final states.

5 Tests of fixed spin and parity hypotheses

The \(H \rightarrow \gamma \gamma \) and \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) analyses are improved with respect to the previous ATLAS publication of Ref. [4]. These analyses are described in some detail in the following subsections. The spin and parity analysis in the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) channel has also been improved, as discussed in detail in a separate publication [8]. In the following, only a brief overview of this analysis is given. The expected and observed results of the individual channels and of their combination are presented in Sect. 5.5.

5.1 Statistical treatment

The analyses rely on discriminant observables chosen to be sensitive to the spin and parity of the signal.

Fig. 1
figure 1

Expected distributions of kinematic variables sensitive to the spin of the resonance considered in the \(H \rightarrow \gamma \gamma \) analysis, a transverse momentum of the \(\gamma \gamma \) system \(p_{\text {T}} ^{\gamma \gamma }\) and b the production angle of the two photons in the Collins–Soper frame \(|\cos \theta ^*|\), for a SM Higgs boson and for spin-2 particles with three different choices of the QCD couplings

A likelihood function, \(\mathcal{L}(\mathrm{data}~|~J^P, \mu , \vec {\theta })\), that depends on the spin-parity assumption of the signal is constructed as a product of conditional probabilities over binned distributions of the discriminant observables in each channel:

$$\begin{aligned} \mathcal {L}(\mathrm{data}~|~ J^P, \mu , \vec {\theta })= & {} \displaystyle \prod _{j}^{N_{\mathrm {chann.}}} \prod _{i}^{N_{\mathrm {bins}}} P\big (N_{i,j} ~| ~\mu _{j} \cdot S^{(J^P)}_{i, j}(\vec {\theta }) \nonumber \\&+ B_{i,j}(\vec {\theta }) \big ) \cdot \mathcal {A}_{j}(\vec {\theta }) \;, \end{aligned}$$
(4)

where \(\mu _{j}\) represents the parameter associated with the signal rate normalised to the SM prediction in each channel j.Footnote 3 The symbol \(\vec {\theta }\) represents all nuisance parameters. The likelihood function is a product of Poisson distributions P corresponding to the observation of \(N_{i,j}\) events in each bin i of the discriminant observables, given the expectations for the signal, \(S_{i,j}^{(J^P)}(\vec {\theta })\), and for the background, \(B_{i,j}(\vec {\theta })\). Some of the nuisance parameters are constrained by auxiliary measurements. Corresponding constraints are represented by the functions \(\mathcal {A}_{j}(\vec {\theta })\).

While the couplings are predicted for the SM Higgs boson, they are not known a priori for the alternative hypotheses, defined as \(J^{P}_{\mathrm {alt} }\), as discussed in Sect. 3. In order to be insensitive to assumptions on the couplings of the non-SM resonance (the alternative hypotheses) to SM particles, the numbers of signal events in each channel, for each different LHC centre-of-mass energy and for each tested hypothesis, are treated as independent parameters in the likelihood and fitted to the data when deriving results on the spin and parity hypotheses.

The test statistic \(\tilde{q}\) used to distinguish between the two spin-parity hypotheses is based on a ratio of profiled likelihoods [31, 32]:

$$\begin{aligned} \tilde{q} = \log \frac{\mathcal {L}\left( J^{P}_{\mathrm {SM} }, \hat{\hat{\mu }}_{J^{P}_{\mathrm {SM} }}, \hat{\hat{\theta }}_{J^{P}_{\mathrm {SM} }}\right) }{\mathcal {L}\left( J^{P}_{\mathrm {alt} }, \hat{\hat{\mu }}_{J^{P}_{\mathrm {alt} }}, \hat{\hat{\theta }}_{J^{P}_{\mathrm {alt} }} \right) }, \end{aligned}$$
(5)

where \(\mathcal {L}(J^P, \hat{\hat{\mu }}_{J^P}, \hat{\hat{\theta }}_{J^P})\) is the maximum-likelihood estimator, evaluated under either the SM \(J^{P}_{\mathrm {SM} }= 0^{+}\) or the alternative \(J^{P}_{\mathrm {alt} }\) spin-parity hypothesis. The parameters \(\hat{\hat{\mu }}_{J^P}\) and \(\hat{\hat{\theta }}_{J^P}\) represent the values of the signal strength and nuisance parameters fitted to the data under each spin and parity hypothesis. The distributions of the test statistic for both hypotheses are obtained using ensemble tests of MC pseudo-experiments. For each hypothesis test, about 70, 000 pseudo-experiments were generated. The generation of the pseudo-experiments uses the numbers of signal and background events in each channel obtained from maximum-likelihood fits to data. In the fits of each pseudo-experiment, these and all other nuisance parameters are profiled, i.e. fitted to the value that maximises the likelihood for each value of the parameter of interest. When generating the distributions of the test statistic for a given spin-parity hypothesis, the expectation values of the signal strengths are fixed to those obtained in the fit to the data under the same spin-parity assumption. The distributions of \(\tilde{q}\) are used to determine the corresponding p-values \(p(J^{P}_{\mathrm {SM} })=p^\mathrm{SM}\) and \(p(J^{P}_{\mathrm {alt} })=p^\mathrm{alt}\). For a tested hypothesis \(J^{P}_{\mathrm {alt} }\), the observed (expected) p-values are obtained by integrating the corresponding distributions of the test statistic above the observed value of \(\tilde{q}\) (above the median of the \(J^{P}_{\mathrm {SM} }\) \(\tilde{q}\) distribution). When the measured data are in agreement with the tested hypothesis, the observed value of \(\tilde{q}\) is distributed such that all p-values are equally probable.

Very small values of the integral of the distribution of the test statistic for the \(J^{P}_{\mathrm {alt} }\) hypothesis, corresponding to large values of \(\tilde{q}\), are interpreted as the data being in disagreement with the tested hypothesis in favour of the SM hypothesis.

Fig. 2
figure 2

Observed signal fraction per category for the \(H \rightarrow \gamma \gamma \) analysis, and comparison to expected values for a SM Higgs boson and for a spin-2 particle with different choices of QCD couplings. a The 11 categories described in the text are displayed, corresponding to the \(p_{\text {T}} ^{\gamma \gamma }<300~\text {GeV}\) selection; b the high-\(p_{\text {T}} ^{\gamma \gamma }\) category is discarded and the signal fractions are renormalised over the 10 remaining categories, corresponding to the \(p_{\text {T}} ^{\gamma \gamma }<125~\text {GeV}\) selection

The exclusion of the alternative \(J^{P}_{\mathrm {alt} }\) hypothesis in favour of the SM \(J^{P}_{\mathrm {SM} }\) hypothesis is evaluated in terms of the modified confidence level \(\mathrm{CL}_\mathrm{s}(J^{P}_{\mathrm {alt} })\), defined as [33]:

$$\begin{aligned} \mathrm{CL}_\mathrm{s}(J^{P}_{\mathrm {alt} }) = \frac{p (J^{P}_{\mathrm {alt} })}{ 1- p (J^{P}_{\mathrm {SM} })}\;. \end{aligned}$$
(6)

5.2 Spin analysis in the \(H \rightarrow \gamma \gamma \) channel

The analysis in the \(H \rightarrow \gamma \gamma \) channel is sensitive to a possible spin-2 state. Since the spin-2 models investigated in the present paper are different from those assumed in Ref. [4], the analysis has been redesigned, to improve its sensitivity to the new models.

The selection of \(H\rightarrow \gamma \gamma \) candidate events is based on the procedure of other recent ATLAS \(H \rightarrow \gamma \gamma \) analyses (see for example Ref. [20]). Events are selected if they satisfy a diphoton trigger criterion requiring loose photon identification, with transverse momentum \(p_{\text {T}}\) thresholds of 35 and 25 GeV for the photon with the highest (\(\gamma _1\)) and second-highest (\(\gamma _2\)) \(p_{\text {T}}\), respectively. During the offline selection two photons are further required to be in a fiducial pseudorapidity region, defined by \(|\eta ^\gamma |<2.37\), where the barrel/end-cap transition region \(1.37 < |\eta ^\gamma | < 1.56\) is excluded. The transverse momentum of the photons must satisfy \(p_{\text {T}} ^{\gamma _1}>0.35\cdot m_{\gamma \gamma }\) and \(p_{\text {T}} ^{\gamma _2}>0.25\cdot m_{\gamma \gamma }\), and only events with a diphoton invariant mass \(m_{\gamma \gamma }\) between 105 and 160 GeV are retained. For the events passing this selection, a further requirement is applied on the diphoton transverse momentum, \(p_{\text {T}} ^{\gamma \gamma }<300~\text {GeV}\), motivated by the assumed validity limit of the spin-2 EFT model, as explained in Sect. 3. After this selection, 17,220 events are left at a centre-of-mass energy \(\sqrt{s}=7~\text {TeV}\) and 94,540 events at \(\sqrt{s}=8~\text {TeV}\).

Kinematic variables sensitive to the spin of the resonance are the diphoton transverse momentum \(p_{\text {T}} ^{\gamma \gamma }\) and the production angle of the two photons, measured in the Collins–Soper frame [34]:

$$\begin{aligned} |\cos \theta ^*|\ = \frac{|\sinh (\Delta \eta ^{\gamma \gamma }) |}{\sqrt{1+ ( p_{\text {T}} ^{\gamma \gamma }/m_{\gamma \gamma })^2}}\frac{2p_{\text {T}} ^{\gamma _1}p_{\text {T}} ^{\gamma _2}}{m_{\gamma \gamma }^2}, \end{aligned}$$
(7)

where \(\Delta \eta ^{\gamma \gamma }\) is the separation in pseudorapidity of the two photons.

The predicted distributions of these variables, for events passing the selection, are shown in Fig. 1, for a SM Higgs boson and for a spin-2 particle with different QCD couplings. For the \(\kappa _q\ne \kappa _g\) cases, the enhanced high-\(p_{\text {T}} ^{\gamma \gamma }\) tail offers the best discrimination, whereas for \(\kappa _q=\kappa _g\) the most sensitive variable is \(|\cos \theta ^*|\).

To exploit the signal distribution in both \(p_{\text {T}} ^{\gamma \gamma }\) and \(|\cos \theta ^*|\), the selected events are divided into 11 mutually exclusive categories: 10 categories (labelled from C1 to C10) collect events with \(p_{\text {T}} ^{\gamma \gamma }<125~\text {GeV}\), divided into 10 bins of equal size in \(|\cos \theta ^*|\), while the 11th category (labelled C11) groups all events with \(p_{\text {T}} ^{\gamma \gamma }\ge 125~\text {GeV}\). As described in Sect. 3, for the non-UC spin-2 models the analysis is performed with two \(p_{\text {T}} ^{\gamma \gamma }\) selections, namely \(p_{\text {T}} ^{\gamma \gamma }<300~\text {GeV}\) and \(p_{\text {T}} ^{\gamma \gamma }<125~\text {GeV}\): the latter case corresponds to not using the 11th category.

The number of signal events above the continuum background can be estimated through a fit to the observed \(m_{\gamma \gamma }\) distribution in each category. The \(m_{\gamma \gamma }\) distribution is modelled in each category as the sum of one-dimensional probability density functions (pdf) for signal and background distributions:

$$\begin{aligned} f^{[c]}(m_{\gamma \gamma }|J)= & {} \frac{n_B^{[c]}f_B^{[c]}(m_{\gamma \gamma })+(n_J^{[c]}+n_\mathrm{bias}^{[c]})f_S^{[c]}(m_{\gamma \gamma })}{n_B^{[c]}+n_J^{[c]}+n_\mathrm{bias}^{[c]}},\nonumber \\ \end{aligned}$$
(8)

where J is the spin hypothesis, \(n_B^{[c]}\) and \(n_J^{[c]}\) are the background and the signal yield in category c, and \(f_B^{[c]}(m_{\gamma \gamma }),~f_S^{[c]}(m_{\gamma \gamma })\) are the \(m_{\gamma \gamma }\) pdfs for the background and the signal, respectively. The signal pdf \(f_S^{[c]}(m_{\gamma \gamma })\) is modelled as a weighted sum of a Crystal Ball function, describing the core and the lower mass tail, and of a Gaussian component that improves the description of the tail for higher mass values. For each category, \(f_S^{[c]}(m_{\gamma \gamma })\) is fitted to the simulated \(m_{\gamma \gamma }\) distribution of the SM Higgs boson and verified to be consistent also with the spin-2 models. The background pdf \(f_B^{[c]}(m_{\gamma \gamma })\) is empirically modelled as an exponential of a first- or second-degree polynomial. The choice of such a parameterisation can induce a bias (“spurious signal”) in the fitted signal yield, which is accounted for by the term \(n_\mathrm{bias}^{[c]}\). The size of the expected bias is determined as described in Refs. [20, 22], and ranges between 0.6 and 4 events, depending on the category (with the signal ranging from 15 to more than 100 events). In the statistical analysis, \(n_\mathrm{bias}^{[c]}\) is constrained for each category by multiplying the likelihood function by a Gaussian function centred at zero and with a width determined by the size of the expected bias.

Defining \(n_S\) as the total signal yield (summed over all categories), the expected fraction of signal events belonging to each category, \(\Phi _J^{[c]}\equiv \frac{n_J^{[c]}}{n_S}\), depends on the spin hypothesis J. The values of \(\Phi _J^{[c]}\) extracted from the data can be compared to their expected values for each spin hypothesis, as shown in Fig. 2 for the data collected at \(\sqrt{s}=8~\text {TeV}\).

For the non-UC scenario the 11th (high-\(p_{\text {T}} ^{\gamma \gamma }\)) category provides strong discrimination power against the non-SM hypothesis, as visible in Fig. 2a.

To discriminate between the SM spin-0 (\(J^{P}_{\mathrm {SM} }= 0^{+}\)) and alternative spin-2 hypotheses (\(J^{P}_{\mathrm {alt} }\)), two likelihood functions \(\mathcal{L}_{J^{P}_{\mathrm {SM} }},~\mathcal{L}_{J^{P}_{\mathrm {alt} }}\) are built, following the general approach described in Eq. (4):

$$\begin{aligned} -\ln \mathcal{L}_J= & {} \sum _{c} \left\{ \left( n_B^{[c]}+n_S\Phi _J^{[c]}+n_\mathrm{bias}^{[c]}\right) \right. \nonumber \\&- \sum _{e\in [c]} \ln ~\Big [ n_B^{[c]}f_B^{[c]}\left( m_{\gamma \gamma }^{(e)}\right) \nonumber \\&\left. +\left( n_S\Phi _J^{[c]}+n_\mathrm{bias}^{[c]}\right) f_S^{[c]}(m_{\gamma \gamma }^{(e)})\Big ] \right\} \end{aligned}$$
(9)

where \(\sum _{c}\) runs over all categories and \(\sum _{e\in [c]}\) runs over all events in category c. The total signal yield \(n_S\) is a free parameter in the likelihood model. The spin hypothesis being tested enters the likelihood function through the fractions of signal per category, \(\Phi _J^{[c]}\).

Several systematic uncertainties enter this model. They are implemented for each spin hypothesis as nuisance parameters, \(\theta _J\), constrained by multiplicative Gaussian terms in the likelihood function (not included in Eq. (9) for simplicity).

The signal fractions, \(\Phi _J^{[c]}\), for the SM Higgs boson are affected by uncertainties on the \(p_{\text {T}} \) spectrum of the resonance and on the size of the interference between the resonance and continuum production. The former is computed as described in Ref. [20]. The relative impact on the signal fractions is less than \(\pm 1\,\%\) for categories 1 to 8 (\(p_{\text {T}} ^{\gamma \gamma }<125~\text {GeV}\) and \(|\cos \theta ^*|<0.8\)), and becomes as large as \(\pm 13\,\%\) for categories 10 and 11. The correction for the interference is evaluated according to Refs. [35, 36]. The systematic uncertainty is conservatively assumed to equal the correction itself, and its relative impact ranges between \(\pm 0.1\,\%\) and \(\pm 1.8\,\%\).

No systematic uncertainty is assigned to the simulated \(p_{\text {T}} ^{X} \) distribution of the spin-2 models. The effect of the interference between the resonance and continuum production is essentially not known, as it depends on the width, \(\Gamma _X\), of the resonance, which is unknown. The results presented here only hold under the assumption of a narrow width for the resonance, such that interference effects can be neglected.

Additional systematic uncertainties come from the calibration of the photon energy scale and energy resolution and affect the signal parameterisation \(f_S^{[c]}\). These uncertainties are evaluated as described in Ref. [12].

5.3 Spin and parity analysis in the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) channel

The analysis of the spin and parity in the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) channel is described in detail in a separate publication [8]. In the following a brief summary is provided. The selection is restricted to events containing two charged leptons of different flavour (one electron and one muon). The \(e \nu \mu \nu \) channel is the most sensitive one [19]. The same-flavour channels (\(e \nu e \nu \) and \(\mu \nu \mu \nu \)) are not expected to add much in terms of sensitivity due to the presence of large backgrounds that cannot be removed without greatly reducing the acceptance of the alternative models considered in this analysis. The leading lepton is required to have \(p_{\text {T}} >22\) GeV and to match the object reconstructed by the trigger, while the sub-leading lepton needs to have \(p_{\text {T}} >15\) GeV. While the spin-0 analyses select only events with no jets in the final state (no observed jets with \(p_{\text {T}} >25\) GeV within \(|\eta | < 2.5\) or with \(p_{\text {T}} >30\) GeV within \({2.5 < |\eta | < 4.5}\)), the spin-2 analysis enlarges the acceptance by allowing for zero or one jet (selected according to the above mentioned criteria).

The major sources of background after the dilepton selection are \(Z/\gamma ^{*}\)+jets (Drell–Yan) events, diboson (\(WW, WZ/ \gamma ^*, ZZ/ \gamma ^{*}\)), top-quark (\(t\bar{t}\) and single top) production, and W bosons produced in association with hadronic jets (W+jets), where a jet is misidentified as a lepton. The contribution from misidentified leptons is significantly reduced by the requirement of two high-\(p_{\text {T}} \) isolated leptons. Drell–Yan events are suppressed through requirements on some of the dilepton variablesFootnote 4 (\(p_{\text {T}} ^{\ell \ell }\) \(> 20\) GeV, \(\Delta \phi _{\ell \ell }\) \(< 2.8\)), while a cut on \(m_{\ell \ell }\) (\(m_{\ell \ell }< 80\) GeV) targets the WW background. For alternative spin models with non-universal couplings, as discussed in Sect. 3, an additional upper bound is imposed on the Higgs boson \(p_{\text {T}} \), reconstructed as the transverse component of the vector sum of the momenta of the two charged leptons and the missing transverse momentum. Additionally, for events containing one jet, which include substantial top-quark and W+jets backgrounds, b-jet and \(Z\rightarrow \tau ^+\tau ^-\) vetoes are applied, together with transverse mass requirements: the larger of the transverse masses of the two W bosons (each computed using the corresponding lepton and the missing transverse momentum) in the event is required to be larger than 50 GeV, while the total transverse mass of the WW system (defined with the two leptons and the missing transverse momentum) is required to be below 150 GeV.

Control regions (CRs) are defined for the WW, top-quark and Drell–Yan backgrounds, which are the most important ones after the topological selection described above. The CRs are used to normalise the background event yields with a fit to the rates observed in data. The simulation is then used to transfer these normalisations to the signal region (SR). The \(W\)+jets background is estimated entirely from data, while non-WW diboson backgrounds are estimated using MC simulation and cross-checked in a validation region.

After the signal region selection, 4730 and 1569 candidate events are found in data in the 0-jet and 1-jet categories, respectively. For the latter category, the number decreases to 1567 and 1511 events when applying a selection on the Higgs boson \(p_{\text {T}} \) of less than 300 GeV and less than 125 GeV, respectively. In total 218 (77) events are expected from a SM Higgs boson signal in the 0-jet (1-jet) category, while about 4390 (1413) events are expected for the total background.

A BDT algorithm is used in both the fixed spin hypothesis tests and the tensor structure analyses. For spin-2 studies, the strategy follows the one adopted in Ref. [4], with the main difference being that the 1-jet channel has been added. Two BDT discriminants are trained to distinguish between the SM hypothesis and the background (BDT\(_0\)), and the alternative spin hypothesis and the background (BDT\(_2\)). Both BDTs employ the same variables, namely \(m_{\ell \ell }\), \(p_{\text {T}} ^{\ell \ell }\), \(\Delta \phi _{\ell \ell }\) and \(m_\mathrm{T}\), which provide the best discrimination between signal hypotheses and backgrounds, also in the presence of one jet in the final state. All background components are used in the trainings. In total, five BDT\(_2\) trainings are performed for the alternative spin hypotheses (one for the spin-2 UC scenario and two for each of the two spin-2 non-UC hypotheses corresponding to the different \(p_{\text {T}} ^{X}\) selections), plus one training of BDT\(_0\) for the SM Higgs boson hypothesis.

For the spin-0 fixed hypothesis test and HWW tensor structure studies, the first discriminant, BDT\(_0\), is the same as the one used for the spin-2 analysis, trained to disentangle the SM hypothesis from the background. A second BDT discriminant, \(\text{ BDT }_{bad hbox}\), is obtained by training the SM signal versus the alternative signal sample (the pure CP-even or CP-odd BSM hypotheses), and then applied to all CP-mixing fractions. No background component is involved in this case. The variables used for the \(\text{ BDT }_{bad hbox}\) trainings are \(m_{\ell \ell }\), \(\Delta \phi _{\ell \ell }\), \(p_{\text {T}} ^{\ell \ell }\)  and the missing transverse momentum for the CP-even analysis and \(m_{\ell \ell }\), \(\Delta \phi _{\ell \ell }\), \(E_{\ell \ell \nu \nu }\)  and \(\Delta p_\mathrm{T}\) for the CP-odd analysis. The training strategy is different from the one used in the spin-2 analysis because, while the spin-2 signal is very similar to the background, the spin-0 signals are all similar to each other, while being different from the main background components. Therefore, in the latter case, training the signal hypotheses against each other improves the sensitivity. The resulting BDT variable is afterwards used in binned likelihood fits to test the data for compatibility with the presence of a SM or BSM Higgs boson.

Several sources of systematic uncertainty are considered, both from experimental and theoretical sources, and are described in detail in Ref. [8]. The correlations induced among the different background sources by the presence of other processes in the control regions are fully taken into account in the statistical procedure. The most important systematic uncertainties are found to be those related to the modelling of the WW background, to the estimate of the W+jets background (originating from the data-driven method employed) and, for the spin-2 results in particular, to the \(Z\rightarrow \tau \tau \) modelling.

5.4 Spin and parity analysis in the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) channel

The reconstruction of physics objects and event selection used for the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) analysis is identical to the one presented in Ref. [12]. The main improvement with respect to the previous ATLAS publication of Ref. [4] is the introduction of a BDT discriminant designed to optimise the separation between the signal and the most relevant background process.

Events containing four reconstructed leptons (electrons or muons) in the final state are selected using single-lepton and dilepton triggers. The selected events are classified according to their final state: \(4\mu ,~2e2\mu ,~2\mu 2e\) and 4e, where for the decay modes \(2e2\mu \) and \(2\mu 2e\) the first pair is defined to be the one with the dilepton mass closest to the Z boson mass. Each muon (electron) must satisfy \(p_{\text {T}} > 6\) GeV (\(p_{\text {T}} > 7\) GeV) and be measured in the pseudorapidity range \(|\eta |<\) 2.7 (\(|\eta |<\) 2.47). Higgs boson candidates are formed by selecting two same-flavour, opposite-charge lepton pairs in an event. The lepton with the highest \(p_{\text {T}} \) in the quadruplet must have \(p_{\text {T}} >20\) GeV, and the leptons with the second- and third-highest \(p_{\text {T}} \) must have \(p_{\text {T}} >15\) GeV and \(p_{\text {T}} >\)10 GeV, respectively. The lepton pair with the mass closest to the Z boson mass is referred to as the leading lepton pair and its invariant mass as \(m_{12}\). The requirement \(50~\text {GeV}< m_{12} < 106~\text {GeV}\) is applied. The other lepton pair is chosen from the remaining leptons as the pair closest in mass to the Z boson. Its mass, denoted hereafter by \(m_{34}\), must satisfy \(12~\text {GeV}< m_{34} < 115 ~\text {GeV}\). Further requirements are made on the impact parameters of the leptons relative to the interaction vertex and their isolation in both the tracker and calorimeter.

The main background process affecting the selection of \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) events is the non-resonant production of \(ZZ^*\) pairs. This background has the same final state as the signal events and hereafter is referred to as the irreducible background. It is estimated from simulation and normalised to the expected SM cross section calculated at NLO [37, 38]. The reducible sources of background come from Z+jets and \(t\bar{t}\) processes, where additional leptons arise due to misidentified jets or heavy-flavour decays. The rate and composition of the reducible backgrounds are evaluated using data-driven techniques, separately for the two final states with sub-leading muons \( \ell \ell +\mu \mu \) and those with sub-leading electrons \(\ell \ell +ee\).

Only events with an invariant mass of the four-lepton system, denoted by \(m_{4\ell }\), satisfying the signal region definition 115 GeV \(<m_{4\ell }<\)130 GeV are selected. The expected signal and background yields in the signal region and the observed events in data are reported in Table 3.

Table 3 Expected signal, background and total yields, including their total uncertainties, and observed events in data, in the 115 GeV \(< m_{4\ell } <\) 130 GeV signal region. The number of expected signal events is given for a SM Higgs boson mass of 125.5 GeV
Fig. 3
figure 3

Definitions of the angular observables sensitive to the spin and parity of the resonance in the \(X \rightarrow ZZ^{*}\rightarrow 4\ell \) decay

The choice of production and decay angles used in this analysis is presented in Fig. 3, where the following definitions are used:

  • \(\theta _1\) and \(\theta _2\) are defined as the angles between final-state leptons with negative charge and the direction of flight of their respective Z bosons, in the four-lepton rest frame;

  • \(\Phi \) is the angle between the decay planes of two lepton pairs (matched to the two Z boson decays) expressed in the four-lepton rest frame;

  • \(\Phi _1\) is the angle between the decay plane of the leading lepton pair and a plane defined by the \(Z_1\) momentum (the Z boson associated with the leading lepton pair) in the four-lepton rest frame and the positive direction of the collision axis;

  • \(\theta ^{*}\) is the production angle of the \(Z_1\) defined in the four-lepton rest frame.

The final-state observables sensitive to the spin and parity of a boson decaying to \(ZZ^{*}\rightarrow 4\ell \) are the two production angles \(\theta ^{*}\) and \(\Phi _1\) and the three decay angles \(\Phi \), \(\theta _1\) and \(\theta _2\). In the case of a spin-0 boson, the differential production cross section does not depend on the production variables \(\cos (\theta ^{*})\) and \(\Phi _1\). It should be noted that, as the Higgs boson mass is below \(2m_Z\), the shapes of the mass distributions of the intermediate Z bosons, \(m_{12}\) and \(m_{34}\), are sensitive to the spin and parity of the resonance. In Fig. 4 the distributions of the final-state observables sensitive to the spin and parity of the decaying resonance are presented. The distributions are shown for the SM \(J^P=0^+\) and \(J^P=0^-\) simulated events, as well as for \(ZZ^{*}\) production and reducible backgrounds in the signal region \(115~\text {GeV}<m_{4\ell }< 130~\text {GeV}\). The events observed in data are superimposed on each plot.

Fig. 4
figure 4

Distributions of some of the final-state observables sensitive to the spin and parity of the resonance in the \(H\rightarrow ZZ^{*}\rightarrow 4\ell \) signal region \(115~\text {GeV}<m_{4\ell }< 130~\text {GeV}\) for data (points with errors), backgrounds (filled histograms) and predictions for two spin hypotheses (SM solid line and alternatives dashed lines). ac Invariant masses \(m_{12}\) , \(m_{34}\) and decay \(\cos \theta _1\), respectively; df \(\Phi \), \(\cos \theta ^*\) and \(\Phi _1\), respectively

Two approaches were pursued to develop the discriminants used to distinguish between different spin and parity hypotheses. The first uses the theoretical differential decay rate for the final-state observables sensitive to parity to construct a matrix-element-based likelihood ratio analysis (\(J^P\)-MELA). The second approach is based on a BDT.

For the \(J^P\)-MELA approach [3, 9], the probability of observing an event with given kinematics can be calculated. This probability is corrected for detector acceptance and analysis selection, which are obtained from the simulated signal MC samples. The full pdf also includes a term for incorrect pairing of the leptons in the \(4\mu \) and 4e channels. For a given pair of spin-parity hypotheses under test, the final discriminant is defined as the ratio of the pdf for a given hypothesis to the sum of the pdfs for both hypotheses.

For the BDT approach, a \(J^P\) discriminant is formed for each pair of spin-parity states to be tested, by training a BDT on the variables of simulated signal events which fall in the signal mass window \(115~\text {GeV}<m_{4\ell } <130~\text {GeV}\). For the \(0^+\) versus \(0^-\) test, only the parity-sensitive observables \(\Phi \), \(\theta _1\), \(\theta _ 2\), \(m_{12}\) and \(m_{34}\) are used in the BDT training. For the spin-2 test, the production angles \(\theta ^*\) and \(\Phi _1\) are also included.

Fig. 5
figure 5

The distributions of the discriminant BDT\(_{ZZ}\) versus the \(J^P\)-MELA discriminant for the SM \(J^P=0^+\) Higgs boson and for the backgrounds in the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) signal region \(115~\text {GeV}\ <m_{4\ell } <130~\text {GeV}\)

Both analyses are complemented with a BDT discriminant designed to separate the signal from the \(ZZ^*\) background. These discriminants are hereafter referred to as BDT\(_{ZZ}\). For the \(J^P\)-MELA analysis, the BDT\(_{ZZ}\) discriminant is fully equivalent to the one described in Refs. [12, 18]. For the BDT analysis the discriminating variables used for the background BDT\(_{ZZ}\) are the invariant mass, pseudorapidity, and transverse momentum of the four-lepton system, and a matrix-element-based kinematic discriminant \(K_D\) defined in Ref. [16]. The results from both methods are obtained from likelihood fits to the two-dimensional distributions of the background BDTs and of the spin- and parity-sensitive discriminants. In this way, the small correlation between these variables are taken into account in the analyses. The distribution of the background discriminant BDT\(_{ZZ}\) versus the \(J^P\)-MELA discriminant is presented in Fig. 5 for the SM \(J^P=0^+\) signal, the backgrounds, and the data. The projections of this distribution on the \(J^P\)-MELA and the BDT\(_{ZZ}\) variables, for different signal hypotheses, the backgrounds, and the data, are shown in Fig. 6. In this paper, only results based on the \(J^P\)-MELA approach are reported. The BDT approach was used as a cross-check and produced compatible results.

Fig. 6
figure 6

Distributions of the \(J^P\)-MELA and of the \(\mathrm{BDT}_{ZZ}\) discriminants in the \(H\rightarrow ZZ^{*}\rightarrow 4\ell \) signal region \(115~\text {GeV}<m_{4\ell }< 130~\text {GeV}\) for the data (points with errors), the backgrounds (filled histograms), and for predictions for several spin and parity hypotheses. The SM hypothesis is shown by the solid line while the alternative hypotheses are shown by the dashed lines. The signal distributions are normalised to the signal strength fitted in data. ac \(J^P\)-MELA discriminants for \(0^+\) SM vs \(0^-\), \(0^+\) SM vs \(0^+_h\) and \(0^+\) SM vs \(2^+\), respectively; df \(\mathrm{BDT}_{ZZ}\) discriminant for \(0^+\) SM vs \(0^-\), \(0^+\) SM vs \(0^+_h\) and \(0^+\) SM vs \(2^+\), respectively

Two general types of systematic effects impact the analyses using fixed spin and parity hypotheses: uncertainties on discriminant shapes due to experimental effects, and uncertainties on background normalisations from theory uncertainties and data-driven background estimates. The systematic uncertainties on the shape are included in the analysis by creating discriminant shapes corresponding to variations of one standard deviation in the associated sources of systematic uncertainty. The systematic uncertainties on the normalisation are included as additional nuisance parameters in the likelihood.

The list of sources of systematic uncertainty common to all ATLAS \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) analyses is presented in Ref. [18]. The relative impact of these sources on the final separation for all tested hypotheses is evaluated and sources affecting the final separation (given in Sect. 5.5) by less than \(\pm 0.5\,\%\) are neglected.

The main sources of systematic uncertainties are related to the experimental error on the Higgs boson mass, the modelling of the irreducible \(ZZ^{*}\) background, the uncertainty on the integrated luminosity and the experimental uncertainties on the electron and muon reconstruction. The uncertainty on the Higgs boson mass affects the final result since it impacts the shapes of the \(m_{12}\), \(m_{34}\), \(\cos \theta _{1}\) and \(\cos \theta _{2}\) variables. For the \(J^P\)-MELA method, the uncertainty on the estimate of the fraction of \(4\mu \) and 4e candidates with an incorrect pairing of leptons is also considered. This uncertainty is derived by comparing the corresponding prediction obtained from the Powheg and JHU MC generators for the SM hypothesis. A variation of \(\pm 10\,\%\) of the incorrect pairing fraction is applied to all spin and parity hypotheses.

Table 4 Relative impact of the main systematic uncertainties on the expected separation (expressed in terms of numbers of standard deviations) between the SM \(J^P=0^+\) and \(J^P=0^-\) hypotheses for the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) \(J^P\)-MELA analysis
Fig. 7
figure 7

Examples of distributions of the test statistic \(\tilde{q}\) defined in Sect. 5.1, for the combination of decay channels. a \(0^+\) versus \(0^-\); b \(0^+\) versus \(0^+ _h\); c \(0^+\) versus the spin-2 model with universal couplings (\(\kappa _{q} = \kappa _{g}\)); d \(0^+\) versus the spin-2 model with \(\kappa _{q} = 2 \kappa _{g}\) and the \(p_{\text {T}} \) selection at 125 GeV. The observed values are indicated by the vertical solid line and the expected medians by the dashed lines. The shaded areas correspond to the integrals of the expected distributions used to compute the p-values for the rejection of each hypothesis

The influence of the main systematic uncertainties on the separation between the SM \(J^P=0^+\) and \(J^P=0^-\) hypotheses for the \(J^P\)-MELA analysis is presented in Table 4. The total relative impact of all systematic uncertainties on the separation between the hypotheses (expressed in terms of numbers of standard deviations) is estimated to be about \(\pm 3\,\%\).

5.5 Individual and combined results

The distributions of discriminant variables in data agree with the SM predictions for all three channels, and exclusion ranges for alternative spin hypotheses are derived. Some examples of distributions of the test statistic \(\tilde{q}\) (defined in Sect. 5.1) used to derive the results are presented in Fig. 7. In this figure, the observed value is indicated by the vertical solid line and the expected medians by the dashed lines. The shaded areas correspond to the integrals of the expected distributions used to compute the p-values for the rejection of each hypothesis. The signal strengths per decay channel and per centre-of-mass energy are treated as independent parameters in each fit. Their values are compatible with the SM predictions.

The results obtained from the fit to the data, expressed in terms of p-values for different tested hypotheses and observed \(\mathrm{CL}_\mathrm{s}\) for the alternative hypotheses, are summarised in Tables 5 and 6. As shown in Table 5, the sensitivity to reject alternative hypotheses is driven by the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) and the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) channels. The \(H \rightarrow \gamma \gamma \) channel has sizeable sensitivity only to spin-2 models where the \(p_{\text {T}} ^{X}<125 \text {GeV}\) selection is not applied. In all cases the data prefer the SM hypothesis to the alternative models, with the exception of some of the spin-2 models for the \(H \rightarrow \gamma \gamma \) channel. In this case both hypotheses have similar observed p-values, but neither of the two is below 10 %.

As summarised in Table 6, the p-values of the combined results for the three channels show good agreement between the data and the SM hypothesis for all performed tests. All tested alternative hypotheses are rejected at a more than 99.9 % confidence level (CL) in favour of the SM hypothesis.

Table 5 Expected and observed p-values for different spin-parity hypotheses, for each of the three channels \(H \rightarrow \gamma \gamma \), \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \), and \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \). The observed \(\mathrm{CL}_\mathrm{s}\) for the alternative hypotheses are reported in the last column. The expected and observed p-values and the observed \(\mathrm{CL}_\mathrm{s}\) are defined in Sect. 5.5 and the alternative hypotheses are those described in Sect. 3
Table 6 Expected and observed p-values for different spin-parity hypotheses, for the combination of the three channels: \(H \rightarrow \gamma \gamma \) , \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) and \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \). The observed \(\mathrm{CL}_\mathrm{s}\) for the alternative hypothesis is reported in the last column. The expected and observed p-values and the observed \(\mathrm{CL}_\mathrm{s}\) are defined in Sect. 5.5. The definitions of alternative hypotheses are given in Sect. 3

6 Study of CP-mixing and of the HVV interaction tensor structure

Following the discussion in Sect. 3, measurements of the HVV interaction tensor couplings \(\kappa _\mathrm{SM}\), \(\kappa _{AVV}\), \(\kappa _{HVV}\) and of the mixing angle \(\alpha \) are performed. The measurements consist of fitting the ratios of couplings \((\tilde{\kappa } _{AVV}/\kappa _\mathrm{SM}) \cdot \tan \alpha \) and \(\tilde{\kappa } _{HVV}/\kappa _\mathrm{SM}\) to the discriminant observables for the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) and \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) processes and in their combination. In the fitting procedure only one ratio of couplings \((\tilde{\kappa } _{AVV}/\kappa _\mathrm{SM}) \cdot \tan \alpha \) or \(\tilde{\kappa } _{HVV}/\kappa _\mathrm{SM}\) is considered at a time, while the other one is assumed to be absent.

6.1 Statistical treatment

The measurement of the tensor structure of the HVV interaction is based on a profiled likelihood [31, 32] that contains the discriminant observables sensitive to the EFT couplings. The signal rates in the different channels and for different centre-of-mass energies are treated as independent parameters. Therefore, the global signal normalisation is not used to constrain the EFT couplings. The ratios of the BSM to SM couplings, \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\), are each separately fit to the discriminant observables in data. The test statistic used to derive the confidence intervals on the parameters of interest is \(q' = -2 \ln (\lambda )\), where \(\lambda \) is the profiled likelihood [31, 32]. The results presented in the following rely on the asymptotic approximation [31, 32] for the test statistic. This approximation was cross-checked with Monte Carlo ensemble tests that confirm its validity in the range of the parameters for which the 95 % CL limits are derived.

6.2 Tensor structure analyses in the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) channel

The \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) analysis used to study the spin-0 tensor structure is already described in Sect. 5.3 and detailed in Ref. [8]. Only the 0-jet category is considered and the BDT\(_0\) and \(\text{ BDT }_{bad hbox}\) are used as discriminant variables in the likelihood defined to measure the spin-0 tensor structure couplings. The only difference with respect to the spin hypothesis test is that, in this analysis, the BSM spin-0 couplings are treated as continuous variables in the test statistic.

6.3 Tensor structure analyses in the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) channel

To allow for a cross-check and validation of the obtained results, two different fitting methods based on the analytical calculation of the leading-order matrix element of the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) process are used.

The method of the matrix-element-observable fit is based on modelling the distributions of the final-state observables in each bin of coupling ratios using Monte Carlo simulation. Using the Lagrangian defined in Eq. (1), which is linear in the coupling constants \(\kappa _\mathrm{SM}\), \(\kappa _{HVV}\) and \(\kappa _{AVV}\), the differential cross section at each point in the phase space can be expressed as a term corresponding to the SM amplitude, plus two additional terms, linear and quadratic in the coupling constants. In this way it is possible to define two observables for each coupling, the so-called first- and second-order optimal observables, upon which the amplitude depends at each point of the phase space. For each event, they contain the full kinematic information about the couplings, which can thus be extracted from a fit to their shapes. More details of the method can be found in Refs. [3942].

The observables sensitive to the presence and structure of \(\kappa _\mathrm{SM}\), \(\kappa _{HVV}\) and \(\kappa _{AVV}\) considered in the current analysis are defined as follows:

$$\begin{aligned}&O_1(\kappa _{HVV}) = \frac{2\mathfrak {R}[\mathrm{ME}(\kappa _\mathrm{SM} \ne 0;\; \kappa _{HVV}, \kappa _{AVV} = 0;\; \alpha =0 )^* \cdot \mathrm{ME}( \kappa _{HVV} \ne 0;\; \kappa _\mathrm{SM}, \kappa _{AVV}=0;\; \alpha =0)] }{|\mathrm{ME}(\kappa _\mathrm{SM} \ne 0;\; \kappa _{HVV}, \kappa _{AVV} = 0;\; \alpha =0)|^2}, \nonumber \\&O_2(\kappa _{HVV}) = \frac{|\mathrm{ME}(\kappa _{HVV} \ne 0;\; \kappa _\mathrm{SM}, \kappa _{AVV}=0;\; \alpha =0 )|^2}{|\mathrm{ME}(\kappa _\mathrm{SM} \ne 0;\; \kappa _{HVV}, \kappa _{AVV} = 0;\; \alpha =0)|^2}, \nonumber \\&O_1(\kappa _{AVV},\alpha ) = \frac{2\mathfrak {R}[\mathrm{ME}(\kappa _\mathrm{SM} \ne 0;\; \kappa _{HVV}, \kappa _{AVV} = 0;\; \alpha =0 )^* \cdot \mathrm{ME}( \kappa _{AVV} \ne 0;\; \kappa _\mathrm{SM}, \kappa _{HVV}=0;\; \alpha =\pi /2)] }{|\mathrm{ME}(\kappa _\mathrm{SM} \ne 0;\; \kappa _{HVV}, \kappa _{AVV} = 0;\; \alpha =0)|^2}, \nonumber \\&O_2(\kappa _{AVV},\alpha ) =\frac{|\mathrm{ME}( \kappa _{AVV} \ne 0;\; \kappa _\mathrm{SM}, \kappa _{HVV}=0;\; \alpha =\pi /2)|^2}{|\mathrm{ME}(\kappa _\mathrm{SM} \ne 0;\; \kappa _{HVV}, \kappa _{AVV} = 0;\; \alpha =0)|^2}. \end{aligned}$$
(10)

Here \(\mathrm{ME}(\kappa _\mathrm{SM},\kappa _{HVV},\kappa _{AVV},\alpha )\) denotes the leading-order matrix element of the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) process. These definitions correspond to the first- and second-order optimal observables for a BSM amplitude with a three-component structure.

The observables \(O_{1,2}(\kappa _{HVV})\) and \(O_{1,2}(\kappa _{AVV},\alpha )\) are used for the \(\tilde{\kappa } _{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa } _{AVV}/\kappa _\mathrm{SM}) \cdot \tan \alpha \) individual fits respectively. In order to suppress the \(ZZ^{*}\) background, a kinematic BDT discriminant similar to those described in Sect. 5.4 is used as an additional observable in all fits. The BDT training is performed independently for each final state using observables with small sensitivity to parity: \(\eta _{4\ell }\), \(p_{\mathrm{T},4\ell }\), \(m_{4\ell }\), \(\cos (\theta ^*)\) and \(\Phi _1\). This BDT discriminant is denoted hereafter by \(\mathrm{BDT(}ZZ\mathrm {)} \).

To simplify their use in the analysis, all observables defined in Eq. (10) undergo a pdf transformation such that each observable becomes normally distributed in the Standard Model case. These transformed observables are referred to hereafter as \(TO_{1,2}(\kappa _{HVV})\) and \(TO_{1,2}(\kappa _{AVV},\alpha )\) respectively. The distributions of transformed observables for the Monte Carlo signal samples generated with \((\tilde{\kappa } _{HVV}/\kappa _\mathrm{SM} =0,\pm 1; \tilde{\kappa } _{AVV} =0)\) and \(((\tilde{\kappa } _{AVV}/\kappa _\mathrm{SM}) \cdot \tan \alpha =0,\pm 5; \tilde{\kappa } _{HVV} =0)\) are shown in Fig. 8.

Fig. 8
figure 8

Distributions of the observables used in the matrix-element-observable fit. a \(TO_1(\tilde{\kappa }_{HVV}) + TO_2(\tilde{\kappa }_{HVV})\), b \(TO_1(\tilde{\kappa }_{HVV}) - TO_2(\tilde{\kappa }_{HVV})\), for the Monte Carlo signal generated with \((\tilde{\kappa } _{HVV}/\kappa _\mathrm{SM} =0,\pm 1; \kappa _{AVV} =0)\). c \(TO_{1}(\tilde{\kappa }_{AVV} , \alpha )\), d \(TO_{2}(\tilde{\kappa }_{AVV} , \alpha )\) for the Monte Carlo signal generated with \(((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM}) \cdot \tan \alpha =0,\pm 5; \kappa _{HVV} =0)\). e \(\mathrm{BDT(}ZZ\mathrm {)} \) for the Monte Carlo signal generated with \((\tilde{\kappa } _{HVV}/\kappa _\mathrm{SM} =0,\pm 1; \kappa _{AVV} =0)\). The expected background contributions are shown as filled histograms on each plot

Table 7 Fitted values of \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\) and 95 % CL excluded regions obtained in \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) analysis. The expected values are estimated for the signal strength measured in data and assuming best-fit values for all other nuisance parameters. Only data collected at \(\sqrt{s}=8\) TeV are used. The symbol “n.a.” denotes the absence of 95 % CL sensitivity
Fig. 9
figure 9

Expected and observed distributions of the test statistic for fits of a \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and b \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\) for the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) analysis. The expected curves are calculated assuming the SM \(J^P=0 ^+\) signal and produced with the SM signal strength \(\mu =1\) and with the signal strengths fitted to data. The horizontal dotted black lines represent the levels of \(-2 \ln \lambda \) above which the values of coupling ratios under study are excluded above \(68\,\%\) and \(95\,\%\) CL, respectively

The contributions of all backgrounds considered in this analysis are also included. By construction the \(TO_2\) observables are sensitive to the modulus of the \(\tilde{\kappa } _{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa } _{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\) ratios: their distributions change with the strength of the respective coupling. These observables are insensitive to the relative sign of \(\tilde{\kappa } _{HVV}\) and \(\tilde{\kappa } _{AVV}\) with respect to \(\kappa _\mathrm{SM}\). The sign sensitivity comes from the \(TO_1\) observables, which are based on the interference terms: their distributions feature pronounced sign-dependent asymmetries. It was also found that the observables \(TO_1(\tilde{\kappa }_{HVV})\) and \(TO_2(\tilde{\kappa }_{HVV})\) are linearly correlated. To maximise the population of analysis histograms with currently available Monte Carlo event samples, it is desirable to reduce this correlation. This is achieved by considering the modified observables \(TO_1(\tilde{\kappa }_{HVV}) + TO_2(\tilde{\kappa }_{HVV})\) and \(TO_1(\tilde{\kappa }_{HVV}) - TO_2(\tilde{\kappa }_{HVV})\) in the current analysis.

The analysis is performed in several steps. First, multi-dimensional histograms of observables are created in 81 bins of \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\) for all fits. The predicted shapes of the observables for the signal are produced by reweighting the base Monte Carlo sample described in Sect. 4. The corresponding weights are derived using the analytical calculation of the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) matrix elements at leading order in perturbative QCD. The weights are calculated and applied at the Monte Carlo generator level. The observables used in the analysis are evaluated after detector simulation, accounting for the detector acceptance, resolution and reconstruction efficiency. The distributions of observables for backgrounds are estimated using Monte Carlo (for the irreducible background) and data-driven techniques (for the reducible backgrounds) described in Sect. 5 and Refs. [12, 18].

The distributions of observables are three-dimensional: \(TO_{1}(\tilde{\kappa }_{AVV}\), \(\alpha ), TO_{2}(\tilde{\kappa }_{AVV}, \alpha )\), \(\mathrm{BDT(}ZZ\mathrm {)}\) and \(TO_1(\tilde{\kappa }_{HVV}) + TO_2(\tilde{\kappa }_{HVV})\), \(TO_1(\tilde{\kappa }_{HVV}) - TO_2(\tilde{\kappa }_{HVV})\), \(\mathrm{BDT(}ZZ\mathrm {)}\) respectively. To obtain a reliable description for bins with an insufficient number of Monte Carlo events, the Kernel Density Estimation [43] smoothing procedure is applied to signal and background multi-dimensional histograms. In the smoothing procedure the smearing is done separately in four bins of \(\mathrm{BDT(}ZZ\mathrm {)}\), preserving the original normalisation.

Table 8 Expected and observed best-fit values of \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\) and \(95\%\) CL excluded regions obtained in the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) analysis. The expected values are estimated for the signal strength measured in data and assuming best-fit values for all other nuisance parameters. The data for \(\sqrt{s}=7\) TeV and \(\sqrt{s}=8\) TeV are combined
Fig. 10
figure 10

Expected distributions of the test statistic for the combination of \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) and \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) analyses as a function of BSM coupling ratios a \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and b \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\). The expected values are estimated for the signal strengths measured in data and assuming best-fit values for all other nuisance parameters. The 68 % and 95 % CL exclusion regions are indicated as lying above the corresponding horizontal lines. The individual distributions for \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) and \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) channels are shown

Fig. 11
figure 11

Expected and observed distributions of the test statistic for \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) and \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) analyses and their combinations. The distributions are shown as a function of the BSM coupling ratios \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\). The 68 % and 95 % CL exclusion regions are indicated as lying above the corresponding horizontal lines. a, b Individual \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) , \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) and combined observed distributions. c, d expected and observed combined distributions. The expected distributions are presented for the SM signal strength \(\mu =1\) and for the signal strengths obtained from the fit to data

The final pdfs used in the fits are obtained by applying linear histogram interpolation between the multi-dimensional bins of \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\). The individual likelihood functions per centre-of-mass energy (\(\sqrt{s}\)) and final state (FS) are:

$$\begin{aligned}&\mathcal {L}\left( \bar{\Omega } \Big | \frac{\tilde{\kappa }_{HVV}}{\kappa _\mathrm{SM}}, \frac{\tilde{\kappa }_{AVV}}{\kappa _\mathrm{SM}}\tan {\alpha },\bar{\theta } \right) \nonumber \\&\quad = \prod \limits _{i} P\left[ \bar{ \Omega }_{i} \Big | s_{i} \left( \frac{\tilde{\kappa }_{HVV}}{\kappa _\mathrm{SM}}, \frac{\tilde{\kappa }_{AVV}}{\kappa _\mathrm{SM}}\tan {\alpha } , \bar{\theta } \right) + b_{i}(\bar{\theta }) \right] , \end{aligned}$$
(11)

where P is the probability density function for the data vector \( \bar{\Omega }\), given the signal model s and background model b. The index i runs over all the bins of multi-dimensional histograms of observables and \(\bar{\theta }\) represents the vector of nuisance parameters corresponding to systematic uncertainties. Fits to data are performed by minimising the negative log-likelihood function with respect to the ratios of the couplings:

$$\begin{aligned}&L\left( \bar{\Omega } \Big | \frac{\tilde{\kappa }_{HVV}}{\kappa _\mathrm{SM}}, \frac{\tilde{\kappa }_{AVV}}{\kappa _\mathrm{SM}}\tan {\alpha }, \bar{\theta } \right) \nonumber \\&\quad = -2 \ln \prod \limits _{\sqrt{s}} \prod \limits _\mathrm{FS} \mathcal {L}\left( \bar{\Omega } \Big | \frac{\tilde{\kappa }_{HVV}}{\kappa _\mathrm{SM}}, \frac{\tilde{\kappa }_{AVV}}{\kappa _\mathrm{SM}}\tan {\alpha }, \bar{\theta } \right) . \end{aligned}$$
(12)

The test statistic \(q' = -2\ln (\lambda ) \) is defined as the profiled value of L of Eq. (12). To ensure the correctness of the statistical treatment and the absence of significant biases, a series of tests were performed before applying the fit to the data. Asimov datasets [31, 32] created from independently generated Monte Carlo samples with \(\tilde{\kappa }_{HVV} /{\kappa _\mathrm{SM}}\) and \((\tilde{\kappa } _{AVV}/\kappa _\mathrm{SM}) \cdot \tan {\alpha }\) equal to \(0, \pm 2, \pm 4, \pm 6, \pm 8 \) and \( \pm 10\) were injected into the analysis procedure. The tests were repeated for samples corresponding to 1 and 100 times the LHC Run-I integrated luminosity. In all cases the fitted values of coupling constants were found to be in agreement with the injected values within statistical uncertainties.

The results of the matrix-element-observable fit were validated and cross-checked using a nine-dimensional matrix-element method (9D fit). The method implements a multivariate per-event extended likelihood that is sensitive to both the \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\) mixing parameters and is based on nine experimental observables. The probability model is constructed with separate components for signal, the SM \(ZZ^*\) background and the reducible background. The background components are assumed to be independent of the Higgs boson tensor structure, so all of the sensitivity to mixing parameters comes from the signal component. Each component depends on nine experimental observables: \(m_{4\ell }\), \(p_{\mathrm{T},4\ell }\), \(\eta _{4\ell }\), \(\cos \theta ^{*}\), \(\cos \theta _{1}\), \(\cos \theta _{2}\), \(\Phi \), \(m_{12}\) and \(m_{34}\) (described in Sect. 5.4).

Table 9 Expected and observed best-fit values of (a) \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and (b) \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\) and \(95\,\%\) CL excluded regions obtained in the combination of \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) and \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) analyses. The expected values are estimated for the signal strengths measured in data and assuming best-fit values for all other nuisance parameters. The signal strengths are treated independently per decay channel and per collision energy

The main sources of systematic uncertainty for the tensor structure measurements are the same as discussed in Sect. 5 since they are based on the same four-lepton variables. Several additional sources of uncertainty, specific to each of the methods, are also taken into account. For the matrix-element-observable fit, the uncertainty related to the Kernel Density Estimation smoothing procedure applied to signal and background multi-dimensional histograms is considered. To estimate the influence of this uncertainty on the final result, a procedure similar to the one described in Sect. 5 is employed. The impact of the different sources of systematic uncertainty on the final results is evaluated by comparing the BSM exclusion limits obtained with a specific systematic uncertainty included or excluded in the fit, while excluding all other systematic uncertainties. A similar conclusion holds in the fixed hypothesis test: the systematic uncertainties have a very limited impact on the final result. The most important uncertainties are related to the estimates of the reducible backgrounds. The relative impact of these uncertainties on the final 95 % CL exclusion limit on BSM couplings was found to be around \(\pm 1\,\%\). The second most important group of sources of systematic uncertainty is related to the theoretical uncertainties on the production cross section of the \(ZZ^{*}\) background process. Their relative impact on the final result is found to be less than \(\pm 1\,\%\). The precision of the tensor structure analysis is thus dominated by the statistical errors.

In this paper, only results based on the matrix-element-observable approach are reported. The 9D approach was used as a cross-check and produced results compatible with the matrix-element approach.

6.4 Individual and combined results

The results of the tensor structure analyses performed in the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) channel are reported in Ref. [8] and, for completeness, they are also summarised in Table 7.

The distributions of the test statistic for fits of \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\) measured in the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) analysis are shown in Fig. 9.

The expected curves are calculated assuming the SM \(J^P=0 ^+\) signal, both with the SM signal strength, \(\mu =1\), and with the signal strength fitted to data, \(\hat{\mu }\). The fitted values of \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\), together with the intervals where these couplings are excluded at above the \(95\%\) CL, are reported in Table 8. The fitted values agree with the SM predictions within uncertainties.

The measurements from the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) and \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) channels are combined under the assumption that the BSM ratios of couplings \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\) are the same for the W and Z vector bosons. A common test statistic is obtained by combining the profiled likelihoods of the individual channels. The expected distributions of the likelihoods, for the signal strength values obtained from the fits to the data (\(\mu = \hat{\mu }\)), are presented in Fig. 10.

The observed distributions of profiled likelihoods for the combination of \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) and \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) measurements are presented in Fig. 11. The asymmetric shape of the expected and observed limits in the \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) results is mainly due to the interference between the BSM and the SM contributions that gives maximum deviation from the SM predictions for negative relative values of the BSM couplings.

Here the signal normalisations are treated as independent nuisance parameters of the different decay channels and the different centre-of-mass energies. The other nuisance parameters related to the experimental and theoretical uncertainties are treated as correlated when appropriate. The resulting 95 % CL exclusion regions for the combinations of \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) and \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) channels are listed in Table 9.

7 Conclusion

Studies of the spin and parity of the observed Higgs boson in the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \), \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) and \(H \rightarrow \gamma \gamma \) decay processes are presented. The investigations are based on 4.5 and \(20.3\;\mathrm{fb}^{-1}\) of pp collision data collected by the ATLAS experiment at the LHC at \(\sqrt{s}=7\) TeV and \(\sqrt{s}=8\) TeV, respectively. The SM Higgs boson hypothesis, corresponding to the quantum numbers \(J^{P}=0^{+}\), is tested against several alternative spin and parity models. The models considered include non-SM spin-0 and spin-2 models with universal and non-universal couplings to quarks and gluons. The combination of the three decay processes allows the exclusion of all considered non-SM spin hypotheses at a more than 99.9 % CL in favour of the SM spin-0 hypothesis.

The tensor structure of the HVV interaction in the spin-0 hypothesis is also investigated using the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) and \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) decays. Only one BSM tensor coupling is investigated at a time, while the other one is set to zero. The observed distributions of the variables sensitive the ratios of the BSM to SM tensor couplings, \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\), are compatible with the SM predictions.

Values of the BSM tensor couplings outside of the intervals \(-0.75< \tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}< 2.45\) and \(-2.85< (\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }< 0.95\) are excluded at the 95 % CL for the \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) process. For the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) process the ranges \(-2.2< \tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}< -1.0\) and \(-0.85< \tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}< 0.4\) and \(-6.0< (\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }< 5.0\) are excluded at the 95 % CL.

The results from the \(H \rightarrow WW^{*} \rightarrow e \nu \mu \nu \) and \(H \rightarrow ZZ^{*} \rightarrow 4 \ell \) decay channels are combined under the assumption that the \(\tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}\) and \((\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }\) couplings have the same values for the HWW and HZZ processes. As a result of this combination, the regions outside of \(-0.73< \tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}< 0.63\) and \(-2.18< (\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }< 0.83\) intervals are excluded at the 95 % CL. The corresponding expected not-excluded intervals at the 95 % CL, assuming the SM Higgs boson hypothesis and the signal strength values measured in data, are \(-0.55< \tilde{\kappa }_{HVV}/\kappa _\mathrm{SM}< 4.80\) and \(-2.33< (\tilde{\kappa }_{AVV}/\kappa _\mathrm{SM})\cdot \tan {\alpha }< 2.30\).