Introduction

Most magnetic materials order with simple magnetic structures in which spins are collinear or coplanar. Noncoplanar magnetic structures are relatively rare, but are of great current interest, because they can exhibit topological character and exotic physical properties1,2. For example, the finite scalar spin chirality of noncoplanar spin textures can generate a topological magneto-optical effect3 and anomalous quantum Hall effect4,5, even in the absence of spin-orbit coupling. Topologically-nontrivial spin textures are typically multi-q structures, which superpose magnetic modulations with symmetry-related wavevectors q2. Multi-q spin textures with long-wavelength modulations, such as skyrmion and hedgehog crystals, are well-studied as hosts of topology-driven phenomena6,7,8. In this context, multi-q antiferromagnets are increasingly important9, because they offer higher densities of topological objects with the potential to generate stronger physical responses10.

To probe the relationships between spin structure, interactions, topology, and physical response, it is crucial to identify real materials that host noncoplanar spin textures. This has proved a challenging task, for three main reasons. First, it is necessary to identify noncoplanar spin textures that are robust to subleading effects such as magnetic anisotropies, spin-lattice coupling11,12, fluctuations13,14,15,16, and anisotropic interactions17, which usually favor collinear states. Second, most noncoplanar states are found in metals, such as USb18,19 and γ-Mn alloys20,21,22,23,24,25, and are often stable only under an applied magnetic field6,26. On the one hand, itinerant electrons can support the generation of physical responses; on the other hand, modeling the magnetic interactions of metals presents fundamental challenges27,28,29,30,31,32, such that insulators are often more suitable as model materials. Third, neutron-diffraction measurements play a central role in solving magnetic structures, but suffer from a “multi-q problem”: Such measurements are generally unable to distinguish 1-q from multi-q structures33, due to orientational averaging for powders or domain averaging for single crystals. Therefore, multi-q spin textures are challenging to stabilize in models, and to identify in real materials.

Here, we identify the Mott-insulating double perovskites Ba2YRuO6 and Ba2LuRuO634,35,36,37 as prototypical examples of noncoplanar 3-q magnetism on the face-centered cubic (FCC) lattice in zero magnetic field. We obtain evidence for 3-q magnetism from a spin-wave analysis of neutron spectroscopy data. By optimizing the magnetic structure and interactions simultaneously against our data, we show that the 3-q structure is stabilized by biquadratic interactions within an antiferromagnetic Heisenberg-Kitaev model. Our study experimentally establishes that noncoplanar multi-q states are stabilized in frustrated FCC antiferromagnets, identifies cubic double perovskites as model materials to realize this behavior, and identifies guiding principles to facilitate design of materials with noncoplanar magnetic states.

Results

Theory and materials selection

Our study is motivated by theoretical results for the FCC antiferromagnet13,38,39,40,41. The nearest-neighbor Heisenberg-Kitaev spin Hamiltonian on the FCC lattice can be written as

$$H=J\sum\limits_{\left\langle i,j\right\rangle }{{{{\bf{S}}}}}_{i}\cdot {{{{\bf{S}}}}}_{j}+K\sum\limits_{{\left\langle i,j\right\rangle }_{\gamma }}{S}_{i}^{\gamma }{S}_{j}^{\gamma },$$
(1)

where Si is a Ru5+ spin with quantum number S = 3/2, J and K denote the Heisenberg and Kitaev interactions, respectively39, and \(\gamma \in \left\{x,y,z\right\}\) is perpendicular to the cubic face containing the bond between neighbors \(\langle i,j\rangle\). For antiferromagnetic J > 0 only, the model is frustrated, and orderings with q ∈ [1, q, 0] are degenerate13,40,41. The degenerate manifold includes q = [1, 0, 0] (“Type I”) ordering, which is favored by fluctuations13,14,42 and is observed in Ba2YRuO6 and Ba2LuRuO634. Henceforth, we therefore restrict our discussion to q = [1, 0, 0] ordering. For a collinear structure, spins may be either parallel or perpendicular to q; the former is stabilized by K < 0 and the latter by K > 039,40,41.

Figure 1a shows the collinear (1-q) and noncollinear (multi-q) structures associated with Type I antiferromagnetism. An unusual property of the FCC lattice is that 1-q, 2-q, and 3-q structures are energetically degenerate for all bilinear interactions that stabilize Type I ordering40,41. Moreover, uniaxial anisotropy (~ \({S}_{z}^{2}\)) and antisymmetric exchange terms are forbidden by \(Fm\bar{3}m\) symmetries, and quartic anisotropy (~ \({S}_{x}^{4}+{S}_{y}^{4}+{S}_{z}^{4}\)) is forbidden for S = 3/2 operators. Consequently, interactions that would usually favor collinear magnetic structures are inactive in the S = 3/2 FCC antiferromagnet. This remarkable property potentially allows noncollinear structures to appear.

Fig. 1: Candidate magnetic structures and powder neutron diffraction patterns.
figure 1

a Symmetry-allowed magnetic structures with propagation vector q = [1, 0, 0] on the FCC lattice for Ba2MRuO6 (space group \(Fm\bar{3}m\); a = 8.29 and 8.24 Å for M = Y and Lu, respectively). The 1-q, 2-q, and 3-q structures are shown for the \(m{{{{\rm{X}}}}}_{3}^{+}\) irrep (left) and the \(m{{{{\rm{X}}}}}_{5}^{+}\) irrep (right). Spins along different directions are colored differently; note that 1-q, 2-q, and 3-q structures have [100], \(\left\langle 110\right\rangle\), and \(\left\langle 111\right\rangle\) spin directions, respectively. In ref. 52, these structures (left to right) are labeled E1a, E2d, E3c, E1b, E2a, E3a. b Elastic scattering data (− 1.3 ≤ E ≤ 1.3 meV) measured at T = 5 K with Ei = 11.8 meV for Ba2YRuO6 and Ba2LuRuO6 (black circles), Rietveld refinements (red lines), and data–fit (blue lines). Tick marks show (top to bottom): nuclear, impurity M2O3, and magnetic phases. Note that the Rietveld refinements depend only on the irrep and are identical for 1-q, 2-q, and 3-q structures of the same irrep.

To identify candidate systems for 3-q spin textures among the diverse magnetic ground states of double perovskites43,44,45,46,47,48,49,50,51, we consider two criteria: Type I antiferromagnetic ordering, and strictly cubic symmetry below the magnetic ordering temperature, TN. The second criterion is key because 3-q structures have cubic symmetry, while 1-q and 2-q structures have tetragonal or orthorhombic symmetry that could drive a crystallographic distortion via spin-lattice coupling [Fig. 1a]. We investigate Ba2YRuO6 and Ba2LuRuO6 because they are chemically well-ordered and show no evidence for low-temperature deviations from cubic symmetry34,36. Moreover, recent first-principles calculations predict that their magnetic structures might not be collinear52, in apparent contradiction with interpretations of previous experiments34.

Experiments

We prepared ~ 8 g polycrystalline samples of Ba2YRuO6 and Ba2LuRuO6 by solid-state reaction (see Methods). The magnetic ordering temperature TN ≈ 37 K is the same for both samples, and is suppressed compared to the Weiss temperature θ ~ −500 K, indicating strong magnetic frustration36. Rietveld refinements to X-ray diffraction data confirm cubic symmetry above and below TN (see Methods and Supplementary Fig. 1). The level of Y/Ru site mixing was undetectable (<3%) and Lu/Ru site mixing was 6(2)%, with minor impurity phases of ≈ 1 wt.% and ≈ 4 wt.% for M = Y and Lu, respectively (see Supplementary Table 1). We performed inelastic neutron-scattering measurements on the SEQUOIA instrument at ORNL53 using incident neutron energies Ei = 62 and 11.8 meV, yielding elastic energy resolutions δins ≈ 1.68 and 0.27 meV, respectively.

Rietveld refinements to elastic magnetic scattering

Figure 1b shows magnetic Rietveld refinements to our elastic neutron-scattering data measured at T ≈ 5 K. Applying the q = [1, 0, 0] propagation vector to \(Fm\bar{3}m\) crystal symmetry generates two magnetic irreducible representations (irreps), notated \(m{{{{\rm{X}}}}}_{3}^{+}\) and \(m{{{{\rm{X}}}}}_{5}^{+}\)54,55,56,57. These irreps can be distinguished by their magnetic Bragg profiles. The \(m{{{{\rm{X}}}}}_{5}^{+}\) irrep agrees well with our elastic-scattering data for both materials; Rietveld refinements yield ordered magnetic moment lengths of 2.56(2) and 2.43(2) μB per Ru for Ba2YRuO6 and Ba2LuRuO6, respectively. Since the Ru5+ magnetic form factor has not been accurately determined, we tested several 4d magnetic form factors58; while this choice does not qualitatively affect our results, the form factor for Zr+ (isoelectronic with Ru5+) yields optimal agreement with our data and is used throughout. The Zr+ form factor also yielded better agreement with our data than the interpolated Ru5+ form factor used in ref. 59. In contrast to the \(m{{{{\rm{X}}}}}_{5}^{+}\) irrep, the \(m{{{{\rm{X}}}}}_{3}^{+}\) irrep strongly disagrees with our data, as it yields zero intensity for the strong (100) magnetic Bragg peak. This can be understood intuitively for a collinear 1-q structure, because neutrons are only sensitive to spin components perpendicular to the scattering wavevector, and the \(m{{{{\rm{X}}}}}_{3}^{+}\) irrep has Sq while the \(m{{{{\rm{X}}}}}_{5}^{+}\) irrep has S ⊥ q [Fig. 1a]. A previous neutron-diffraction study of Ba2YRuO6 and Ba2LuRuO6 considered only collinear 1-q structures34, but could not rule out multi-q structures, due to the multi-q problem.

Overview of inelastic magnetic scattering

To overcome the multi-q problem, we consider the energy dependence of neutron-scattering data60. To explain this choice of approach, Fig. 2 shows calculated spin-wave spectra for the 1-q, 2-q and 3-q structures with the \(m{{{{\rm{X}}}}}_{5}^{+}\) irrep. Here, only antiferromagnetic Heisenberg interactions are included. Qualitative differences between the calculated powder-averaged spectra for 1-q, 2-q and 3-q structures are apparent; e.g., the 3-q calculation shows a cutoff of magnetic signal for energies above ~ 0.4JS, whereas the magnetic signal for the the 1-q calculation extends to ~ 0.7JS. These calculations motivate the use of powder-averaged inelastic neutron scattering data to distinguish between these candidate ground states, and will inform our discussion of the experimental spectra.

Fig. 2: Calculated powder-averaged spin-wave spectra for candidate magnetic structures.
figure 2

Calculated spectra are shown for 1-q, 2-q, and 3-q structures (left to right) with the \(m{{{{\rm{X}}}}}_{5}^{+}\) irrep. Only antiferromagnetic Heisenberg J is included, and all other interactions are zero.

Figure 3a shows our experimental inelastic data measured with Ei = 62 meV at T ≈ 5 K. A structured inelastic signal appears at T < TN for small scattering wavevectors, Q ≲ 2 Å−1, which we identify as magnon scattering. The magnon scattering is very similar for M = Y and Lu, except the features for M = Lu appear slightly broader, which may reflect the slightly greater degree of M/Ru site mixing in this compound (see Supplementary Fig. 1 and Supplementary Table 1). The top of the magnetic band overlaps with an intense phonon signal for Q ≳ 2 Å−1. Figure 3b shows the scattering intensity integrated over 4.0 ≤ Q ≤ 4.5 Å−1, from which we extract the average energy Eph and width σph of this phonon band via Gaussian fits for each material. The energy overlap of magnon and phonon modes suggests that spin-lattice coupling may be significant, which we consider below.

Fig. 3: Broadband inelastic neutron-scattering data (Ei = 62 meV) measured at T = 5 K.
figure 3

Upper and lower panels show data for Ba2YRuO6 and Ba2LuRuO6, respectively. a Intensity as a color plot, and (b) energy dependence integrated over 4.0 ≤ Q ≤ 4.5 Å−1, where experimental data are shown as black circles, and Gaussian fits to the ~14 meV phonon band as red lines. Error bars in experimental data indicate 1 s.d.

Optimization of magnetic structure and Hamiltonian

Our starting point for modeling the magnetic scattering is the nearest-neighbor Heisenberg-Kitaev model, Eq. (1). Antiferromagnetic J > 0 has been shown to be important in Ba2YRuO637, and we include K > 0 because it is needed to stabilize magnetic ordering with the \(m{{{{\rm{X}}}}}_{5}^{+}\) irrep, as observed experimentally34. We consider three additional interactions. First, the symmetric off-diagonal interaction \({H}_{\Gamma }=\Gamma {\sum }_{{\left\langle i,j\right\rangle }_{\gamma }}({S}_{i}^{\alpha }{S}_{j}^{\beta }+{S}_{i}^{\beta }{S}_{j}^{\alpha })\) is the only additional bilinear nearest-neighbor interaction allowed by symmetry39. Second, the Heisenberg next-nearest neighbor interaction \({H}_{2}={J}_{2}{\sum }_{\left\langle \left\langle i,j\right\rangle \right\rangle }{{{{\bf{S}}}}}_{i}\cdot {{{{\bf{S}}}}}_{j}\) has been invoked for Ba2YRuO637; we require J2 ≤ 0 to stabilize Type I ordering. Third, the nearest-neighbor biquadratic coupling \({H}_{{{{\rm{bq}}}}}={J}_{{{{\rm{bq}}}}}{\sum }_{\left\langle i,j\right\rangle }{({{{{\bf{S}}}}}_{i}\cdot {{{{\bf{S}}}}}_{j})}^{2}\) has been invoked in density-functional-theory calculations for 4d double perovskites due to their increased electron hopping relative to 3d analogs52. While the 4-spin exchange enters into the Hamiltonian at the same order as biquadratic exchange, these terms can be combined for the FCC lattice52, so we do not consider them separately. For Jbq = 0, the classical energy of 1-q, 2-q, and 3-q structures is equal for all K, Γ, and J2 that stabilize Type I ordering. Nonzero Jbq removes this degeneracy and stabilizes 1-q ordering for Jbq < 0 and 3-q ordering for Jbq > 0 [Fig. 4a]. Importantly, since single-ion anisotropies are forbidden for S = 3/2 in a cubic environment, biquadratic exchange is the only physically-plausible mechanism that can remove the degeneracy of 1-q and 3-q structures.

Fig. 4: Phase diagram and goodness-of-fit for candidate magnetic structures.
figure 4

a Schematic phase diagram showing the magnetic ground states of the J-K-Jbq model. b Goodness-of-fit metric Rwp for candidate magnetic structures and interaction models of Ba2YRuO6 (upper graph) and Ba2LuRuO6 (lower graph). The graphs show Rwp for refinements of the Heisenberg-Kitaev (J-K) model including a third refined parameter Γ (red bars), J2 (blue bars), or Jbq (green bars); note that the 2-q structure is stable only for Jbq = 0.

We performed extensive fits to our inelastic neutron-scattering data to optimize the magnetic interactions for each candidate magnetic structure. For each structure associated with the \(m{{{{\rm{X}}}}}_{5}^{+}\) irrep (1-q, 2-q, or 3-q), we optimized three spin Hamiltonian parameters (J, K, and either Γ, J2, or Jbq) against the broadband inelastic data shown in Fig. 5a and the energy dependence near the (100) magnetic Bragg position shown in Fig. 5b. The broadband inelastic data provides an overview of the key features of the spectrum, while the low-energy data provides information about the magnon gap, which is an important constraint as we discuss further below. The powder-averaged magnon spectrum was calculated within the established approach of linear spin-wave theory61,62 using the SpinW program63, updated to include Jbq64. We included a renormalization factor that takes into account higher-order corrections in the 1/S expansion, which is necessary to extract a correct value of Jbq since the unrenormalized theory would yield a value of Jbq that is 2.25 times too small65. The parameter values were optimized to minimize the sum of squared residuals using nonlinear least-squares refinement; our general approach is similar to66,67,68. For each refined model, several refinements were performed with different initial parameter values, to check for alternative χ2 minima, and derivative-based and derivative-free optimizers were used (see Methods). An overall intensity scale factor was optimized in our refinements; we also optimized a linear-in-E intensity offset to account for phonon scattering. To account for magnon-phonon coupling, we calculated the energy-dependent broadening of the magnon spectrum as \(\delta (E)={\delta }_{{{{\rm{ins}}}}}(E)+A{e}^{-{(E-{E}_{{{{\rm{ph}}}}})}^{2}/2{\delta }_{{{{\rm{ph}}}}}^{2}}\), where δ(E) is the overall Gaussian energy width, δins(E) is the instrumental resolution, and A is a refined parameter that phenomenologically accounts for magnon broadening due to coupling with phonons at E ~ Eph.

Fig. 5: Inelastic neutron-scattering data and model fits.
figure 5

a Broadband inelastic neutron-scattering data (Ei = 62 meV) and spin-wave fits for different magnetic structures, showing (left to right) experimental data, 1-q fit, 2-q fit, and 3-q fit. Optimal fits are shown for each structure and correspond to the following models: J-K-Jbq (3-q structures and 1-q structure for M = Y) and J-K-J2 (2-q structures and 1-q structure for M = Lu). b Low-energy inelastic neutron-scattering data (Ei = 11.8 meV) and 3-q model calculations, showing (left to right) a cut at Q = 0.7450 ± 0.0175 Å−1 comparing experimental data (black circles), spin-wave fit (solid red lines) and fit to a sigmoid function defined in the text (dotted green lines); experimental data as a Q-E slice; and spin-wave calculation. Note that the maximum energy transfer is restricted to ≈ 6 meV at Q = 0.745 Å−1 by kinematic constraints. Error bars in experimental data indicate 1 s.d.

Figure 5a compares our broadband inelastic data (Ei = 62 meV) with the best fit for each of the 1-q, 2-q, and 3-q structures. The data show two V-shaped features centered near the (100) and (120) magnetic Bragg peaks at ≈ 0.76 and ≈ 1.70 Å−1, respectively, with a sharp cutoff of magnetic signal for energies above ~ 14 meV. For both materials, these characteristics are best reproduced by the 3-q structure, while the 1-q structure disagrees with our experimental data. These observations are confirmed by the goodness-of-fit metric Rwp [Fig. 4b]. For both materials and for every interaction model we considered, the 3-q structure yields better agreement with our data than the 1-q or 2-q structure. Notably, the goodness-of-fit is more sensitive to the structure than the precise magnetic interactions; indeed, the qualitative differences between 1-q and 3-q spectra are more apparent for Heisenberg exchange only [Fig. 2]. The global best fit is for the 3-q structure and J, K, and Jbq interactions with the refined values given in Table 1. Refined values of A indicate magnon broadening due to magnon-phonon coupling is larger for Ba2LuRuO6. Importantly, for both materials, the biquadratic term is significant, with Jbq/J ~ 0.06. Hence, our key results are that only the 3-q spin texture agrees well with our neutron data, and this state is stabilized by biquadratic interactions. Similar results for both materials suggest that this state is insensitive to a low level of chemical disorder, as is observed in Ba2LuRuO6 only (see Supplementary Fig. 1 and Supplementary Table 1).

Table 1 Refined values of magnetic interaction parameters for the J-K-Jbq model and 3-q structure

Origin of magnon gap

Our model provides insight into the mechanism of gap opening35. Low-energy inelastic neutron scattering measurements (Ei = 11.8 meV) shown in Fig. 5b reveal gaps of 2.59(6) meV and 2.49(11) meV for M = Y and Lu, respectively, estimated by fits to a sigmoid function, \(I(E)\propto 1/[1+{e}^{-{\sigma }_{g}(E-{E}_{g})}]\), where Eg and σg are the fitted gap energy and width, respectively. Figure 5b also compares our low-energy inelastic data with the 3-q magnon spectrum for the optimal J-K-Jbq model [Table 1]. This calculation reproduces the observed gap. In contrast, the J-K-J2 model that yields the next-best Rwp does not yield a gap for any of the candidate magnetic structures, and can therefore be discounted (see Supplementary Fig. 2). Since single-ion anisotropies are forbidden here, the mechanism of gap opening is subtle. A fully isotropic model (J > 0 only) possesses gapless Goldstone modes and accidental zero-energy modes42,69. Both types of excitation must become gapped to explain the observed gap in neutron spectra, and whether this occurs depends on both structure and interactions. For a 1-q structure with Sq (stable for Jbq < 0 and K < 0), the spectrum is fully gapped70; however, this magnetic structure was ruled out by elastic neutron data [Fig. 1]. The 1-q structure with Sq (stable for Jbq < 0 and K > 0) is consistent with elastic neutron data, but its spectrum remains gapless for nonzero K because of the continuous rotational symmetry in the spin plane. By contrast, if the structure is 3-q, a gap is present at the linear spin-wave level if both Jbq > 0 and K > 0, since these interactions together favor \(\left\langle 111\right\rangle\) spin alignment. We conclude that it is difficult to explain the elastic neutron data and the observed gap with a 1-q structure. However, these observations can be explained by a 3-q model with Jbq > 0 and K > 0, which also yields optimal agreement with our high-energy inelastic neutron-scattering data.

Continuous symmetries and their associated gapless Goldstone modes are maintained when considering quantum effects beyond linear spin-wave theory, which may have a similar effect to negative Jbq by favoring collinear structures71 and gapping the accidental zero-energy modes42,70. We note that the energy scale of quantum order-by-disorder is relatively small, on the order of 10−3J42, and can be easily surpassed by the expected biquadratic exchange interaction in these materials. Ref. 52 suggests that the combined Jbq and 4-spin interaction is on the order of 0.1J for Ba2YRuO6, in reasonable agreement with our own fits. Importantly, this Jbq value results in an energy difference between 1-q and 3-q orderings that is much larger than any energy difference induced by order by disorder.

Discussion

We have identified Ba2YRuO6 and Ba2LuRuO6 as model materials that host noncoplanar 3-q structures stabilized by biquadratic interactions in zero applied field. Macroscopic topological physical responses may be generated synthesizing thin films of these materials with [111] strain72. Our experimental results strikingly confirm recent first-principles predictions52. The positive sign of Jbq suggests that the effect of inter-site electron hopping outweighs spin-lattice coupling, since the latter would give a negative contribution to Jbq11,12. Crucially, we quantify the interactions that stabilize the noncoplanar state, in contrast to proposed 3-q structures in NiS273,74,75, MnTe276, and UO277,78,79,80, where the relevant interactions are not yet well understood. Our work provides guiding principles to facilitate the identification of multi-q spin textures. First, double perovskites offer a rich materials space in which 1-q and multi-q structures may be nearly degenerate on the FCC lattice. In candidate materials, the crystal symmetry should be higher than a 1-q model would imply. Second, magnets that are not deep inside the Mott-insulating regime are expected to have larger Jbq and, consequently, more robust 3-q orderings. This criterion hints that cubic Ba2YOsO681,82 may also host a 3-q state, due to its extended Os 5d orbitals, potentially offering a route to investigate the effect of increased electron hopping. For small Jbq, we anticipate a thermally-induced transition from 3-q to 1-q ordering, since thermal fluctuations favor collinear states. Third, quartic single-ion anisotropy may play a role in FCC magnets with S > 3/2; in particular, easy-〈111〉 axis anisotropy should favor 3-q ordering. The relevance of 3-q ordering extends beyond the FCC lattice: two themes are the presence of 3-fold or 6-fold symmetries, such as in triangular, honeycomb, and kagome lattices, and the presence of higher-order interactions than Heisenberg exchange, such as biquadratic83,84 and ring exchange terms85. A methodological insight that may be relevant for several lattices60 is that established spin-wave methods can distinguish 1-q and multi-q structures by optimizing structure and interactions simultaneously. This result highlights that neutron-scattering experiments on bulk polycrystalline insulators are complementary to approaches such as γ-ray emission spectroscopy21 and scanning-tunneling spectroscopies on conducting or thin-film materials86.

Methods

Sample preparation and characterization

Polycrystalline samples were prepared by conventional solid state reactions35. Rare-earth oxides and RuO2 powder were first dried at 900 °C overnight. A stoichiometric mixture of BaCO3, M2O3 (M = Ba or Lu), and RuO2 was thoroughly mixed, pelletized, and fired at 1315 °C for a week with intermittent regrinding. Magnetic properties were measured with a Quantum Design Magnetic Property Measurement System in the temperature range 2 < T < 300 K. The temperature dependence of magnetization suggests both compounds order antiferromagnetically below ~ 37 K, consistent with refs. 34,35,36.

Neutron scattering data

Inelastic neutron-scattering data (Ei = 60 and 11.8 meV) were corrected for detector efficency using a vanadium standard, for absorption, and for background scattering by subtraction of empty-container measurements. The data were placed in absolute intensity units (mb meV−1 Ru−1) by normalization to the nuclear Bragg profile.

Rietveld refinements were performed using the Fullprof program87,88. The peak-shape was modeled as a Gaussian with H2 = \({(U{\tan }^{2}\theta +W)}^{2}\), where H is the full-width at half-maximum of the peak, and U and W are refined parameters89. We also refined the cell parameter, oxygen position parameter, magnetic moment length, intensity scale factor, zero offset, and fourth-order polynomial background terms, and a scale factor for the Y2O3 or Lu2O3 impurity phase. Atomic displacement parameters were neglected due to the low sample temperature and limited Q-coverage of the data.

X-ray diffraction data

X-ray diffraction data were collected at 20 K and 300 K on portions of the same samples measured by neutrons, using a Panalytical XPert Pro diffractometer and Cu Kα radiation (λ = 1.540598 Å, Ge(111) monochromator). The Bragg-Brentano geometry was used. The low-temperature collection used an Oxford Phenix cryostat.

Rietveld refinements were performed using the Fullprof program87,88. The peak-shape was modeled using a pseudo-Voigt function, with refined U, V, W, X and η0 parameters, and four refined asymmetry parameters. Background was modeled using 6th-order Chebychev polynomials. Sample displacement and micro-absorption corrections were refined. Reflections from the Al sample holder were fitted using LeBail profile matching.

We obtained excellent agreement between refined and experimental profiles within the published cubic \(Fm\bar{3}m\) model34 at 20 K and 300 K (see Supplementary Fig. 1). No peak splitting or selective broadening was observed. This result confirms that Ba2LuRuO6 and Ba2YRuO6 are cubic above and below TN, in agreement with previous studies34,36.

The level of chemical disorder and impurity phases were quantified using Rietveld refinements. Refined values of all structural parameters are given in Supplementary Table 1, including the level of M/Ru site mixing, an overall atomic-displacement (Biso) factor, and the oxygen position parameter x. Two minor impurity phases were identified: M2O3 (0.34(3) wt.% for M=Y, 0.93(9) wt.% for M=Lu) and Ba3MRu2O9 (0.71(10) wt.% for M=Y, 2.6(4) wt.% for M=Lu). The value of Biso is larger for Ba2LuRuO6 than for Ba2YRuO6, in agreement with ref. 34. In Ba2YRuO6, the level of Y/Ru site mixing refines to zero with an uncertainty of a few percent, consistent with previous high-sensitivity 89Y magic-angle-spinning NMR measurements36. In Ba2LuRuO6, we refined a small but nonzero amount of Lu/Ru site mixing of 6(2)%. As discussed in ref. 36, the difference of x from \(x=\frac{1}{4}\) is an indicator of the level of site ordering. As such, the values of x reported in Supplementary Table 1 are consistent with slightly increased site mixing for M=Lu compared with M=Y, which may also be reflected in the larger Biso for M=Lu.

Spin-wave model fitting

Refinements were performed against two inelastic neutron-scattering data sets simultaneously: a slice with 0.3 ≤ Q ≤ 2.5 Å−1 and 2.5 ≤ E ≤ 20 meV (Ei = 60 meV data), and a cut at Q = 0.7450 ± 0.0175 Å−1 with 0.75 ≤ E ≤ 6.25 meV (Ei = 11.8 meV data). All data points were weighted by their uncertainties as 1/σ2. We minimized the function

$${\chi}^{2}=\sum\limits_{Q,E}{\left[\frac{{I}_{{{{\rm{expt}}}}}(Q,E)-s{I}_{{{{\rm{calc}}}}}(Q,E)-bE}{\sigma (Q,E)}\right]}^{2},$$
(2)

where subscript “expt” and “calc” indicate measured and calculated magnon spectra, respectively, σ is an experimental uncertainty, s is a refined overall scale factor, and b is a refined linear-in-E background term; it is assumed that s and b are equal for both data sets. To account for the instrumental resolution, calculations were convoluted with an energy-dependent Gaussian energy broadening (elastic FWHM = 1.68 and 0.27 meV for Ei = 62 and 11.8 meV, respectively) and a constant Gaussian Q-broadening (FWHM = 0.08 and 0.054 Å−1 for Ei = 62 and 11.8 meV, respectively). The parameter values were optimized using the Minuit nonlinear least-squares program using its derivative-based MIGRAD and derivative-free SIMPLEX algorithms90,91. For each refined model, at least four separate refinements were performed with different initial parameter values, to check for alternative χ2 minima. Notably, minima with similar overall χ2 were found for the 3-q structure with J, K and Jbq interactions. These minima corresponded to smaller values of K, which has a noticeable effect on the magnon spectra only at low energies. These minima yield a magnon gap that is significantly smaller than the observed gap, and were therefore neglected.