Introduction

Magnetic frustration plays an indispensable role in the formation of exotic states of matter. At the level of nearest-neighbor interactions, there are two main sources of magnetic frustration: geometric frustration coming from the lattice structure itself, and exchange frustration originating from anisotropic spin interactions. By generating strong quantum fluctuations, both types of frustration can stabilize a quantum spin liquid (QSL) state1, which offers a unique insight into how collective quantum behavior deviates from classical intuition2. While geometric frustration is commonly discussed in the context of non-bipartite (e.g., triangular, kagome, and pyrochlore) lattices, the most well-known example for exchange frustration is the Kitaev model on the honeycomb lattice, whose ground state is an exactly solvable QSL characterized by emergent gauge fluxes and Majorana fermions as elementary excitations3.

The S = 1/2 triangular lattice antiferromagnet (TLAF) provides a quintessential manifestation of geometric frustration as the three antiferromagnetic interactions surrounding each triangle cannot be simultaneously satisfied. In the presence of both nearest-neighbor (J1) and next-nearest-neighbor (J2) Heisenberg interactions, this system is suggested to host a QSL ground state4,5,6,7,8,9,10, which is notably stabilized by a very small J2 ≈ 0.06J1 (i.e., QSL arises for 0.06 < J2/J1 < 0.16, see Fig. 1). Consequently, even though the purely nearest-neighbor Heisenberg TLAF (J2 = 0) develops 120° magnetic order, its spin dynamics are still heavily influenced by quantum fluctuations originating from the proximate QSL phase. In particular, the measured excitation spectrum, which contains a strong continuum, cannot be reproduced by spin-wave theory even with non-linear (1/S) corrections11,12,13,14. Instead, it has been accurately modeled through a Schwinger boson approach based on fractionalized excitations (spinons) of the nearby QSL15,16. In this framework, the 120° ordered phase and its low-energy magnons are described through a spontaneous Bose–Einstein condensation of spinons and bound states of spinon pairs, respectively (i.e., spinons are confined). However, due to the proximity to a quantum melting point, a phase boundary of the QSL, spinons exhibit markedly less confinement at higher energies and freely propagate across multiple lattice spacings. Consequently, they manifest as an intense multi-spinon continuum above the one-magnon bands. Notably, this scenario is in stark contrast to unfrustrated two-dimensional quantum antiferromagnets, where the interacting magnon picture usually provides a quantitatively correct description of the excitation spectrum17,18.

Fig. 1: Phase diagram of the S = 1/2 XXZ TLAF.
figure 1

Δ = 1 (Δ = 0) corresponds to the Heisenberg (XY) model. The phase boundaries are obtained from the DMRG calculation in ref. 19. The shaded region indicates a QSL phase. The two unshaded regions correspond to the 120° and stripe long-range order, respectively. Eight data points excluding Ba2La2CoTe2O12 (BLCTO, a solid blue star symbol) are based on the parameters given in previous inelastic neutron scattering studies15,16,32,62,63,64,65,66. The error bars associated with BLCTO depict the uncertainties derived from the analysis of our high-temperature INS data.

Full size image

In the presence of anisotropic spin interactions, the S = 1/2 TLAF serves as a rich environment for combining geometric frustration with exchange frustration. Recent theoretical efforts have studied a general symmetry-allowed spin model for the S = 1/2 TLAF and revealed a rich phase diagram with intriguing phases not accessible by the purely isotropic (i.e., Heisenberg) model19,20. However, experimental exploration of this spin model is still in its early stages due to the lack of suitable S = 1/2 TLAF materials. Surprisingly, this challenge persists even for the planar XXZ anisotropy, which, among the various symmetry-allowed anisotropic exchange interactions, stands out as the simplest yet fundamentally important component:

$$H={\sum}_{n,{\left\langle {ij}\right\rangle }_{n}}{J}_{n}\left({S}_{i}^{x}{S}_{j}^{x}+{S}_{i}^{y}{S}_{j}^{y}+\varDelta {S}_{i}^{z}{S}_{j}^{z}\right).$$
(1)

Here, Jn represents the interactions between nth nearest neighbors, and Δ < 1 denotes the degree of the XXZ anisotropy. This anisotropy smoothly transforms the Heisenberg model (Δ = 1) into the XY model (Δ = 0), each constituting its own universality class of phase transitions with distinct properties (e.g., Berezinskii–Kosterlitz–Thouless transition21,22). Thus, the planar XXZ anisotropy is an integral part of many foundational models in quantum magnetism and has been extensively studied in, for example, one-dimensional spin chains23.

In a S = 1/2 TLAF, the planar XXZ anisotropy (Δ < 1) is expected to suppress the inherent quantum fluctuations, according to recent theoretical works19,20,24. This is evident in the phase diagram of the J1–J2–Δ model (see Fig. 1) calculated by the density matrix renormalization group (DMRG)19,24. As Δ deviates from 1, the QSL phase that separates the 120° and stripe magnetic orders is gradually suppressed and completely vanishes around Δ ~ 0.3. This suggests that a magnon-based description of the spin dynamics may become more accurate in an S = 1/2 TLAF for small enough Δ. However, as shown in Fig. 1, such an effect of the XXZ anisotropy on S = 1/2 TLAFs has yet to be investigated experimentally. Even though the XXZ model has already been applied to some S = 1/2 TLAF experimental systems such as Ba3CoSb2O9 (refs. 16,25), the anisotropy is almost negligible in these materials (Δ ≥ 0.9) and thus inadequate to test the above hypothesis. In contrast, for S = 1/2 TLAFs with Ising-type XXZ anisotropy (Δ > 1), recent experimental studies have unveiled fascinating static and dynamic magnetic properties associated with strong quantum fluctuations of the suggested spin supersolid phase26,27,28. This further calls for an experimental investigation into the case with strong planar XXZ anisotropy as it will enable us to understand the connection between the XY, Heisenberg, and Ising limits in S = 1/2 TLAFs, each exhibiting markedly different physical behaviors.

Here, we make a significant stride toward understanding the interplay between planar XXZ anisotropy and geometrical frustration in a S = 1/2 TLAF. We use inelastic neutron scattering (INS) and advanced theoretical calculations to study the magnetic excitations of Ba2La2CoTe2O12 (BLCTO), a Co2+-based S = 1/2 TLAF that exhibits long-range 120° order on the a–b plane below TN = 3.26 K29. First, by analyzing the excitation spectrum at T > TN with a state-of-the-art calculation technique, we establish that the spin Hamiltonian of BLCTO contains a substantial planar XXZ anisotropy: Δ = 0.56(2). Second, we present evidence of strong quantum fluctuations in BLCTO; the spectrum below TN exhibits continuum excitations up to approximately four times the single-magnon bandwidth, and these excitations possess a larger spectral weight than the single-magnon modes. A comparison with two advanced theoretical calculations suggests that a framework based on fractionalized spinons yields better quantitative agreement with the measured excitation spectrum than that of interacting magnons, highlighting the pivotal role played by quantum fluctuations even in the highly anisotropic regime of the phase diagram in Fig. 1. We also highlight another distinctive aspect of the planar XXZ anisotropy in S = 1/2 TLAFs: the pronounced flatness of the single-magnon and continuum excitations, which can be related to a higher-order van Hove singularity (VHS) in the magnon spectrum near Δ ~ 0.6.

Results and discussion

BLCTO possesses a trigonal crystal structure (\(R\bar{3}\) space group) with Co2+ sites forming layered triangular lattices29 (Fig. 2a). The large separation between adjacent Co2+ triangular layers (~9.16 Å) implies highly two-dimensional magnetic behavior. The Co2+ site is surrounded by six O2− ligands configuring octahedral coordination (Fig. 2b). As found in many other Co2+ systems with a similar environment14,30,31,32, spin-orbit coupling (SOC) lifts the degeneracy of the T1 manifold of the Co2+ ions in BLCTO, resulting in a Kramers’ doublet ground state (Fig. 2c) that gives rise to a pseudospin-1/2 entity ( Jeff = 1/2). The most direct means of verifying the Jeff = 1/2 picture in this class of materials is the observation of an excitation from Jeff = 1/2 to Jeff = 3/2 multiplets (Fig. 2c)30,31,33,34. Our INS measurements at 5 K (above TN) clearly show this transition (see Fig. 2d); an excitation corresponding to this transition is observed at ~28 meV, consistent with the energy scale of the single-ion Co2+ SOC strength (λSOC ~ 26 meV). Thus, BLCTO can be effectively treated as an S = 1/2 triangular lattice system when only the ground state doublet is thermally populated, i.e., for temperatures T < 100 K satisfying 3kBT < E1/2→3/2, where E1/2→3/2 is the transition energy from Jeff = 1/2 to Jeff = 3/2. Several hallmark characteristics of a S = 1/2 TLAF have been identified in BLCTO, including a substantial frustration factor f = θCW/TN ~ 30 (Fig. 2f, g), a field-induced 1/3–magnetization plateau phase29, and a total magnetic entropy close to Rln2 as estimated from heat capacity measurements (Fig. 2h, i).

Fig. 2: Anisotropic quantum magnetism of Co2+ ions in BLCTO.
figure 2

a Crystallographic unit cell of BLCTO. b Large trigonal compression of a CoO6 octahedron in BLCTO29. c Schematic representation of the energy level structure of Co2+ with cubic crystal electric field, SOC, and trigonal distortion (δ > 0). λSOC denotes the single-ion SOC strength of Co2+ (~26 meV). d The INS spectrum (I(Q, ω), arb. units) of BLCTO measured at 5 K ( > TN), demonstrating the presence of Jeff = 1/2 → Jeff = 3/2 excitations. e The energy level structure of Co2+ with different strength ratios of trigonal distortion (δ) to SOC (see Supplementary Notes). The dashed vertical blue line indicates a value δ/λSOC = 1.854 for BLCTO. f Temperature-dependent magnetization of BLCTO. The inset highlights a phase transition at TN = 3.26 K. g The Curie–Weiss behavior of BLCTO. The solid orange lines in (f) and (g) are the result of curve fitting based on the modified Curie–Weiss law (see “Methods”). h Temperature-dependent specific heat (Cp) of BLCTO under zero (gray circles) and 9-T (blue triangles) magnetic field, presented on a logarithmic scale. The solid black line is the non-magnetic contribution estimated by the Debye model (see “Methods”). i Remnant specific heat after subtraction of the non-magnetic component from (h) (gray circles) and the corresponding magnetic entropy (solid orange line).

Full size image

However, as we show in this work, what sets BLCTO apart from other S = 1/2 TLAFs is its significant planar XXZ anisotropy. In general, a strong coupling between spin and orbital degrees of freedom in a Jeff = 1/2 ground state facilitates the emergence of strong anisotropy in exchange interactions35,36. In BLCTO, the Jeff = 1/2 ground state coexists with a trigonal distortion of the octahedral environment (Fig. 2b), which can generate sizable XXZ anisotropy37,38. The proportional relationship between XXZ anisotropy and trigonal distortion in a Co2+ magnet, each quantified by Δ and the deviation of an O–Co–O angle from 90o (See Supplementary Fig. 1), is evident in Supplementary Table 1. BLCTO exhibits an exceptionally large trigonal compression of CoO6 octahedra29 compared to the other compounds (Fig. 2b). This is consistent with the fact that the measured E1/2→3/2 is much smaller than its theoretical value 3λsoc/2 ~ 39 meV without the distortion, implying the presence of a large trigonal crystal field33,34 (Fig. 2e). For a more detailed explanation, see Supplementary Notes. These observations collectively make BLCTO a compelling case for strong XXZ anisotropy and, therefore, an attractive model system to study the unexplored region of the phase diagram shown in Fig. 1.

The strong quantum fluctuations present in S = 1/2 TLAFs disqualify many of the conventional methods for determining a material’s spin Hamiltonian. Specifically, attempts to fit the magnetic excitations in the ordered phase (T < TN) using linear spin-wave theory (LSWT) or non-linear spin-wave theory (NLSWT) do not produce reliable parameters. To overcome this challenge, we have examined the spin Hamiltonian of BLCTO by analyzing its excitation spectra measured above TN. In this regime, the spin dynamics exhibit a semi-classical behavior owing to substantial thermal fluctuations. We analyzed the momentum- and energy-resolved paramagnetic scattering profile using Landau–Lifshitz dynamics (LLD) simulations (see “Methods”). This methodology provides more comprehensive insights than the traditional energy-integrated diffuse scattering analysis based on the self-consistent Gaussian approximation.

The energy-resolved excitation spectra of paramagnetic BLCTO unambiguously reveal the presence of strong planar XXZ anisotropy. Figure 3a shows momentum- and energy-resolved excitation spectra of polycrystalline BLCTO measured at four temperatures above TN. We first compare the entire dataset to LLD simulations with (Δ = 0.56, Fig. 3b) and without (Δ = 1, Fig. 3c) XXZ anisotropy. These two models yield distinct energy-dependent profiles, with the main differences highlighted in Fig. 3d which shows the overall energy dependence for T = 1.63TN by integrating over multiple magnetic zones in the powder spectrum. The isotropic model (Δ = 1) fails to replicate the concentration of the INS cross-section in the low-energy region, and this discrepancy is remedied by reducing Δ (see Supplementary Fig. 4). For more quantitative analysis, we conducted a brute-force comparison between the entire dataset in Fig. 3a and the INS cross-sections (I(|q|, ω)) calculated from LLD across a wide two-dimensional parameter space: 2 meV <J1 < 2.9 meV and 0.3 <Δ < 1.0, assuming J2 = 0 (see “Methods”). The result, depicted in Fig. 3e, reveals a clear minimum of the chi-square between the observed and simulated INS cross-sections (\({\chi }^{2}\)(J1, Δ)) at J1 = 2.62 meV and Δ = 0.56(2). Indeed, the corresponding LLD simulation result (Fig. 3b) shows an excellent agreement with the data, whose detailed comparison can be found in Supplementary Fig. 5. This optimal parameter set clearly indicates the presence of strong planar XXZ anisotropy in BLCTO, in stark contrast to other previously reported examples of S = 1/2 TLAFs (see Fig. 1). The same conclusion is derived from our additional analysis based on the energy-integrated diffuse scattering profile, as shown in Supplementary Fig. 9.

Fig. 3: Quantitative analysis of magnetic excitations in the paramagnetic phase of BLCTO (T > TN).
figure 3

a The INS data for BLCTO (I(|Q|, ω), arb. units) measured at various temperatures above TN. The incident neutron energies of the four datasets were 6 meV, 10 meV, 10 meV, and 10 meV, respectively. b Optimal fit of the LLD simulation to the data in (a), resulting from a parameter set J1 = 2.62 meV and Δ = 0.56. c Excitation spectra calculated from the LLD simulation with J1 = 2.62 meV and Δ = 1 (Heisenberg). d Comparison between the measured energy-dependent paramagnetic scattering profile and the LLD simulations for Δ = 0.56 and 1. Standard deviations of the data points are much smaller than the data symbol. e The goodness-of-fit quantified through chi-square (\({\chi }^{2}\)) for different J1 and Δ values (see “Methods”). The position of the minimal \({\chi }^{2}\) (J1 = 2.62 meV and Δ = 0.56(2)) is indicated by a green star.

Full size image

Importantly, this result remains robust even when the analysis is extended to include next-nearest-neighbor interactions J2 (see Supplementary Note and Supplementary Figs. 6–8) and bond-dependent anisotropic exchange interactions (see Supplementary Note and Supplementary Fig. 15). Notably, J2 has a stronger impact on the |q|-dependence of I(|q|, ω), while the effect of Δ is more pronounced along the ω–axis (Supplementary Fig. 4–6). The optimal parameter set obtained from the J1–J2–Δ model (J1 = 2.70(5) meV, Δ = 0.54(3), J2 = 0.03(1)J1) is marked as a blue star in Fig. 1. Another noteworthy component is the inter-layer coupling (Jc), which was suggested to be ferromagnetic based on the interlayer spin configuration confirmed by neutron diffraction29. Yet the layered crystal structure (Fig. 2a) and a broad maximum of the temperature-dependent magnetization (Fig. 2f) suggest strong 2D magnetic character in BLCTO, i.e., Jc should be much smaller than J1. In turn, adding such a small Jc will only result in a marginal change of a powder-averaged excitation spectrum (see Supplementary Fig. 16).

Having established the presence of strong planar XXZ anisotropy in BLCTO, we now examine the spin dynamics in the ordered phase to determine the impact of quantum fluctuations. Notably, this provides, to our knowledge for the first time, experimental insights into the nature of magnetism in a 120°-ordered S = 1/2 TLAF with strong planar XXZ anisotropy. Figures 4a, e show the measured excitation spectra of BLCTO at 1.8 K (<TN = 3.26 K). In addition to the one-magnon spectrum located below ~2 meV, the data reveal remarkable features at higher energies. A substantial region of the |Q|–E space above the one-magnon region is covered by a continuum signal, extending up to roughly four times the one-magnon bandwidth; see Fig. 4i. Notably, the energy-dependent profile of Fig. 4e demonstrates a multi-level structure inside the continuum with two local maxima centered at around 2.2 meV and 3.1 meV (Fig. 4j), respectively designated by purple and red arrows throughout Fig. 4. Most importantly, the spectral weight of this high-energy continuum surpasses that of the one-magnon band: \(\frac{{I}_{{{{\rm{highE}}}}}}{{I}_{1-{{{\rm{mag}}}}}} \sim 1.34\), according to Fig. 4i. Given that the fine quality of our sample (see “Methods”) reasonably excludes structural disorder as a contributing factor on the continuum, this observation suggests that quantum fluctuations play a significant role in the S = 1/2 TLAF with strong planar XXZ anisotropy.

Fig. 4: Magnetic excitations of BLCTO in the 120° ordered phase (T < TN).
figure 4

a INS data (I(|Q|, ω), arb. units) measured at 1.65 K (= 0.506TN) with Ei = 10 meV. b–d Corresponding INS cross-sections calculated from LSWT, NLSWT, and Schwinger boson theory. The calculations were conducted with J1 = 1.55 meV or 1.77 meV, Δ = 0.56, and J2 = 0. e INS data measured at 1.65 K (= 0.506TN) with Ei = 6 meV. f–h Equivalent to (b–d), but convoluted with the coverage and instrumental resolution of the data in (e). i–j Measured and calculated energy-dependent INS cross-sections in the low-|Q| region (0.5 \({{{\text{\AA }}}}^{-1}\) < |Q| < 1.5 \({{{\text{\AA }}}}^{-1}\)). The dashed vertical lines in (i, j) denote the one-magnon bandwidth. k, l Single-crystal excitation spectra along high-symmetry directions (I(q,ω), arb. units), calculated by NLSWT and Schwinger boson theory, respectively. m–o, |Q| dependence of the measured (Ei = 6 meV) and calculated INS cross-sections at the energies indicated by the red, purple, and gray arrows in (e, j). Standard deviations of the data points in (i, j) and (m–o) are much smaller than the data symbol. Only the calculation results in panels (a–j) and (m–o) are convoluted with instrumental resolution.

Full size image

For a deeper understanding, we compare our data with NLSWT and Schwinger boson calculations, which are theoretical approaches based on interacting magnons and fractionalized spinons, respectively. Our Schwinger boson approach employs a 1/N expansion centered around a saddle point (SP) solution, where N represents the number of boson flavors. Although the SP solution captures the 120° magnetic order through spontaneous Bose–Einstein condensation of Schwinger bosons, it cannot account for collective modes of this magnetic order, which are magnons. This limitation arises because the SP Hamiltonian models a non-interacting spinon gas, whereas magnons emerge as bound states of two spinons. Thus, by incorporating the 1/N correction, which acts as a counter-diagram for the single-spinon poles of the SP dynamical spin susceptibility and captures the single-magnon poles as two-spinon bound states15,16,39 (see Supplementary Fig. 18), we achieve a spectrum consisting of both low-energy single-magnon modes and a two-spinon scattering continuum above them. Upon incorporating a higher order of the 1/N expansion (1/N2), an effective two-magnon continuum will also appear as the pair creation of two-spinon bound states, extending to twice the highest energy of the two-spinon bound states. However, the spectral weight of this two-magnon continuum is expected to be smaller than that of the multi-spinon continuum.

While the Schwinger boson theory is known to better describe the excitation spectra of S = 1/2 Heisenberg TLAFs (refs. 15,16,39), one may expect NLSWT to be a more suitable approach in the XY limit, as described in the Introduction. Thus, it is not a priori clear which approach better agrees with the excitation spectrum of BLCTO, and comparing the results of each calculation with the BLCTO data reveals how important quantum fluctuations are in the spin dynamics of S = 1/2 TLAFs with strong planar XXZ anisotropy. For each calculation, we used Δ = 0.56(2) determined from our analysis of the high-temperature spin dynamics when J2 = 0 (Fig. 3). However, adjusting J1 to be between 1.55 meV and 1.77 meV (depending on the calculation method, see Fig. 4b–d and Supplementary Fig. 13) was necessary to align with the overall scale of our INS data measured below TN. This modification is due to the inability of our semi-classical LLD simulation to account for the effective magnetic moment μeff\(\sim \,\)g(S(S + 1))1/2 at high temperature40. Consequently, the LLD result overestimates J1 by a factor of at most \({\left(\frac{S\left(S+1\right)}{{S}^{2}}\right)}^{1/2}\) (i.e., about 1.732 for S = 1/2), which is comparable to the ratio between the values of J1 obtained from our high- and low-temperature analyses. Indeed, the smaller J1 values derived at T < TN better align with the experimental M–H curve of BLCTO29 (see Supplementary Notes). We also note that anisotropic g-factors are taken into account in the calculations for both T > TN and T < TN, which can substantially change the distribution of theoretical INS cross-sections (see “Methods” and Supplementary Notes).

Figure 4c–d, g–h shows the NLSWT and Schwinger boson calculation results convoluted with the instrumental resolution profile for Ei = 10 meV (6 meV). While both calculations qualitatively capture the multi-level structure at 1.5 meV, 2.2 meV, and 3.1 meV (Fig. 4i–l), detailed features in the measured spectrum (e.g., an intense flat mode around 1.5 meV, see Fig. 4e) are better described by the Schwinger boson calculation, as seen in Fig. 4h. Most importantly, these two calculations exhibit distinct spectral-weight ratios of the high-energy continuum to the one-magnon component: \(\frac{{I}_{{{{\rm{highE}}}}}}{{I}_{1-{{{\rm{mag}}}}}}\). Based on the plots shown in Fig. 4i–j, we obtained a spectral-weight ratio between 0.22 and 0.25 for the NLSWT calculation and between 1.49 and 1.68 for the Schwinger boson calculation. Indeed, NLSWT significantly underestimates the spectral weight of the continuum above the one-magnon band, as evident in Fig. 4i–j, m, and n. In contrast, the Schwinger boson theory captures \(\frac{{I}_{{{{\rm{highE}}}}}}{{I}_{1-{{{\rm{mag}}}}}}\) > 1 and the intensity modulation along |Q| (Fig. 4m–o) observed in the data. Note that the spectral weight above ~4 meV found in the data (approximately 21% of the total spectral weight), which is not captured by the Schwinger boson theory, is due to the presence of higher-order n-spinon states with n > 2, which are not implemented in our current calculation16. Including these terms in the calculation is expected to transfer the overestimated spectral weight around the 3rd level (red arrow in Fig. 4j) to higher energies and better describe the INS data. To summarize, the spinon-based picture underpinning Schwinger boson theory is found to be a better framework for reproducing the excitation spectrum of BLCTO, suggesting that spinons are fundamental ingredients for describing the spin dynamics of S = 1/2 TLAFs with significant planar XXZ anisotropy. This is presumably because a quantum melting point remains proximate to the 120° ordered phase even under strong planar XXZ anisotropy, which may largely weaken the confinement of spinons. Thus, the noticeable suppression of quantum fluctuations by planar XXZ anisotropy is expected to be found at Δ much smaller than ~0.5. Nevertheless, it is noteworthy that the Schwinger boson calculation overestimates the \(\frac{{I}_{{{{\rm{highE}}}}}}{{I}_{1-{{{\rm{mag}}}}}}\) ratio found in our data, which signals that BLCTO is still less quantum than Heisenberg TLAFs, as anticipated from the phase diagram in Fig. 1.

We now turn our attention to another interesting feature of the magnetic excitations that result from strong planar XXZ anisotropy. In addition to the intense continuum signal, the 1st level (gray arrows in Fig. 4) and 2nd level (purple arrows in Fig. 4) excitations exhibit a flat profile along the momentum transfer axis. In the language of magnons, the flat continuum signal can be interpreted as originating from the flatness of the single-magnon dispersion within a portion of the Brillouin zone. Since there are many magnons at the approximately constant energy E of this flat region, there are many channels for creating two magnons with total energy 2E, resulting in a pronounced peak in the two-magnon density of states at energy 2E. This rationale is evidenced in Supplementary Fig. 11: when the overall one-magnon dispersion becomes noticeably flat (Δ < 0.5), the NLSWT spectrum manifests a flat, isolated profile of a two-magnon continuum.

Understanding the origin of the wide flat region in the single-magnon dispersion reveals a unique role played by the planar XXZ anisotropy. To see this, it is useful to describe the magnons in a local reference frame where the spins are rotated in the plane of the 120° order such that they all point in the same direction41,42. In this rotating reference frame, the magnon spectrum has only one band, which supports a Goldstone mode at the Γ point for any value of the planar XXZ anisotropy Δ. While the K points also host Goldstone modes in the Heisenberg limit (Δ = 1), they become gapped for a generic value of Δ and even evolve into local band maxima in the XY limit (Δ = 0). Hence, as Δ is progressively reduced, each K point must undergo a transition from a local minimum to a local maximum at a critical value Δ = Δ0. Due to the high symmetry of each K point, this transition at the critical anisotropy Δ0 must happen through a higher-order VHS, at which all first and second derivatives of the magnon band dispersion vanish and the magnon density of states exhibits a power-law divergence ρ(ε) ~ |ε – E|–1/3 43,44,45,46. To show this, we consider the general symmetry-allowed Taylor expansion of the magnon dispersion up to the third order around a given K point. If we introduce the relative momentum (kx, ky), the threefold rotation symmetry around (0, 0) (i.e., the K point itself) and the reflection symmetry ky → −ky imply that this Taylor expansion takes the form:

$$E\,({k_x},{k_y})=E\,(0,\,0)+{c}_{2}\,\left({k}_{x}^{2}+\,{k}_{y}^{2}\right)+{c}_{3}\,\left({k}_{x}^{3}-3{k}_{x}{k}_{y}^{2}\right).$$
(2)

Clearly, the transition between the local minimum and the local maximum is driven by a sign change of the lowest-order nonconstant term, i.e., the second-order term of coefficient c2. Since c2 = 0 at the transition (Δ = Δ0), the lowest order magnon dispersion around the K point is then determined by the third order term, which precisely corresponds to the higher order VHS (with a shape referred to as a monkey saddle) discussed in, for example, ref. 43. The magnon dispersions on the two sides of the transition and at the transition itself are shown in Fig. 5. We also emphasize that, by returning from the rotating reference frame to the laboratory frame, two additional shifted copies of the original magnon dispersion arise in energy-momentum space, resulting in flat regions coming from the higher-order VHS not only around the K points but also around the Γ point. The above arguments derived from Eq. (2) are independent of the models for elementary excitations, i.e., they hold for LSWT, NLSWT, and Schwinger Boson calculations alike. Indeed, all three different calculation methods consistently suggest that planar XXZ anisotropy enhances the flatness of one-magnon bands, and the flatness is maximized at appropriate values of Δ much lower than 1, which corresponds to Δ0 (Supplementary Figs. 10–12 and 19).

Fig. 5: Flat magnon dispersion (“monkey saddle”) around the K points induced by planar XXZ anisotropy (Δ).
figure 5

Magnon energy E against magnon momentum (kx, ky) around one of the K points (black cross) for Δ > Δ0 (left), Δ = Δ0 (center), and Δ < Δ0 (right), as given by Eq. (2) for c2 > 0, c2 = 0, and c2 < 0, respectively. While the K point is a local minimum (maximum) for Δ > Δ0 (Δ < Δ0), it realizes a higher-order VHS at the critical anisotropy Δ = Δ0. Note that smaller (larger) magnon energies are marked by red (yellow) color.

Full size image

Interestingly, the experimentally determined anisotropy Δ of BLCTO (0.56(2) for the J1–Δ model) is not far from the theoretical critical anisotropy Δ0. While LSWT and NLSWT give Δ0 ≈ 0.25 and 0.35, respectively, significantly smaller than the experimental value for BLCTO, Schwinger boson theory, which has been found to be the most suitable approach for computing the spin dynamics, gives Δ0 ≈ 0.6 (see Supplementary Fig. 14). Indeed, our Schwinger boson calculations corresponding to the case of BLCTO (Δ = 0.56) show that the nearly flat dispersion at ~1.5 meV persists along the entire high-symmetry line Γ–K–M (see the single-crystal spectrum in Fig. 4l). However, the dispersion around the Γ and K points becomes less flat when moving away from Δ0 ≈ 0.6 (Supplementary Fig. 14), resulting in a less flat powder-averaged spectrum (Supplementary Fig. 12). Thus, these flat excitations originating from a higher-order VHS are unique features of the strongly anisotropic XXZ model realized in BLCTO, and provide a clear distinction from both the XY (Δ = 0) and Heisenberg (Δ = 1) limits.

Finally, we discuss the validity and limitations of our spin Hamiltonian. Although we proposed a relatively simple J1–J2–Δ model for the spin Hamiltonian of BLCTO (or approximately, the J1–Δ model as J2 is very small), bond-dependent anisotropic interactions (J±± and Jz± in ref. 20), which transform into Kitaev (K) and Gamma (Γ) terms in local cubic axes19,20, are in principle allowed in BLCTO by symmetry. Notably, Co2+ systems have proven to be an excellent platform for discovering significant bond-dependent anisotropic exchange interactions33,34,47,48. Although the isolated CoO6 octahedra in BLCTO differ from previously reported Kitaev materials that consist of an edge-shared octahedron network of ligands (this is an important prerequisite of the Jackeli–Khaliullin mechanism49), a possibility of Kitaev interactions in an isolated octahedron network was suggested theoretically50,51. Thus, we additionally analyzed the paramagnetic excitation profiles with the J1–J2–Δ–J±±–Jz± model. While single-crystal INS measurements will be necessary to quantify J±± and Jz± with better accuracy, our analysis suggests that J±± and Jz± are not significant (see Supplementary Notes and Supplementary Fig. 15). The correlation between this outcome and the existence of a nearly gapless Goldstone mode (Supplementary Fig. 17) also merits more discussion, which is included in Supplementary Notes.

In summary, we have investigated the spin dynamics of BLCTO using INS, marking to our best knowledge the first comprehensive experimental study of an S = 1/2 TLAF with strong planar-type XXZ anisotropy. We implemented a new reliable method to extract the spin Hamiltonian of a S = 1/2 TLAF from energy- and momentum-resolved spin dynamics at T > TN, which suggests an XXZ anisotropy of Δ ~ 0.56(2) in BLCTO. The full excitation spectrum below TN reveals an intense and structured continuum signal distributed up to an energy around four times higher than the top of the single-magnon band. This observation, coupled with comparisons to NLSWT and Schwinger boson calculations, supports the persistence of strong quantum fluctuations and indicates that the anomalous continuum scattering is more accurately described by a two-spinon continuum even in the highly anisotropic regime of a S = 1/2 TLAF. The observed flatness of both single-magnon and continuum excitations, along with the associated higher-order VHS predicted by our calculations, further highlights strong planar XXZ anisotropy as an intriguing element in the generalized spin model of S = 1/2 TLAFs. Our results offer a rare experimental perspective on a highly anisotropic S = 1/2 TLAF that bridges the Heisenberg and XY models.

Methods

Sample preparation

Poly-crystalline BLCTO samples were prepared by the solid-state reaction of BaCO, (99.98%, Sigma Aldrich), La2O3 (99.99%, Sigma Aldrich), CoO (99.995%, Alfa Aesar), and TeO2 (99.995%, Sigma Aldrich). They were mixed in stoichiometric quantities by grinding in an agate mortar to produce a starting charge of approximately 15 g of powder. The charge was placed in a 50 ml platinum crucible with a lid and sintered under ambient air condition at 1100 °C for 36 h using an electric resistance box furnace. The heating rate was 100 °C/h. After heating for 36 h, the charge was cooled down to room temperature, and the resulting medium purple powder was reground in an agate mortar and pressed in a 20 mm diameter stainless steel die to produce two pellets. The pellets were placed into the covered platinum crucible and sintered again under identical conditions as the first sintering sequence. The resultant material with a uniform purple color was characterized by powder X-ray diffraction, which confirmed the formation of BLCTO. Further characterization using powder neutron diffraction (see the relevant subsection below) verified the high purity and quality of the sample, as displayed in Supplementary Fig. 2 and Supplementary Table 2.

Bulk property measurements

The magnetic susceptibility of BLCTO was measured by a commercial magnetometer MPMS-XL (Quantum Design, USA). Following the previous ref. 52, we used a modified Curie–Weiss law to fit our M(T) data:

$$\chi \left(T\right)={\chi }_{0}+\frac{C}{T-{\theta }_{{{{\rm{CW}}}}}},$$
(3)

where \({\chi }_{0}\) is an additional van Vleck-like term attributed to the non-negligible population of the Jeff = 3/2 state (see Fig. 2d) at high temperatures. The fitting results for temperatures between 100 K and 150 K are \({\chi }_{0}=4.5(45)\times {10}^{-4}\) emu mol−1 Oe−1, C = 3.73(21) emu mol−1 Oe−1 K−1, and \({\theta }_{{{{\rm{CW}}}}}\) = −99(6) K; see orange solid lines in Fig. 2f, g.

Heat capacity was measured by using a quantum design instrument. The sample consisted of a 4.75 mg pressed pellet held to the calorimeter stage with grease. A non-magnetic (= phonon) contribution to our measured data was estimated by fitting the range of 40–200 K to the simplified Debye model. We used two different Debye temperatures for the fitting, which is justified by the co-existence of heavy (La, Te) and light (O) elements. This comparison yields \({\theta }_{{{{\rm{D}}}}1}\) = 237(2) K and \({\theta }_{{{{\rm{D}}}}2}\) = 746(3) K. While this resulted in a reasonable estimate (Fig. 2i), we acknowledge the potential inaccuracies due to the more complex nature of the actual phonon density-of-states spectrum in BLCTO. Both the magnetic susceptibility and heat capacity results are consistent with ref. 29.

Powder neutron diffraction

For crystal structure characterization, we conducted neutron powder diffraction measurements on the high-resolution powder diffractometer (HB-2A) at the High Flux Isotope Reactor (HFIR), ORNL. The data were collected at 200 K using a constant incident neutron wavelength of 1.539 Å. Rietveld refinements were performed using the FullProf software package53.

Powder INS

We performed two INS measurements using 10.4 g of powder BLCTO sealed with He exchange gas within a thin-walled cylindrical aluminum sample container. The spin-orbit exciton signal was obtained from the SEQUOIA time-of-flight spectrometer at the Spallation Neutron Source (SNS), ORNL. The data were collected at 5 K, 20 K, 100 K, 200 K, and 300 K with incident neutron energies of Ei = 40 meV and 65 meV with standard high-resolution chopper conditions. The low-energy magnetic excitation data were collected at the CNCS time-of-flight spectrometer at the SNS, ORNL. The data were collected at 1.65 K, 5.3 K, 5.6 K, 20.2 K, and 40.4 K using multiple incident neutron energies (Ei = 1.5, 3.3, 6.0, and 10.0 meV for 1.65 K, Ei = 6.0 meV for 5.3 K, and Ei = 10 meV for 5.6, 20.2, and 40.4 K) with standard high-flux chopper conditions. A background signal for each measurement was acquired by measuring a separate empty holder. Unless otherwise described, these data were subtracted from our main data. The instrumental energy resolution was modeled by utilizing the Pychop package in Mantid54. The momentum resolution of each dataset was estimated from the full width at half-maximum (FWHM) of the Bragg peaks of BLCTO and/or aluminum. As a result, we obtained dQ = 0.036 Å−1, 0.044 Å−1, and 0.06 Å−1 for Ei = 3.3 meV, 6.0 meV, and 10.0 meV, respectively. The data cuts along energy transfer E (e.g., Figs. 3d and 4i, j) were obtained by averaging dynamical structure factor values at the same E but different |Q|. We used the same averaging process for the data cuts along |Q|.

Calculating and fitting paramagnetic excitations

To calculate the energy-resolved magnetic excitations of paramagnetic BLCTO, we performed a LLD simulation with the spin system governed by the spin Hamiltonian in Eq. (1). This was done by using the LLD simulation package Su(n)ny55, whose detailed working principles can be found in refs. 56,57,58. We used a 45 × 45 × 2 supercell of BLCTO (12,150 Co2+ sites) for the simulation.

To implement the same degree of thermal fluctuations as the measured data (Fig. 3a) in LLD simulations, we set the simulation temperature of LLD through the following steps. First, we normalized the experimental temperatures by the material’s TN, real = 3.26 K (e.g., 5.3 K = 1.63 TN, real). Then, we computed the theoretical TN of a spin system as a function of exchange parameters (e.g. TN (J1, J2, Δ) for the J1–J2–Δ model) across the entire range of the three-dimensional (J1, J2, Δ) parameter space to be investigated. This calculation was done using a separate classical Monte-Carlo simulation based on LLD and simulated annealing. In this simulation, a Langevin time step dt and the damping constant λ were set to 0.11 meV−1 and 0.1 meV−1, respectively. After waiting for the 1,000 Langevin time steps for equilibration, the 80,000 Langevin time steps were used for the sampling. The effective TN of the simulation was determined from the temperature dependence of the calculated heat capacity, which in turn was derived from the variance of the total energy values sampled throughout the 80,000-time steps. Using the obtained TN (J1, J2, Δ) and the experimental temperature normalized by TN, real, we assigned parameter-dependent simulation temperatures. For instance, we used T = 1.63TN (J1, J2, Δ) for the simulation corresponding to the data measured at 5.3 K = 1.63TN,real.

For the LLD simulation to calculate dynamical structure factors, a Langevin time step was set to dt = 0.11 meV−1, 0.11 meV−1, 0.04 meV−1, and 0.07 meV−1 for simulations at T = 1.63 TN, 1.72 TN, 6.20 TN, and 12.40 TN, respectively. The damping constant λ was fixed at 0.1. The first 1500 Langevin time steps were discarded to wait for equilibration. The calculated dynamical structure factor S(Q,ω) was converted to INS cross-sections I(|Q|, ω) through a process involving powder averaging, applying the magnetic form factor of Co2+, applying neutron polarization factors (1 − QiQj/|Q|2 for Sij(Q, ω)), with an anisotropic g-tensor (see Supplementary Notes and Supplementary Fig. 3) and instrumental resolution convolution. Note that the same treatments were applied to our linear/non-linear spin-wave calculations and Schwinger Boson calculations. To ensure statistically reliable outcomes, we repeated the simulation ten times and used the averaged I(|Q|, ω) for comparisons with the measured I(|Q|, ω).

The goodness-of-fit between the data and LLD simulations was quantified by the chi-square \(({\chi }^{2})\):

$${\chi }^{2}={\sum}_{i,j}\frac{{\left[{I}_{\exp }\left({\left|{{{\bf{Q}}}}\right|}_{i},{{{{\rm{\omega }}}}}_{j}\right)-\left(b+f{I}_{{{{\rm{calc}}}}}\left({\left|{{{\bf{Q}}}}\right|}_{i},{{{{\rm{\omega }}}}}_{j}\right)\right)\right]}^{2}}{{({\sigma }_{{ij}})}^{2}\,},$$
(4)

where i and j are indices for data points in the |Q|–ω space, Iexp(|Q|, ω) and Icalc(|Q|, ω) are the measured and calculated INS cross-sections, and \({\sigma }_{{ij}}\) is the standard deviation of \({I}_{\exp }\left({\left|{{{\bf{Q}}}}\right|}_{i},\,{{{{\rm{\omega }}}}}_{j}\right)\). f and b are scale and constant background parameters that are refined to minimize \({\chi }^{2}\) for given Iexp(|Q|, ω) and Icalc(|Q|, ω) maps, which can be uniquely determined by using Eq. 4 in ref. 59. For this \({\chi }^{2}\) analysis, we considered the four INS data shown in Fig. 3a simultaneously with different values of f and b for each effective temperature. The masked regions in Fig. 3a (white areas), except for those outside the energy-momentum coverage of time-of-flight powder neutron spectroscopy, were excluded from the analysis as they contain spurious signals or a quasi-elastic background signal.

Spin-wave and Schwinger Boson calculations

LSWT calculations were performed using the SpinW library60. NLSWT calculations were made based on 1/S corrections. The first 1/S corrections to LSWT in both the magnon energies and the dynamical spin structure factor are obtained by the standard approach described in ref. 41 and ref. 42. The only differences are that (a) the magnon energies are evaluated in the on-shell approximation and (b) the first 1/S corrections are included in both the one-magnon and the two-magnon energies (see also ref. 61). The explanation of Schwinger Boson calculations is provided in Supplementary Notes.