Introduction

Interactions between various microscopic degrees of freedom with similar energy scales can induce strong competition and frustration, leading to exotic phenomena. The f-electron materials show strong spin–orbit coupling (SOC), and therefore the spin and orbital degrees of freedom should be described by the total angular momentum J. The crystalline electric field (CEF) further splits the total angular momentum J, and the low-lying crystal field states can form rather complex ground states, including spin liquids1,2,3,4,5,6,7, spin ices8,9,10 and hidden ordered phases11,12,13. In most circumstances, the hidden-order phase transition is signaled by the change of bulk properties such as the magnetic susceptibility and heat capacity14; but unveiling its microscopic nature is difficult because the hidden-order parameter cannot be directly disclosed by microscopic probes such as neutron diffraction or muon spin rotation/relaxation11. Efforts have been made through the study of the collective excitations associated with the hidden orders15,16,17,18,19. For example, in the canonical hidden-order material URu2Si2, the antiferromagnetic spin excitations appear at both the commensurate and incommensurate wavevectors, and exhibit spin gaps in the hidden-order phase18. In the case of CeB6, complex ferromagnetic and antiferromagnetic spin excitations were observed along with a spin exciton19. However, whether or how these collective modes drive the hidden-order phase transition remains a matter of debate.

The recent discovered rare-earth magnet TmMgGaO4 may provide a new opportunity to examine the exotic ordering phenomenon of f-electrons20. This material has the same crystal structure as the spin liquid candidate YbMgGaO4, which crystallizes in the \(R\bar 3m\) space group with a quasi-two-dimensional triangular lattice1. In TmMgGaO4, the Tm3+ ion possesses an electron configuration 4f12, in which the orbital and spin angular momentum (L = 5, S = 1) are entangled into the total angular momentum J = 6 due to the strong SOC. The corresponding 13 states are further split under the D3d CEF. It was suggested that the relevant low-energy degrees of freedom in Tm3+ ion is a pair of nearly degenerate singlets that are well separated from other CEF levels, resulting in a quasi-doublet state21. This indicates the magnetic interaction of TmMgGaO4 should be different from that of YbMgGaO4 where Yb3+ ions are Kramers ions22. The thermodynamic and magnetic susceptibility measurements in TmMgGaO4 further revealed a phase transition at low temperature and suggested a conventional stripe-type dipolar magnetic order state with Ising interactions21, which is yet to be verified by microscopic measurements.

In this paper, we report neutron scattering and thermodynamic studies of the phase transitions and spin correlations of TmMgGaO4 single crystals. Our data reveal that the longitudinal component of the effective spins, Sz, presents dipolar behavior and forms a three sublattice magnetic structure, leading to a magnetic Bragg peak at the K point in the neutron diffraction spectrum. The transverse components, Sx and Sy, however, exhibit multipolar behavior and develop a ferro-multipolar order, which is mostly hidden in elastic neutron diffraction, but is revealed by the collective spin excitations in the inelastic channel. We show that the spin correlations and thermodynamics of TmMgGaO4 can be accurately described by the transverse field Ising model with an intertwined dipolar and multipolar order on the triangular lattice.

Results

Susceptibility, magnetization and heat capacity

We first measure the magnetic susceptibility of our TmMgGaO4 single crystals prepared by the floating zone technique (Methods). With an external field applied along the c axis, paramagnetic behavior is revealed at low temperatures with a Curie–Weiss temperature of −19.1 K (Fig. 1a). No clear anomaly or splitting of the zero-field-cooling (ZFC) and field-cooling (FC) data can be distinguished, suggesting the absence of phase transition above 2 K. When the magnetic field along the c axis increases, the system is driven into a nearly polarized state marked by the saturated magnetization above 5 T (Fig. 1b). The corresponding Landé-g factor for effective S = 1/2 state is around 12.11, which is close to the upper limit of 2JgJ = 14. This indicates a dominant Jz = ±6 components in the wave functions of the low-lying two singlets, leading to a strong Ising character, consistent with previous report21. In a sharp contrast, when the field is applied in the ab plane, a much weaker magnetic response is observed (Fig. 1a, b). The heat capacity measurement shows an anomaly at ∼1 K, indicating a phase transition (Fig. 1c). The integral magnetic entropy below 30 K is close to Rln2, which is expected for a system with Seff = 1/2, in agreement with the quasi-doublet CEF ground state of the Tm3+ ions in TmMgGaO4.

Fig. 1
figure 1

Thermodynamic property and neutron diffraction measurements of TmMgGaO4 single crystals. a Temperature dependence of the magnetic susceptibility χ measured under ZFC and FC conditions with external fields of 10 kOe applied parallel and perpendicular to the c axis. The inset shows the linear fitting of the inverse susceptibility. b Field dependence of the magnetization at T = 2 K. Linear fitting of the magnetization at high field indicates the Landé-g factor of 12.11(5). c Magnetic heat capacity and magnetic entropy measured under zero field. The phonon contribution is subtracted by comparing measurements of TmMgGaO4 with the non-magnetic reference compound LuMgGaO4. The magnetic entropy is obtained by integrating C/T from 0.25 K. Indication of a Schottky anomaly is observed below 0.4 K, which is likely caused by the strong hyperfine interactions. d Q-scans across the magnetic Bragg peak Q = (1/3, 1/3, 0) along the transverse direction at the indicated temperatures. e Temperature dependence of the fitted peak amplitudes of the Bragg peak at Q = (1/3, 1/3, 0). f Q-scans across the multipolar Bragg peak Q = (1, 0, 0) along the transverse direction at indicated temperatures. g Temperature dependence of the intensity of the Q = (1, 0, 0) peak. The solid and dashed lines in d–f are guides to the eye. h Momentum dependence of the magnetic Bragg peak at 0.05 K. The white dashed lines indicate the zone boundaries. i L dependence of the peak intensity at Q = (2/3, −1/3, L). The color bars indicate scattering intensity in arbitrary unit in linear scale. j Schematic of the three-sublattice magnetic structure of TmMgGaO4. Sy forms ferro-multipolar order along the y direction (black dashed lines) and Sz forms dipolar order (spin up—red, spin 0—blue, spin down—green). The red and green arrows are tilted from the xy plane by ∼32°. The data shown in d, f and g were measured on PANDA and the data in h, i were measured on LET. The wavevector Q is defined as Q = Ha* + Kb* + Lc*; r.l.u. reciprocal lattice unit, cts min−1 counts per minute; error bars, 1 s.d.

Full size image

Magnetic Bragg peaks and spin wave dispersion

To determine the microscopic ground state of TmMgGaO4, we use elastic neutron diffraction to study the magnetic ordering properties. Clear magnetic Bragg peaks are uncovered at K points at low temperatures (Fig. 1d, h), suggesting the presence of a magnetic dipolar order. For the scans along the L direction at the K point, we observe a rod of scattering (Fig. 1i), demonstrating the quasi-two-dimensional nature of the magnetic order. The peak intensity drops gradually on warming and shows a transition at ∼1 K, consistent with the heat capacity measurements (Fig. 1e).

The observation of a magnetic Bragg peak at the K point instead of the M point is in clear contradiction with the stripe ordered ground state for the Ising interactions21. To determine the nature of the K point order, we measured the spin excitations over a wide range of momentum in TmMgGaO4. Figure 2 shows the momentum dependence of the spin excitations at various energies. The constant energy images at low energies display sharp spin response at the K points (Fig. 2a). As the energy increases, the spin excitation disperses outward from the K points and then forms ring-like patterns around the Γ points at higher energies (Fig. 2b–e). Eventually, the spectra reach the band top at the Γ points and vanish above 1.7 meV (Fig. 2f). The overall dispersion can be seen more clearly in the energy dependence of the spectral intensity along the high-symmetry directions in Fig. 3a, which reveals a very sharp spin-wave spectrum. This suggests that the exchange disorder induced by the Mg/Ga site mixing in the non-magnetic layers is not significant, in stark contrast to the conjecture that the Mg/Ga site mixing will introduce strong exchange variation (ref. 23). The spin excitation dispersion is further confirmed by a series of constant-energy cuts through the high-symmetry points (Fig. 4a) and constant-Q cuts (Fig. 4b). In addition, the constant-Q cut at the K point reveals a spin-gap feature at low energies (Fig. 4b). This gap is gradually closed with increasing temperature (Fig. 4b). Meanwhile, the spectral weight is transferred to lower energies, turning into quasi-elastic excitations at higher temperatures (Fig. 4b–d).

Fig. 2
figure 2

Measured and calculated momentum dependence of the spin excitations in TmMgGaO4 at the indicated energies and T = 0.05 K. a–f Raw contour plots of the constant energy images at T = 0.05 K. The signals near Q = (0, 0, 0) in a are due to the elastic contamination from the sample environment close to the direct beam. g–l Calculated spin excitations using the model specified in the text. The dashed lines indicate the zone boundaries. The measurements were performed on LET spectrometer with Ei = 4.8 meV. The color bars indicate scattering intensity in arbitrary unit in linear scale

Full size image
Fig. 3
figure 3

Measured and calculated spin wave dispersions in TmMgGaO4 at T = 0.05 K. a Intensity of the spin-excitation spectra along the high-symmetry momentum directions as illustrated by the black solid lines in c. b The simulated spin excitation dispersion using the model specified in the text. The color bars indicate scattering intensity in arbitrary unit in linear scale. c Sketch of the reciprocal space. Black dashed lines indicate the Brillouin zone boundaries

Full size image
Fig. 4
figure 4

Constant energy cuts along the high-symmetry directions and constant Q cuts at the high-symmetry points. a Constant energy cuts along the Γ–M–K–Γ direction at the indicated energy at 0.05 K. The solid curves indicate the simulated spin excitations using the model specified in the text and the vertical dashed lines indicate the high-symmetry points. b–d Constant Q cuts at the K, M and Γ points at various temperatures. The data presented in a, c and d were measured with Ei = 4.8 meV while the data in b were measured with Ei = 1.7 meV. The gray dashed lines indicate the elastic incoherent scattering. The error bars indicate the standard deviation

Full size image

Discussion

To describe the momentum and energy dependence of the spin excitation spectra, we adopt the linear spin wave (LSW) theory. For conventional magnetic systems in which all \(S_i^\mu\) components exhibit dipolar properties, neutron scattering will probe all three spin channels, Sxx, Syy and Szz, since the neutron is scattered by the magnetic moment through dipole–dipole interactions. Correspondingly, we calculate the spin wave excitations using the SPINW program24 for the pure Ising model, XY model and Heisenberg model (Supplementary Note 2). However, none of these models with a dipolar order is consistent with our data. For example, for the XY model that supports the 120° Néel order with magnetic Bragg peaks at K points, the spin wave dispersion around Γ points should exhibit a minimum instead of a maximum that is observed in our experiments. In addition, the Ising model with a stripe-type order that has been proposed in ref. 21, however, will lead to magnetic Bragg peaks at M points and dispersionless spin excitations, clearly inconsistent with our data.

On the other hand, an intertwined multipolar and dipolar order is in excellent agreement with the experimental data. In TmMgGaO4, the large g-factor in the magnetization experiments indicates strong Ising character of the Tm3+ ions, suggesting that the wave functions are dominated by Jz = ±6 components21. Under the D3d symmetric CEF, these Jz = ±6 dominated components are not allowed to form a symmetry protected doublet. Instead, they must hybridize into two non-magnetic singlets and open an energy gap. The full singlet wave functions dictating the D3d CEF symmetry must take the following form:

$$|{\mathrm{\Psi }}_i^ + \rangle \sim c_6\left( {| + 6\rangle _i + | - 6\rangle _i} \right) + c_3\left( {| + 3\rangle _i - | - 3\rangle _i} \right) + c_0|0\rangle _i,$$
(1)
$$|{\mathrm{\Psi }}_i^ - \rangle \sim c_6^\prime \left( {| + 6\rangle _i - | - 6\rangle _i} \right) + c_3^\prime \left( {| + 3\rangle _i + | - 3\rangle _i} \right).$$
(2)

Here, these two singlets, \(|{\mathrm{\Psi }}_i^ + \rangle\) and \(|{\mathrm{\Psi }}_i^ - \rangle\), carry A1g and A2g representation of the D3d group, respectively. |jz〉 denotes for Jz = jz states at site i and c6, c3, c0, \(c_6^\prime\) and \(c_3^\prime\) are real numbers with \(|c_6| \approx |c_6^\prime | \gg c_3,c_3^\prime ,c_0\). For simplicity, in the following the site index i will be omitted. The local moment can be constructed by the following spin-1/2 operators acting on the quasi-doublet:

$$S^x = \frac{i}{2}\left( {|{\mathrm{\Psi }}^ - \rangle \langle {\mathrm{\Psi }}^ + | - |{\mathrm{\Psi }}^ + \rangle \langle {\mathrm{\Psi }}^ - |} \right),$$
(3)
$$S^y = \frac{1}{2}\left( {|{\mathrm{\Psi }}^ + \rangle \langle {\mathrm{\Psi }}^ + | - |{\mathrm{\Psi }}^ - \rangle \langle {\mathrm{\Psi }}^ - |} \right),$$
(4)
$$S^z = \frac{1}{2}\left( {|{\mathrm{\Psi }}^ + \rangle \langle {\mathrm{\Psi }}^ - | + |{\mathrm{\Psi }}^ - \rangle \langle {\mathrm{\Psi }}^ + |} \right),$$
(5)

where our definition of the spin operators here is a bit different from the conventional choice. Our choice is designed for the particular bases and the wave functions of two singlet states in TmMgGaO4. The transformation of the symmetry operators under CEF symmetry operations (threefold rotation about c axis C3 and time reversal T) take the following form:

$$C_3:S^x \to S^x,S^y \to S^y,S^z \to S^z$$
(6)
$$T:S^x \to S^x,S^y \to S^y,S^z \to - S^z.$$
(7)

We can see that the transverse components Sx and Sy are even under time-reversal and stay invariant under the threefold rotation. These unique properties indicate that the transverse components are multipoles with rank ≥3 that are not directly accessible for neutrons, nor other conventional experimental probes. Meanwhile, the longitudinal component Sz is related to dominated ΔJz = 0 process (e.g., from |+6〉 to |+6〉 or from |−6〉 to |−6〉). Therefore, it behaves as a conventional magnetic dipole along the c axis, and can directly couple to external magnetic field or neutron spins.

To keep the essential physics of this system, it is enough to consider only the Ising-type spin interactions (since the local moments have strong Ising nature) and the CEF splitting between singlets here. With the gap between singlets captured by a transverse field term along y axis \(\left( { - h\mathop {\sum}\nolimits_i {S_i^y} } \right)\) and the Ising interactions taken into account up to second nearest-neighbor, the effective Hamiltonian reads:

$${\cal{H}} = \mathop {\sum}\limits_{\langle ij\rangle } {J_1^{zz}S_i^zS_j^z} + \mathop {\sum}\limits_{\langle \langle ij\rangle \rangle } {J_2^{zz}S_i^zS_j^z} - h\mathop {\sum}\limits_i {S_i^y},$$
(8)

where 〈ij〉 and 〈〈ij〉〉 denote the nearest and next-nearest neighbors, respectively (Supplementary Fig. 2a). Despite the strong Ising nature, the CEF splitting term can create strong quantum fluctuations upon the spins. Based on this model, we re-calculate the spin wave dispersion and find a set of parameters that can accurately describe our data: \(J_1^{zz}\) = 0.54(2) meV, \(J_2^{zz}\) = 0.026(6) meV, h = 0.62(2) meV. The calculated E–k relationship shows excellent agreement with the experimental observation (Figs. 2g–l, 3b). The corresponding magnetic structure is a three-sublattice structure in which the hidden components form a preformed ferro-multipolar order along the y direction in the effective spin space due to the polarization effect from the transverse field while the out-of-plane dipole moments order antiferromagnetically at K points (Fig. 1j). The out-of-plane dipolar order (spin-up, spin-down, spin-0 in the three sublattices, respectively) shows no macroscopic magnetization and is indicated by our neutron diffraction measurements that reveal a magnetic Bragg peak at the K point. The multipoles, however, do not linearly couple to neutrons and therefore are hidden in the neutron diffraction measurements.

Although these multipolar components are not directly visible, the elementary excitations of the multipolar components can be accessible in inelastic neutron experiments. As the multipolar components do not commute with the dipolar ones, measuring the Sz moment will induce spin-flipping events on the multipolar components, leading to coherent multipolar spin wave excitations25,26. As a result, the LSW theory remains a valid description for the excitations, but only the longitudinal Szz channel is involved.

Because multipoles have a complex structure with no spatially uniform magnetization, the multipolar spin wave may display an anisotropic and non-monotonic Q-dependence of the magnetic form factor, which is different from a conventional dipole spin wave11,27. Indeed, Fig. 2 shows that the spin excitations in the second Brillouin zone are not weaker than those of the first Brillouin zone, in stark contrast to a conventional dipolar form factor of Tm3+ that decreases with increasing Q. This further confirms the multipolar nature of the spin excitations (Supplementary Note 3). In addition, because the multipolar moments do not couple directly to the external field, the in-plane magnetization should be invisible to the conventional magnetic probe in the intertwined dipolar and ferro-multipolar order state in TmMgGaO4. This is indeed the case as is illustrated in Fig. 1a, b: little magnetic susceptibility is observed when the external magnetic field is applied in the ab plane. We note that the residual tiny in-plane signal observed could be either due to the imperfection of the crystal alignment for the measurements or due to a magnetic response from higher-order Van–Vleck like process.

More insight into the nature of the hidden-order and local moment structures of f-electron materials can be obtained by comparing TmMgGaO4 with other hidden-order materials. In URu2Si2, neutron scattering experiments suggest that both the commensurate and incommensurate spin excitations only display longitudinal components28. This is in analogy to our data where the transverse excitations are absent due to the multipolar behavior. Moreover, although the hidden-order parameters are in general invisible to standard neutron diffraction experiments, weak Bragg peaks may appear in certain materials presumably owing to the higher-order interactions between neutron spins and the hidden-order parameter, which is the case for Ce1−xLaxB6 and URu2Si2 (refs. 17,29). We therefore carefully performed neutron diffraction survey near the ferro-multipolar order wavevector Γ in TmMgGaO4. Indeed, a very weak Bragg peak at Γ is revealed at 0.485 K. The peak is about three orders of magnitude weaker than the magnetic Bragg peak at the K point, implying that it is indeed possible due to a higher-order interaction (Fig. 1f). The peak weakens with increasing temperature in a similar manner to the magnetic Bragg peak at the K point (Fig. 1g). This is consistent with the presence of an intertwined ferro-multipolar and dipolar order in this compound.

The similar properties between insulating TmMgGaO4 and metallic heavy-fermion hidden-order materials could have important implications. The multipolar behavior in f-electron systems mainly arises from the orbital degrees of freedom in the presence of strong SOC and CEF. In insulating TmMgGaO4, the degeneracy of the ground state is lifted by the CEF effect, leading to a preformed ferro-multipolar order. On the other hand, in the case of heavy fermion systems such as CeB6, the degeneracy of the ground state quartet is likely lifted by the antiferromagnetic interactions between the localized 4f electrons that are possibly mediated by the itinerant conduction electrons. In this sense, the multipolar order can be viewed as an orbital order due to the lifting of the degenerate ground states with or without itinerant electrons. This implies that similar approach and analysis used here could potentially apply in other heavy-fermion hidden-order materials.

Methods

Sample synthesis

Polycrystalline TmMgGaO4 samples were synthesized through the solid-state-reaction method. The starting materials Tm2O3, MgO and Ga2O3 were mixed in stoichiometric quantities and heated in ceramic crucibles at 1500 °C for 4 days in air. The polycrystalline samples were then loaded in rubber tubes and compressed into rods hydrostatically at 300 MPa, which were further sintered at 1500 °C for 8 hs. The resulting feed rods were then transferred into an optical floating zone furnace for the crystal growth. Counter-rotation of the feed rod (24 rpm) and crystal (29 rpm) has been applied during the growth. Large and high-quality TmMgGaO4 single crystals up to 5 cm in length can be obtained with a growth rate of 0.6 mm/h (Supplementary Note 1).

Neutron scattering experiments

The neutron scattering experiments were carried out on the cold three axes spectrometer PANDA at the Heinz Maier-Leibnitz Zentrum, Garching, Germany, and cold neutron multi-chopper spectrometer LET at the Rutherford Appleton Laboratory, Didcot, UK. For the PANDA experiment, we used a vertical focused PG(002) as a monochromator and analyser; the final neutron energy was fixed at Ef = 4.06 meV, resulting in an energy resolution of around 0.1 meV. A Be filter is placed after the sample to avoid the contamination from neutrons with higher orders. One piece of single crystal (3 g) was aligned in the (HK0) plane for the measurements. A closed-cycle refrigerator equipped with a 3He insert was used to reach the base temperature of 485 mK. For the LET time-of-flight neutron scattering experiment, we chose the incident energies of 11.0, 4.8, 2.7 and 1.7 meV with energy resolutions of 0.343, 0.114, 0.047 and 0.027 meV, respectively. A dilution insert for the standard 4He cryostat was used to reach the base temperature of 50 mK. Six pieces of single crystals with a total mass of 17.2 g were co-aligned in the (HK0) plane for the measurements. The data were analyzed using the Horace–Matlab suite30.

All neutron data are presented without background subtraction or symmetrizing. With background subtraction or symmetrizing, the data do not show significant change.