Introduction

Quantum materials with exotic spin liquid ground state properties arose a lot of interest due to their potential in both fundamental science and application in “topological" quantum computing devices1,2. Especially Mott-Hubbard insulating frustrated magnets with strong spin-orbit coupling and effective jeff = 1/2 states have been intensively studied as they are believed to be prime candidates realizing the physics of the exactly solvable Kitaev model on a honeycomb lattice3,4,5,6,7,8. The ground state in this exactly solvable model is a spin liquid state, i.e., a highly entangled topological state of matter without long-range magnetic order where spin-flip excitations fractionalize into itinerant Majorana fermions and emergent gauge fields, which are believed to play a key role in fault-tolerant quantum computing9,10. Especially, the trihalide α-RuCl3 has attracted immense attention as the prime candidate to show Kitaev spin liquid physics since several experimental studies indicated fingerprints of dominant Kitaev interactions in this Mott-Hubbard insulating magnet11,12,13,14,15,16,17,18,19,20,21,22,23,24,25, which was supported by multiple theoretical calculations26,27,28,29,30,31. However, despite the finite Kitaev interactions K, α-RuCl3 establishes a long-range antiferromagnetic zigzag (ZZ) order at low temperatures, indicating the presence of significant non-Kitaev interactions, such as isotropic Heisenberg J or symmetric off-diagonal interactions Γ32,33,34,35.

To date, several experimental techniques have been applied to map out the equilibrium phase diagram of α-RuCl3 in the temperature and magnetic field plane33,36,37,38,39,40,41,42,43,44,45. Signatures of fractionalized excitations have been detected by various spectroscopy techniques11,15,16,46,47, hinting towards a proximate spin–liquid behavior. Initially this has lead to a wide spreading in reported values for the possible interaction strengths and a controversial discussion on the effective spin Hamiltonian capturing the experimental observations48. Nowadays, the parameter space of the effective spin Hamiltonian and the size and sign of the present exchange interactions converge towards a unifying description of α-RuCl3 physical properties. Especially the role of the symmetric off-diagonal exchange interaction Γ has been studied intensively and Sears et al. reported recently that its size is comparable to the anisotropic ferromagnetic Kitaev K exchange interaction22, indicating its key role in understanding α-RuCl3 large anisotropic susceptibilities for magnetic fields applied within χ∥ and perpendicular χ⊥ to the honeycomb planes32,39,49. Despite this, also the orientation of a magnetic field applied within the honeycomb planes has been found to be crucial to resolve strongly angle-dependent low-energy excitations revealing fingerprints of a potential QSL state in α-RuCl319,50,51. Different experimental studies reported that α-RuCl3 undergoes several phase transitions at low temperatures and small finite applied fields before entering the quantum-disordered phase44,52,53,54,55,56, and the appearance of metamagnetic transitions has been predicted theoretically31,57. The emergence of an intermediate field-induced transition at around 6.2 T, depending on the orientation of the external in-plane magnetic field either along or perpendicular to the Ru–Ru bonds44,54,55,56 has been reported and points towards the necessity to include anisotropic inter-layer exchange interactions in the model Hamiltonian. In contrast, the low-field response for in-plane fields up to ≈2 T is still only partially understood, since the precise knowledge about the orientation of the order parameter is difficult to attain with thermodynamic probes. Therefore, its nature has been interpreted differently15,33,39,52,58, while there are first experimental signatures revealing the necessity to consider the present bond-anisotropy stemming from a small inequivalence in the Ru–Ru bond length in α-RuCl359. Nevertheless, the detailed magnetic-temperature (B, T) phase diagram of α-RuCl3 is still under intense debate and an experimental probe that couples sensitively to the ZZ order parameter needed to be utilized.

Magneto-optical (MO) spectroscopy is a well-established and powerful contactless technique to explore magnetic ordering phenomena and the emergence of topological magnetic structures on small scales with remarkable sensitivity60,61,62,63,64,65,66,67,68,69,70,71. Especially the quadratic MO effects are perfectly suited to study the antiferromagnetic ordering vector (L = M↑− M↓ ≠ 0) and recently, direct optical probing of ZZ antiferromagnetic order via optical spectroscopy has been reported72. The main quadratic MO effects (even in M) in reflection are named magnetic linear dichroism (MLD) and magnetic linear birefringence (MLB)62,73,74, which are defined for the reflection of linearly polarized light under normal incidence and depend on the difference in the diagonal components of the dielectric tensor67. In this context, the origin of MLD and MLB can be understood in terms of different absorption (reflection) coefficients parallel and perpendicular to the magnetization M or Néel vector L. These effects manifest themselves in the polarization rotation θ (MLD) of a linearly polarized light upon reflection from the sample or give rise to an elliptical polarization η (MLB). They stem from the spin-orbit and anisotropic exchange interactions and can be related to spin–spin correlation functions62,75,76,77. For symmetry reasons, it follows that the considered second-order MO effects are to lowest order quadratic in the antiferromagnetic order parameter such that the scaling (θ, η) ∝ L2 holds66 (Supplementary Information Notes 8, 9 and 10).

Motivated by these intriguing questions, we performed a systematic MO spectroscopy study to track the evolution of the ZZ order parameter in α-RuCl3 in thermodynamic equilibrium. The orientation-dependent MO response reveals the effect of the present magnetic anisotropy for magnetic fields applied perpendicular and parallel to the Ru–Ru bonds within the honeycomb layers. We show that the remarkable sensitivity of MO spectroscopy helps to clarify the emergence of two different intermediate field-induced and orientation-dependent metamagnetic transitions. Our results provide a detailed picture of the low-field behavior clarifying the influence of unidirectional bond-anisotropy within the honeycomb planes and we derive a value for the anisotropy field strength.

Results

Magnetic linear dichroism

First, we explore the nature of the detected MO signal and its relation to the magnetic order. Here, we apply the Voigt geometry67, which is typically used to study antiferromagnets with spin alignments that are perpendicular to the light wave vector. Below, we derive the relation of the experimentally observed rotation of the polarization plane of the reflected linearly polarized light, but the same consideration applies to the change in the ellipticity. The rotation θ is related to both the amplitude of the order parameter and the relative orientation of the Néel vector to the electric field E of the incident linearly polarized light. In α-RuCl3 the in-plane component of the Néel vector is oriented parallel to the ZZ chain direction. Figure 1a depicts the relative angles of the polarization of incident light and the Néel vector to the vertical polarization by θ0 and ϕ, respectively. The MLD response is then given by63,65

$$\tan ({\theta }_{{{\mbox{MLD}}}})=\frac{\left({r}_{| | }-{r}_{\perp }\right)\tan \left(\phi -{\theta }_{0}\,\right)}{{r}_{| | }+{r}_{\perp }{\tan }^{2}\left(\phi -{\theta }_{0}\,\right)}.$$
(1)
Fig. 1: Magneto-optical experiment.
figure 1

a Experimental setup (HW half-wave plate, P polarizer, C chopper, M beam-splitting mirror, L lens, S sample, PEM photo-elastic modulator, A analyzer, PD photo-diode). Ei and Er correspond to the incident electric field polarization and the one upon reflection from the sample, respectively. The inset displays the effect of MO rotation Θ = θ + iη and the definition of the angles θ0, ϕ, θ. b Polarization dependence of the MO response as a function of the polarization orientation θ0 of the incident light. Gray squares are data, solid line is a fit according to Equation (2). c Universal temperature dependence of the MO response for three different incident polarization orientations of θ0 = 0∘, 45∘, 90∘.

Full size image

More general and taking the presence of both linear and second-order MO effects into account, a total rotation of the polarization θ, in the limit of small rotations, can be expressed as

$$\theta ={{{{\mathcal{A}}}}}^{\mathrm{Lin}}+{{{{\mathcal{A}}}}}^{\mathrm{MLD}}\sin (2(\phi -{\theta }_{0})).$$
(2)

The coefficients \({{{{\mathcal{A}}}}}^{\mathrm{Lin}}\) and \({{{{\mathcal{A}}}}}^{\mathrm{MLD}}\) are the amplitude of linear and quadratic MO effects, respectively. Since in the MLD geometry, the incident light is normal to the surface, the \({{{{\mathcal{A}}}}}^{\mathrm{Lin}}\) is mainly determined by the polar MO Kerr effect (PMOKE). \({{{{\mathcal{A}}}}}^{\mathrm{MLD}}=\frac{1}{2}\left(\frac{{r}_{\parallel }}{{r}_{\perp }}-1\,\right)\), where r∥ and r⊥ are the amplitude reflection coefficients of the light polarized parallel and perpendicular to the Néel vector, depends quadratically on the in-plane component of the antiferromagnetic order parameter L, i.e. \({{{{\mathcal{A}}}}}^{\mathrm{MLD}} \sim {L}^{2}\)66 (Supplementary Information Note 10). It is worth noting that the above relation indicates that the MLD signal, in contrast with the linear MO response, is a harmonic function of the incident polarization θ0, which becomes maximal for ϕ − θ0 = 45∘ and it indicates an extreme sensitivity of the second-order MO response to the orientation of the incident linearly polarized light with respect to the spin-pointing direction. Recently it has been reported that the presence of antiferromagnetic ZZ chains can give rise to a polarization-dependent MO response, which is independent of the spin-pointing direction72. In α-RuCl3 both, the ZZ chain direction and the spin-pointing direction are collinear in small magnetic fields, such that the spin-pointing and ZZ chain direction cause the MLD. Clearly, the MO response shown in Fig. 1b is polarization-dependent, which is manifested in a sinusoidal modulation as expected for MLD. At the same time, the MO response scales with \({{{{\mathcal{A}}}}}^{\mathrm{MLD}}\propto {L}^{2}(B,T)\). Figure 1c shows the temperature dependence of the ZZ order parameter studied for three different polarization orientations of the incident light, θ0. Clearly, the magnitude of θ depends on θ0 and for θ0 = 45∘ becomes zero, while the qualitative temperature-dependent behavior remains, as expected, similar. In subsequent measurements, we chose the s-polarized probe to obtain the maximum signal.

Figure 2a shows the variations of θMLD as a function of temperature. According to ref. 11, the temperature dependence of the ZZ order parameter in zero magnetic field L(T, B = 0T) follows a power-law \({\theta }_{{{\mbox{MLD}}}}\propto {L}^{2}\propto {(1-T/{T}_{N})}^{2\beta }\) below the Néel temperature TN. The deduced Néel temperature TN = (7.19 ± 0.14) K indicates ABC-stacking in the samples under study as opposed to ABAB-stacking accompanied by stacking faults, which causes further transitions above TN56. Within the experimental uncertainty, the data can be fitted using a critical exponent β = 0.19 ± 0.07, which is close to the 2D Ising universality class78. Figure 2(b) displays the phase diagram in the (B, T) plane extracted from the MLD induced rotation at different fixed magnetic fields (iso-magnetic). We find the evolution of the order parameter L(T, B) to scale proportional to \({(H-{H}_{c})}^{\gamma }\), where Hc corresponds to the critical magnetic field above which the antiferromagnetic ZZ order is suppressed and the system enters the magnetically disordered state. The power-law fit exhibits a critical field value of Hc = (7.48 ± 0.3) T and γ = 0.31 ± 0.07, which is again close to the theoretical value of 0.32 for the 2D Ising symmetry class33. These findings support that the MO response clearly displays the evolution of the order parameter. More details on the fitting of the critical behavior can be found in the SI.

Fig. 2: Phase diagram of α-RuCl3 and temperature-dependent magneto-optical probe of the order parameter evolution in magnetic field.
figure 2

a Evolution of the rotation θMLD as a function of the temperature in zero field (black dots). The blue line is a power law fit. b In-plane magnetic field and temperature phase diagram of α-RuCl3 constructed from the rotation θMLD obtained by iso-magnetic MLD measurements. The Néel temperature extracted from the center of a Gaussian fit to the derivative of the rotation [(dθ/dT)/T]B is shown as white circles. Error bars show the standard error of the center of the Gaussian. The color code displays the amplitude of the rotation from 0 to 70 mdeg. The vertical error bars in (a) give an upper bound of 5 mdeg stemming from the long-time average laser fluctuation and horizontal error bars indicate the experimental temperature stability within 0.1 K, respectively. c (1) Bond and out-of-plane spin orientation. The spins enclose an angle of ~32∘ with the ab-plane. (2) Definition of the field direction along different crystallographic orientations in the honeycomb ab-plane. Differently colored bonds indicate the x, y, and z-bonds of the Kitaev Hamiltonian3, with interactions J, K, Γ. (3) Sketch of three possible zigzag domains projected to the 2D ab-plane related by a 120∘ rotation of the ordering Néel vector L. Yellow bond is the elongated bond compared to the blue ones. (4) Sketch of the in-plane field angle-dependent spin-flop transition.

Full size image

Magnetic field orientation-dependent measurements

Figure 2c displays the honeycomb structure, bond-anisotropy, and spin orientation in α-RuCl3. We point out the breaking of C3 symmetry in α-RuCl3 crystals originating from inequivalent Ru–Ru bond lengths, which leads to the existing monoclinic C2/m space group34,35. Phenomenologically, the inequivalence in the Ru–Ru bond lengths causes a change in the present interactions along the stretched bond indicated by \(J^{\prime} ,K^{\prime}\) and \({{\Gamma }}^{\prime}\) (cf. Fig. 2c). This causes the pseudospins, which are tilted by an angle of ≈32∘ out of the honeycomb plane22 to have their in-plane projection being preferentially oriented perpendicular to the stretched bonds. This is a key point that needs to be taken into account to understand the anisotropic MO response in the following results. Nevertheless, the local C2/m symmetry has been found to be broken in multi-domain samples52, due to a randomness in the monoclinic distortion of one Ru–Ru bond. Consequently, there can be three possible and symmetry-allowed ZZ domains in zero field cooled samples below TN, which are related by 120∘ rotations within the plane. However, applying a finite magnetic field along specific crystallographic orientations can change the spin-pointing direction within the honeycomb planes as will be discussed in the following and is schematically depicted in Fig. 2c via a spin-flop process.

Having established the iso-magnetic response of the order parameter L encoded in the MLD response θMLD we turn now to the iso-thermal MLD response of α-RuCl3 at a temperature of 3 K and magnetic field strengths up to ±7 T applied within the honeycomb planes along two different crystallographic directions to investigate the magnetic in-plane anisotropy. We studied two samples from the same batch which have been oriented along different crystallographic directions. Once, for sample (a) the in-plane magnetic Bab field is applied perpendicular to a Ru–Ru bond, i.e., along one of the symmetry-equivalent {1, 1, 0} directions, while the field was directed parallel to the Ru–Ru bonds for sample (b), i.e., along one of the symmetry-equivalent {1, 0, 0} directions (Fig. 2c). The MO measurements were conducted with the magnetic field swept continuously from 0 to ±7 T to systematically track the dynamical MO response (Supplementary Information Notes 2 and 3).

Figure 3a shows that the MO response θMLD differs significantly for two distinct orientations of an in-plane magnetic field Bab, a first experimental evidence for the present in-plane magnetic anisotropy. The purely magnetic origin for this anisotropy is verified by the fact that the temperature-dependent evolution of θMLD and η in zero magnetic field is similar for both samples ruling out possible temperature-related effects like strain or thermoelastic changes (Supplementary Information Note 2). The absolute field-induced change in the rotation defined as Δθ(B) = θMLD(0T) − θMLD(±7 T) for both field configurations at 3 K is in the order of 100 mdeg, which is large and underlines the microscopic impact of strong spin-orbit interactions (~100 meV in α-RuCl379) on the second-order MO responses. The value at 7 T was set to zero to extract the field-induced changes in the rotation between 0 and 7 T at constant temperature (Supplementary Information Note 3). Despite the difference in the full hysteresis loops for the different field orientations, we divide the MO response shown in Fig. 3a into three field regimes to allow a simple description and comparison. In each regime, the MO signal is comparable, while at the crossovers a vivid anisotropic response is present. The first transition at ±1.6 T is a large and steep change in the rotation θMLD (absolute reduction of ≈50 mdeg) for the magnetic field applied perpendicular to the Ru–Ru bonds, whereas the rotation changes only slightly for the in-plane magnetic field applied parallel to the bonds. The marked difference for the two differently oriented samples in regimes I and II are displayed in Fig. 3d in more detail. This clear difference indicates the presence of a metamagnetic transition as will be discussed later. The second transition in the response occurs at around 6.2 T for the field applied perpendicular to the Ru–Ru bonds where an anomaly in θMLD is observed by a kink. Reducing the magnetic field strength continuously from ±7 T back towards 0 T causes the emergence of significant hysteresis in the MLD response opening at a field strength of ≈6 T. The hysteresis loops close at a field strength of ≈0.7 T for both field orientations, but open again for small fields close to 0 T. Here, the integrated hysteresis weight is a factor of ≈1.5 larger for the field applied perpendicular to the bonds than along the bonds pointing again towards a difference in the in-plane anisotropy energy.

Fig. 3: Magnetic linear dichroism of α-RuCl3 for different sample orientations.
figure 3

a Full magnetic field sweep scans of the magneto-optical response for Bab∥{1, 1, 0} and Bab∥{1, 0, 0} at 3 K. The hysteresis loops can be divided into three field regimes I, II, and III, which are discussed in the main text. b Temperature dependence of the MLD response as the magnetic field is aligned along the {1, 1, 0} direction. Labels are discussed in the main text. c Linear MO response extracted from the MLD response for the two different field orientations. The light gray area indicates field-regime I. d Zoom-in into field regime I and II for two subsequent field sweeps, which are divided into four steps as discussed in the main text. Inset shows the zero field time-dependent recovery of MO signal after the field sweep. e High-field regime III. The kink in the rotation for Bab∥{1, 1, 0} highlighted by the dashed line is connected to the transition towards the new AFM phase. The shaded area indicates the emergence of the intermediate magnetic transition.

Full size image

Figure 3b reports the iso-thermal MO responses as a function of the applied magnetic field. We found a similar behavior showing the three regimes I–III for three bath temperatures of 3, 4.5, and 6 K, corresponding to locations in the phase diagram deep insight the ZZ phase, in an intermediate range and close to the critical transition temperature towards the quantum paramagnetic phase. This finding indicates the purely magnetic nature of the MO field response.

In contrast, the odd MO response scales linearly in the applied magnetic field for both samples and shows no pronounced hysteresis for a whole field sweep (Fig. 3c). However, in regime I small spikes can be observed for both samples.

In order to understand the nature of these spikes, MOKE measurements with the sample rotated by 45∘ w.r.t. the incident light wave vector have been performed. We find a clear featureless linear scaling of the obtained rotation with the applied magnetic field (Supplementary Information Note 6). Therefore, we assume that the small spikes are related to second-order MO effects. Since under the real experimental condition, the perfect symmetry cannot be achieved, minor differences in MLD obtained from negative and positive fields are unavoidable.

Low-field magneto-optical response

Figure 3d compares the initial part of the hysteresis in regime I for both crystallographic orientations for subsequent field sweeps from 0 to 7 T. To emphasize similarities, we consider the normalized MO response. Here, we can divide the field-dependent alternation of MLD into the following four distinct steps.

  1. (i)

    The first step shows the following characteristics. First, it displays an initial big, but a gradual and monotonic change in θMLD, which starts immediately when applying a small in-plane magnetic field. Second, it is seemingly independent on the field orientation in the honeycomb plane and terminates at a field strength of ~0.7 T. Third, the absolute amplitude of this first step decreases for subsequent field sweeps, but displays a similar field dependence for both field orientations. Fourth, the MO response does not recover immediately when sweeping the field back to 0 T. However, there is a slow recovery of the MO response within tens of seconds at zero field to a saturating value (see inset 3d). The above mentioned distinct features of step 1, as we explain later, can reasonably be assigned to an initial field-driven gradual domain repopulation or ZZ domain switching.

  2. (ii)

    In step 2, no further change in the magneto-optical response can be observed for any of the subsequent field scans between 0.7 and 1.6 T with distinct in-plane field orientations. This indicates that in step 2 neither the ZZ chain orientation nor the spin-pointing direction and the amplitude of L are changed by the external magnetic field. This observation shows that at around 0.7–1.6 T the magnetic field changes between the three degenerate domains are terminated.

  3. (iii)

    A further increase in the applied field strength leads then to a sharp change (step 3) at ≈1.6 T for the in-plane field aligned perpendicular to the bond. In contrast, this abrupt change is not present for the field directed along the bond. This is a clear indication for a field-induced metamagnetic transition, which we will discuss later in terms of a spin-flop transition originating from the intra-layer bond-anisotropy of the frustrated honeycomb magnet. This anisotropic change in rotation at a field of ≈1.6 T is, in contrast to step 1, completely reproducible in subsequent field sweeps. Further, step 3 is clearly observed and remains sharp at all temperatures demonstrating the field evolution of the MO response is reproducible in the entire ZZ phase (cf. Fig. 3b).

  4. (iv)

    Further strengthen the applied field leads to the emergence of another steady state of the obtained signal from 2 to 3 T (step 4) pointing towards a homogeneously field-aligned ZZ ordering. Here, similar to step 2, neither the ZZ chain nor the spin-pointing direction change. Moreover, the different steps can be clearly observed in subsequent measurements at different temperatures as shown in Fig. 3b.

These different observations confirm the magnetic origin of the field response and provide first important key information to distinguish both the effects of field- and anisotropy-related domain selection and an accompanied spin-flopped phase for in-plane applied magnetic fields. In the first step, the in-plane field leads to an immediate population of ZZ domains with the easy ZZ chain axis closer to the field direction44,52,77,80. This interpretation is further supported by a recent study of the thermal and magneto-elastic properties of α-RuCl3 for field applied in the ab-plane. It was found, that under an in-plane magnetic field α-RuCl3 shows lattice contraction along the {1, 1, 0} direction41. Especially in low fields up to ≈0.5 T the magnetostriction coefficient changes continuously, such that the initial ZZ domain structure and distribution will be gradually changed by the external magnetic field. The fact that the MO hysteresis does not close while reducing the in-plane field back to 0 T points towards irreversible processes. This fits to the initial domain repopulation picture and hysteric magnetocaloric measurements below TN41.

High-field magneto-optical response

Turning to the high-field regime III, the kink which is observed only for one direction (Fig. 3e) indicates clear dependence on the orientation of Bab with respect to the pseudospin-bonds. This kink in the rotation is reproducible during different measurement cycles and likely related to the previously reported first-order transition into an intermediate differently ordered ZZ phase for in-plane magnetic fields of ≈6−6.5 T aligned perpendicular to the Ru–Ru bond44. We observed this kink only at low temperatures since its intensity dramatically decreases by temperature (Fig. 6c) in agreement with the recent reports44,81. We point out that signatures of this metamagnetic transition seem much weaker compared to the spin-flop transition at a field of ≈1.6 T, which is related to the already suppressed amplitude of L close to the critical line and the fact that it might originate from a competition of anisotropic inter-layer exchange interactions as opposed to a change in the in-plane magnetic ordering44.

Low-field anomaly

In the following, we present the results of two individual experiments which further support the observed anisotropic MLD response at ≈1.6 T. Figure 4a shows the temperature dependence of the iso-magnetic MLB η. Since the birefringence originates from spin–spin correlations contributing to the magnetic energy, its derivative scales proportional to the magnetic part of the specific heat62,82,83. The obtained curves for [(dη/dT)/T]B are shown in the inset of Fig. 4a, which resolve changes in η more clearly and display a typical λ-shape as has been reported previously for the magnetic part of the specific heat at the transition for α-RuCl320,84. The red arrows indicate the peak values of these derivatives, which coincide with the vanishing MO rotation reported in Fig. 2b. As expected, the power-law fit \({(H-{H}_{c})}^{{\gamma }_{\eta }}\) gives similar values Hc = (7.21 ± 0.09) T and γη = 0.29 ± 0.04 as for the MLD response. Indications of the transition at 1.6 T are also visible in the temperature-dependence of η, where a small kink in the curves is indicated by a dashed line in Fig. 4b for all three temperatures.

Fig. 4: Magnetic linear birefringence of α-RuCl3.
figure 4

a Iso-magnetic MLB as a function of the temperature. Data are plotted offset for clarity. The inset shows the calculated first derivative of ellipticity (dη/dT) normalized to the bath temperature. Red arrows indicate the maximum of the derivative indicating the magnetic phase transition, which is indicated by the gray dashed arrow in the main figure. b MLB η for Bab∥{1, 1, 0} as a function of the in-plane magnetic field at different temperatures. The arrows indicate the anomalies discussed in the main text.

Full size image

Furthermore, we performed in-plane magnetometry measurements at 3 K (Fig. 5a) using a superconducting quantum interference device (SQUID, Quantum Design, MPMS). The obtained magnetization curves for both field orientations as in the MO experiments cross through the zero point, as expected for a compensated antiferromagnet, excluding any finite ferromagnetic contribution or background signal. Although at first glance, the magnetization curves M(H) do not exhibit any clear signs of a transition at 1.6 T, the standard deviation of the magnetization measurements illustrated in Fig. 5b give an indication (Supplementary Information Note 4). The spin-flop transition is of first-order, i.e., close to the critical field value both the initial ZZ-oriented domains and some already spin-flopped domains coexist, such that the system is driven into a regime of large fluctuations stemming from the competition of anisotropy-stabilized ZZ and already field-driven spin-flopped domains. These fluctuations display the instability of the coexistence of energetically different ZZ phases near the critical spin-flop field strength, which leads to discontinuous jumps in the magnetization accompanied by irreversible behavior. The instability is clearly visible in the standard deviation of the magnetization measurements illustrated in Fig. 5b. This effect has been observed previously in susceptibility measurements at the phase boundary between the antiferromagnetic and spin-flopped phase in the hexagonal antiferromagnet NiO accompanied by an initial field-induced domain alignment85.

Fig. 5: Magnetization measurements.
figure 5

a Magnetization measurement for both field orientations within the ab-plane. b Standard deviation of the magnetization measurements. Details are discussed in the main text.

Full size image

Discussion

Based on these independent and consistent experimental observations, we elaborate on our main findings of the response to the magnetic field orientation along the two different crystallographic orientations and discuss the low-field response in detail. In zero field, the sample comprises three possible ZZ domains in which the spin direction differs by 120∘ with statistically distributed unequal but likely comparable populations. A small external applied field within the ab-plane will then favor ZZ domain(s) for which the Néel vector is most nearly perpendicular to Bab in order to maximize the susceptibility. Such a scenario has been discussed for a similar threefold degenerate domain structure in the uniaxial antiferromagnet NiO85. This is perfectly in line with the initial changes in the MO response indicative of the metastability of the domain population.

During the process of field-induced domain repopulation the volume of the three distinct ZZ chains changes, which immediately is reflected in changes of θMLD72. However, if this domain repopulation would be the only mechanism causing the ZZ degeneracy to be lifted, a further increase of the magnetic field strength should then only affect the amplitude of the order parameter L, while its orientation should be unchanged for small enough field strengths. It follows that no additional abrupt and anisotropic changes in θMLD would be expected for just a gradual repopulation of the ZZ domains. In this regard, the key information provided by the MLD response is encoded in steps 2 and 3. In step 2 the system is approaching a steady-state in an interval of the applied field from ≈0.7 to 1.6 T, that is the initial and field-orientation independent domain reorientation is terminated. Hence, the clear difference at 1.6 T for the different orientations of the magnetic field gives experimental evidence that it has to be caused by a different mechanism than the already terminated domain repopulation. At 1.6 T an abrupt change for the field perpendicular to the bond appears, whereas no clear change is observable for the field along the bond. Hence, there is seemingly a competition between the external applied field and the anisotropic intra-layer interactions stabilizing the orientation of the Néel vector in some domains until the external field overcomes the anisotropy energy. This competition can be modeled based on a modified spin-flop theory and allows to derive a value for the anisotropy constant Ka (Supplementary Information Note 5). We start with the free energy of the system at the ground state, which gives information about the spin-pointing direction and can be generally expressed as77,86,87

$$F=-\frac{1}{2}{\mu }_{0}\left({\chi }_{+}{\sin }^{2}(\alpha )+{\chi }_{-}{\cos }^{2}(\alpha )\right){H}^{2}+{K}_{a}{\sin }^{2}(\alpha -{{\Psi }}).$$
(3)

Here μ0 is the vacuum permeability, χ+ and χ− correspond to the extrema of the in-plane oscillating susceptibility χ∥(ϕ) of α-RuCl3, where χ+ and χ− occur for the magnetic field applied parallel or perpendicular to one of the Ru–Ru bond directions, respectively59. The values of χ+ and χ− at a temperature of 2 K have been reported in ref. 59. The angle of the pseudospins and the magnetic field is parametrized by α and Ψ is the angle between the applied field and the magnetic easy axis, which is perpendicular to one of the stretched Ru–Ru bonds. Equation (3) was numerically evaluated for every single ZZ domain and field orientation to find the angle α at fixed Ψ, which minimizes the energy for each field value H. For this, α was varied between 0 and π for each field value H. For the field oriented along the stretched bond, α = π/2 does not change for increasing field. For anisotropy-stabilized domains at a finite angle with the applied field, we find a continuous change in α(H), which converges towards α = π/2 for increasing field strength. Only for the field applied perpendicular to the stretched bond there is a discontinuity in α(H). In addition, we consider the initial field-induced changes in the ZZ domain volumes on a phenomenological level similar to reported in ref. 77. More information regarding the modeling of the MO response can be found in Supplementary Information Note 5.

The model satisfactorily reproduces the field evolution of the MO response for both in-plane field directions (Fig. 6d). Further, the anisotropy constant Ka can be estimated from the experimentally determined spin-flop field Hsf = 1.6 T applied within the ab-plane

$${K}_{a}=\frac{1}{2}{\mu }_{0}({\chi }_{+}-{\chi }_{-}){H}_{\,{{\mbox{sf}}}\,}^{2}.$$
(4)

The derived value for the effective anisotropy field Ka/μ0 is 0.0027 T. The interpretation of a spin-flop transition is supported by recently reported in-plane susceptibility measurements, where a two-fold oscillation has been observed for fields below 2 T pointing towards the intrinsic in-plane bond-anisotropy with a π-periodicity as opposed to a characteristic sixfold oscillation emerging for larger field strengths indicating the field-induced reorientation of the order parameter and hence spin-pointing direction in α-RuCl344,59. However, this produces only little changes in the magnetization (cf. Fig. 5a), although the orientation of L changes remarkably. This is a reasonable explanation why in previously reported magnetization data, which lack the sensitivity to small changes in L, a change in the low-field susceptibility has been overlooked, although some changes at around 1.5 T in dM/dB have been found previously (39, SI of ref. 43).

Fig. 6: Low-field and high-field anomalies in the magneto-optical response of α-RuCl3.
figure 6

a, b Schematic sketch of the temperature-field phase diagram of α-RuCl3 for two directions, {1, 1, 0} (Bab perpendicular to the Ru–Ru bond) and {1, 0, 0} (Bab along the bond). The three different observed regimes I (ZZ), II (SF), and III (ZZ2) are displayed. In regime I, three initial domains are present. More details on each step (1–4) can be found in the main text. After step 2, the most populated domain is depicted for each field direction. For the fields larger than Bsf, the similar spin orientation is shown. Black square and circle symbols are derived from MLD experiments and show the metamagnetic transition fields for different temperatures. White square and circle symbols show the boundary with QPM phase obtained by MLD and MLB experiments, respectively. c High-field anomalies of dθ/dB at different temperatures. The inset shows the jump in the MLD data at 6.2 T. d The proposed model simulates domain population and the spin-flop transition.

Full size image

Figure 6a, b summarizes the experimental results. Independent of the in-plane field orientation there is an initial change in the ZZ domain population in regime I. According to the direction of the applied field, some of the initial domains become more populated at the expense of the others, but the general evolution of the rotation θMLD in small fields is similar for both field orientations. The experimental data show that this process terminates at a field strength of ~0.7 T. For the field range of 0.7–1.6 T, i.e., step 2, there are already energetically optimal field-aligned domains, although some anisotropy-stabilized domains exist for the field applied along the {1, 1, 0} direction. Then at the transition into regime II at step 3, the external field wins the competition over the bond anisotropy for the magnetic field applied perpendicular to the Ru–Ru bonds, such that the pseudospins in all domains rotate perpendicular to the field directions pushing α-RuCl3 into a single polarized spin-flopped state. A further increase in the field strength, i.e., moving closer to the critical line of the phase transition towards the quantum paramagnetic state, results in a natural decrease of the observed rotation, i.e., the magnitude of the order parameter. Before entering the quantum paramagnetic state above 7 T another anomaly is seen in the MO data. In contrast to the low-field transition, this anomaly at 6.2 T in dθ/dB is not immune to temperature changes and fades off for increasing temperatures from 2.8 to 3.5 K systematically for fields applied perpendicular to the bond (c.f. Fig. 6c). Its dependence on the in-plane field angle is illustrated by the fact, that for Bab∥{1, 0, 0} no signature of an intermediate field-induced transition in the vicinity of 6.2 T can be identified since dθ/dB is almost constant even at the lowest temperature of 2.8 K. The fact that this intermediate field-induced transition strictly depends on the orientation of Bab w.r.t. to the crystallographic axis conveys the emergence of an anisotropy-related novel magnetically ordered phase. This phase was named ZZ2 and its origin has been discussed in terms of inter-layer anisotropies very recently44. It was understood in terms of a field-induced inhomogeneous spin canting between different threefold and sixfold ZZ stackings for Bab∥{1, 1, 0}, while the canting is homogeneous for Bab∥{1, 0, 0}44. In addition, it has been discussed in terms of an inverse melting phenomenon54. It is worth to note that the MO response is sensitive to this transition and its first-order character fits to the observed large MO hysteresis effect. Nevertheless, a microscopic origin of this field-induced transition, nor an estimate for the underlying inter-layer exchange interactions can be given based on our observations, but calls for future MO measurements at elevated magnetic fields.

In summary, we have reported magneto-optical spectroscopy measurements on the Kitaev spin liquid candidate material α-RuCl3. Our study establishes magneto-optical spectroscopy as a versatile experimental tool to elucidate exotic phases of quantum materials. We observed two intermediate metamagnetic transitions at ≈1.6 T and 6.2 T for the magnetic field applied perpendicular to the Ru–Ru bond, while none of these transitions appears for the magnetic field aligned parallel to the Ru–Ru bonds. We clarified the nature of the low-field transition and discussed it in terms of a spin-flop transition, where the external field overcomes the anisotropy energy to align the Néel vector nearly orthogonal to the field direction. The effective anisotropy field has been determined to be 0.0027 T. Further, we confirmed the emergence of the previously observed high-field intermediate phase. Our results point out the importance of anisotropic intra- and inter-layer bond anisotropies and the necessity to include those in future theoretical calculations. Besides that, we illuminated the importance of the in-plane field angle, which calls for future MO studies in the high-field proximate spin liquid phase of α-RuCl3. The spin-flop transition at a moderate field strength motivates further studies on α-RuCl3, as the reorientation of the Néel vector opens pathways to vary the magnetoresistance almost continuously. Here, measurements of the anisotropic magnetoresistance, which are also even in the magnetization as MLD and MLB63, should be considered for future experiments to access the precise control of the Néel vector88. This could in principle lead to future implementation of α-RuCl3 in antiferromagnetic spintronics89.

Methods

Sample growth, characterization, and orientation

High-quality α-RuCl3 crystals were prepared by vacuum sublimation37. The different samples of the same batch were characterized by SQUID magnetometry, showing a sharp transition at around TN ≈ 7 K in zero applied field corresponding to the ABC stacking order35,56. No additional magnetic transitions above TN are observed, which have been related to a different stacking order ABAB with a two-layer periodicity or strain-introduced stacking faults due to extensive handling or deformation of the crystals. This bulk technique can only provide the first indication of sample quality for an optics study. Cleaving or polishing introduces strain. In this regard, we refrained from any sample treatment and used an as-grown α-RuCl3 sample. The temperature dependence of the equilibrium MO response shown in Fig. 1d shows a clear phase transition at TN ≈ 7 K, confirming good sample quality (Supplementary Information Note 1).

The different α-RuCl3 samples have been oriented via a standard X-ray Laue-diffractometer at room temperature.

Experimental procedures and measurement techniques

For the study of the MO response of α-RuCl3, the high-quality as-grown samples were placed in a helium-cooled cryostat (Oxford Spectromag) with temperatures down to 2.2 K inside the coils of a superconducting magnet with magnetic field strengths up to ±7 T. Figure 1a illustrates the experimental setup. The magnetic field was applied along different crystallographic directions within the crystallographic ab-plane, i.e. within the honeycomb layers. The polarization of incident light was rotated by a half waveplate and after initial tests, set to the purely linearly s-polarized setting, i.e., θ0 = 0 for the maximum signal in zero field. The measurements of the second-order MO response were carried out in the so-called Voigt geometry67 at near-normal incidence, such that the light wave propagation k was perpendicular to the honeycomb layer planes (k⊥ab) and magnetic field vector Bab (Fig. 1a). The MO measurements were performed using a continuous laser with a wavelength of 532 nm and the laser spot was focused to a spot size of 200 × 200 μm on the sample with the power set to 50 μW. Detection of the MO response Θ = θ + iη was done using a polarization modulation technique, in which the relative phase of two orthogonal linear polarizations was modulated that pass through a photoelastic modulator (PEM)90. The change in rotation θ and ellipticity η were probed simultaneously (Supplementary Information Note 7).