Identification of the compressive trigonal crystal field and orbital polarization in strained monolayer α-RuCl₃ on graphite

Abstract

α-RuCl₃ is a tantalizing playground to search for the quantum spin liquid states with leading Kitaev term due to the conformation of Jackeli–Khaliullin mechanism. Suppressing the non-Kitaev interaction to approach the Kitaev limit is the major objective in the experimental design of Kitaev–Jackeli–Khaliullin materials, for which the practical feasibility rests on the manipulation of the exact crystalline structure. A tensile strain, leading to the further compressive trigonal field splitting which would alter the spin-orbital entangled character and tune the intrinsic hopping exchanges, has been widely proposed as the promising strategy to enhance the comparative Kitaev interaction. In this work, we preform in-depth Scan Tunneling Microscope study on the electronic patterns of strained α-RuCl₃ monolayer on graphite substrate. Based on Chen’s derivative rule, a further trigonal field splitting is identified by analyzing the lateral-gradient imaging effect caused by an anisotropic tip. A spatial variation of the magnitude of trigonal crystal field due to the slight inhomogeneity of strain accords with the orbital polarization near the Mott gap. All evidences point to the facts that the compressive trigonal crystal field in the stretched α-RuCl₃ monolayer lifts the degeneracy of t_2g states with higher $\text{e}_{\text{g}}'$ and lower a_1g orbitals, and the Coulomb interactions of the $\text{e}_{\text{g}}'$ orbitals take the main responsibility for the Mott effect.

1 Introduction

α-RuCl₃ is one of the most promising candidates to approach the Kitaev honeycomb model [1] and realize the quantum spin liquid (QSL) states [2–6] with more and more emerging evidences of Kitaev interactions [7–13]. However, from a realistic perspective, the Kitaev interaction is always accompanied with a rich variety of other undesirable magnetic interactions such as Heisenberg interaction, Dzyaloshinskii–Moriya interaction and longer-range interaction [14–17]. The latter spawns many difficulties in engineering the pure Kitaev model in real material and nearly always results in an ordered magnetization at low temperature [16, 18, 19].

The potential of α-RuCl₃ to realize the exotic spin states with dominating Kitaev interaction roots in Jackeli–Khaliullin mechanism which considers the 4d or 5d electrons on a honeycomb lattice consisting of ideal edge-sharing octahedra [2, 3]. Under the interplay of spin-orbit coupling (SOC), electronic correlation, and RuCl₆ octahedral crystal field, the Ru ion presents a 4d⁵ electronic configuration with one hole occupying a $\mathrm{J}_{\mathrm{eff}} = 1/2$ state at upper Hubbard band (UHB) of the $\mathrm{t}_{2\mathrm{g}}$ manifold [5, 6]. Combined with the particular ligand geometry and hopping exchange, the non-Kitaev magnetic interactions are effectively suppressed, making α-RuCl₃ system very close to the Kitaev limit [14]. Nevertheless, it’s experimentally found that the remnant unwanted interactions are still strong enough to destroy the QSL ground state leading to a zigzag antiferromagnetic order in bulk α-RuCl₃ at low temperature [19–21]. The practical complexity is embedded in the substantial trigonal distortion of local octahedral crystal field, which will significantly alter the spin-orbital character and modify the magnetic couplings subtly [14, 17]. Given the competition between various magnetic interactions, the fragile QSL state occupies an extremely restrictive area in the phase diagram [15, 17, 22] which greatly challenges the experimental realization. Controlling the trigonal field splitting versus SOC represents a significant synthetic goal in manipulating the inherent magnetic couplings and designing the Kitaev–Jackeli–Khaliullin materials [14]. Correspondingly, theoretical analysis points out that the compressive trigonal crystal field splitting resulting from a small amount of tensile strain is believed to lead to a considerable modification of the magnetic interactions with enhanced Kitaev term and push the system closer to the Kitaev limit [17, 22–25].

Premised on previous work of monolayer α-RuCl₃ on highly oriented pyrolytic graphite (HOPG) substrate [26] where the p-d hybridization, orbital texture and orbital character of electronic bands are comprehensively unraveled, we also notice that a significant tensile strain has been applied to the α-RuCl₃ monolayer due to the substate effect which would lead to a further crystal field splitting in expectation. In this work, we identify the compressive trigonal splitting and resultant orbital polarization with Scan Tunneling Microscope (STM) via the lateral-gradient imaging effect which derives from the orbital-dependent tunneling matrix and is summarized as Chen’s derivative rule [27]. The tensile strain makes ${\mathrm{t}_{2\mathrm{g}}}$ states across the Mott gap further energetically split into $\mathrm{a}_{1\mathrm{g}}$ singlet and $\mathrm{e}_{\mathrm{g}}'$ doublet, and the lattice mismatch makes the strain overall expansive but slightly non-uniform in plane resulting in the spatially heterogenous magnitude of the trigonal field splitting as well as the lateral corrugation of the energy positions of $\mathrm{a}_{1\mathrm{g}}$ and $\mathrm{e}_{\mathrm{g}}'$ states. Taking the orientation of orbital lobes into consideration, the energy-resolved amplitude of STM topographic undulation helps to confirm the compressive trigonal field splitting with lower $\mathrm{a}_{1\mathrm{g}}$ states and higher $\mathrm{e}_{\mathrm{g}}'$ states.

2 Electronic structures and trigonal field splitting of α-RuCl₃

The atomic structure, electronic structures under the interplay of crystal field splitting, SOC and Hubbard U, and intra-octahedron p-d orbital hybridization of monolayer α-RuCl₃ on graphite substrate are summarized in Fig. 1 (see details of the synthetic method, nanoscale topographic characterization, electronic structure and experimental tunneling spectrum in reference [26]). As shown in Fig. 1(b, c), the 4d states of $\mathrm{Ru}^{3+}$ ion subjected to the primary RuCl₆ octahedral crystal field (CF) are split into the $\mathrm{t}_{2\mathrm{g}}$ triplet and $\mathrm{e}_{\mathrm{g}}$ doublet with an ∼2.0 eV crystal field gap (CFG), leaving a ${\mathrm{t}_{2\mathrm{g}}}^{5}$ orbital occupation. The strong on-site Coulomb repulsion opens a Mott gap (MG) of ∼0.6 eV within the $\mathrm{t}_{2\mathrm{g}}$ manifold, leaving the upper Hubbard band (UHB) and lower Hubbard band (LHB) across the MG. The $\mathrm{e}_{\mathrm{g}}$ and $\mathrm{t}_{2\mathrm{g}}$ orbitals mainly hybridize with $\mathrm{p}_{\sigma}$ and $\mathrm{p}_{\pi}$ orbitals of Cl ions respectively based on the maximized orbital overlap, forming the pdσ and pdπ bonds [Fig. 1(d)], and giving rise to the energy configuration of higher $\mathrm{e}_{\mathrm{g}}\text{-}\mathrm{p}_{\sigma}$ bands and lower $\mathrm{t}_{2\mathrm{g}}\text{-}\mathrm{p}_{\pi}$ bands. Noting that the topmost lobes of Cl-p orbitals are primarily detected by the tip and contribute dominantly to the STM topographic texture, switching the bias energy between $\mathrm{e}_{\mathrm{g}}\text{-}\mathrm{p}_{\sigma}$ and $\mathrm{t}_{2\mathrm{g}} \text{-}\mathrm{p}_{\pi}$ bands would lead to completely different orbital textures determined by the orientations of topmost Cl-p lobes. The lobes of Cl-$\mathrm{p}_{\pi}$ orbital are perpendicular to the associated Ru₂Cl₂ plane [formed by two neighboring Ru ions and two bridging Cl ions, marked as the gray plane in Fig. 1(e)] while the lobes of $\mathrm{p}_{\sigma}$ lie in the Ru₂Cl₂ plane and are perpendicular to the associated Ru–Ru bond [Fig. 1(d, e)]. As a result, the upper lobes of $\mathrm{p}_{\sigma}$ and $\mathrm{p}_{\pi}$ orbitals deviate oppositely with respect to the nearest Ru–Ru bond [the corresponding deviation is indicated by the arrows in Fig. 1(e, f)], resulting in the trimer pattern of $\mathrm{e}_{\mathrm{g}}\text{-}\mathrm{p}_{\sigma}$ bands and Kagome-like pattern of $\mathrm{t}_{2\mathrm{g}}\text{-}\mathrm{p}_{\pi}$ bands in STM image [Fig. 1(d)]. The same Kagome-like texture across the MG gap proves that the lowest conduction band and the highest valence band are Hubbard bands derived from the $\mathrm{t}_{2\mathrm{g}}\text{-}\mathrm{p}_{\pi}$ orbitals [see details in reference [26]].

Figure 1

Schematic drawings illustrating the electronic structure and local distortion of α-RuCl₃. (a) Atomic structure of monolayer α-RuCl₃. The a and b are the in-plane unit cell vectors and c is the out-of-plane [111] axis. (b) Local trigonal distortion and corresponding compressive or elongational splitting of t_2g orbitals. The involvement of SOC entangles different orbitals, leading to split states with different $\text{a}_{1\text{g}}/\text{e}_{\text{g}}'$ proportion. Under the compressive (elongational) trigonal field, the UHB would contain more $\text{e}_{\text{g}}'$ (a_1g) portion, which is indicated by the gradient color. (c) sketch of the density of state. The octahedral crystal field split the Ru-4d orbitals into lower t_2g orbitals and higher e_g orbitals with an energy difference of crystal field gap (CFG), and the Coulomb repulsion opens a Mott gap (MG) across the t_2g manifold. (d) The orbital configurations illustrating the p-d hybridization in the Ru–Cl–Ru frame. Two orthogonal Cl–Ru axes point along x and y direction respectively. (e) Perspective views showing the orbital orientation of $\text{p}_{\sigma}$ and $\text{p}_{\pi}$ orbitals. The green and orange arrows represent the displacement from topmost lobes of $\text{p}_{\sigma}$ or $\text{p}_{\pi}$ orbitals to corresponding Cl ion position. (f) STM textures extracted from reference [26] showing the triangularly arranged trimers for $\text{e}_{\text{g}}\text{-}\text{p}_{\sigma}$ bands and Kagome-like pattern for $\text{t}_{2\text{g}}\text{-}\text{p}_{\pi}$ bands. The Ru-hexagon (gray balls) and topmost Cl ions (blue balls) are superimposed

Any lateral lattice stretch or shrink would prolong or shorten the in-plane Ru–Ru bonds as well as widen or reduce the Ru–Cl–Ru angle, rendering the substantial trigonal compression or elongation of the RuCl₆ octahedra along the [111] principal axis [Fig. 1(b)]. In this case, the $\mathrm{t}_{2\mathrm{g}}$ manifold is further split into $\mathrm{a}_{1\mathrm{g}}$ singlet and $\mathrm{e}_{\mathrm{g}}'$ doublet. Although the $\mathrm{a}_{1\mathrm{g}}$ and $\mathrm{e}_{\mathrm{g}}'$ orbitals both hold the same dominant hybridization with $\mathrm{p}_{\pi}$ orbital as the $\mathrm{t}_{2\mathrm{g}}$ orbital does, the $\mathrm{t}_{2\mathrm{g}}$ octahedral symmetry of the final Hubbard bands will be reduced after the introduction of the splitting, leading to the orbital polarization that the UHB and LHB possess distinct orbital characters with respect to the $\mathrm{a}_{1\mathrm{g}}$ and $\mathrm{e}_{\mathrm{g}}'$ basis. To be more specific, the UHB (LHB) would contain more $\mathrm{e}_{\mathrm{g}}'$ ($\mathrm{a}_{1\mathrm{g}}$) components under the trigonal field compression, and vice versa for the case of elongation. In terms of the cubic $\{\mathrm{x}, \mathrm{y}, \mathrm{z}\}$ coordinate system of local octahedron where the axes orient from one Ru ion to three intra-octahedron Cl ions and the out-of-plane c direction is thus denoted as [111] direction, the spatial expression of $\mathrm{a}_{1\mathrm{g}}$ and $\mathrm{e}_{\mathrm{g}}'$ orbitals are [14]:

$$\begin{aligned}& | {a}_{{1g}} \rangle = \frac{1}{\sqrt{3}} \bigl(| xy \rangle + | xz \rangle + | yz \rangle \bigr),\qquad E_{{a}_{{1g}}} =-2\Delta, \end{aligned}$$

(1)

$$\begin{aligned}& \bigl| {e}_{{g}} ' \bigr\rangle = \textstyle\begin{cases} \frac{1}{\sqrt{6}} (2 | xy \rangle - | xz \rangle - | yz \rangle ), \\ \frac{1}{\sqrt{2}} ( | xz \rangle - | yz \rangle ), \end{cases}\displaystyle \qquad E_{e_{g} '} =\Delta, \end{aligned}$$

(2)

where the Δ represents the trigonal field splitting and $\Delta >0$ ($\Delta <0$) means that the trigonal field is compressive (elongational). By inspecting the analytical expressions one can see that the wavefunction of $\mathrm{a}_{1\mathrm{g}}$ orbital mainly distributes along the out-of-plane [111] direction while the [111] axis is a nodal line for $\mathrm{e}_{\mathrm{g}}'$ orbitals. The inherent hopping exchanges and magnetic couplings will be determined by the ratio of Δ and SOC term λ since the former polarizes the orbital configurations of 4d electrons and the latter mixes the spin-orbital character by entangling the spin with unquenched orbital momentum. In our monolayer α-RuCl₃/HOPG system, an ∼2.67% lattice expansion is validated [26], and thus, a compressive trigonal field and corresponding orbital polarization are anticipated. Since the typical strength of distorted trigonal field (∼10 meV) is an order of magnitude smaller than the Ru 4d SOC (∼100 meV) [28], the latter would play the dominant role to split the $\mathrm{t}_{2\mathrm{g}}$ orbital into higher $\mathrm{J}_{\mathrm{eff}} = 1/2$ singlet and lower $\mathrm{J}_{\mathrm{eff}} = 3/2$ doublet, where the practical effect of the former, by partially quenching the angular momentum, is to alter the Ru 4d electron states from spin-orbit entangled $\mathrm{J}_{\mathrm{eff}}$ states to spin-orbital mixtures consisting of different $\mathrm{a}_{1\mathrm{g}}/\mathrm{e}_{\mathrm{g}}'$ proportion [Fig. 1(b)] determined by the $\Delta /\lambda$. In term of the complexity of the resultant highly mixed spin-orbital structure, the final split eigenstates are labeled as $\mathrm{t}_{2\mathrm{g}}$ states in Fig. 1(c, f) referring to the common original $\mathrm{t}_{2\mathrm{g}}$ representation.

3 Orbital polarization revealed by multiorbital lateral-gradient imaging effect

In the previous work we show that the detected bands ranging from 1.5 eV to $-2~\text{eV}$ are $\mathrm{t}_{2\mathrm{g}}$ bands since the STM images and dI/dV maps obtained in this energy range exhibit the same Kagome-like pattern corresponding to the $\mathrm{p}_{\pi}$ orbitals [26]. However, as shown in Fig. 2(a), an odd observation is that, while maintaining the same $\mathrm{t}_{2\mathrm{g}}$- $\mathrm{p}_{\pi}$ orbital texture for the entire $\mathrm{t}_{2\mathrm{g}}$ bands, the constant-height dI/dV maps for UHB and LHB show that the electronic pattern of one of the Hubbard band deviates conspicuously compared to another [the green spots of different images in Fig. 2(a) represent exactly the same location]. To ensure the spatial correspondence of the dI/dV maps of different bias voltages, the constant-height dI/dV mapping was carefully measured by ramping the bias voltage to obtain the dI/dV spectrum at each position belonging to the chosen area, and the energy-dependent slicing maps showing the spatial distribution of dI/dV values taken at specific energy are displayed in Fig. 2(a). Therefore, the tip-sample relative position at each measuring point is kept during the obtainment of single spectrum. It’s noticed that although the three-fold rotational symmetry is hold in Fig. 2(a), the symmetry center (the black hollows) of the patterns is shifted abruptly by ∼1.8 Å from positive to negative bias voltages (marked as white double arrows). This phenomenon most likely stems from the STM imaging artifact rather than the intrinsic electronic character because the abrupt shift apparently violates the crystal symmetry of α-RuCl₃.

Figure 2

Experimental and simulated differential conductance maps with an anisotropic tip. (a) The constant-height dI/dV maps of different bias voltages. The six green dots mark the position of the bright protrusions at 1.0 V so that a bias-dependent symmetry center translation could be seen in other dI/dV maps. The tip-sample separation was fixed at the setpoint of $\text{V} =-1.2~\text{V}$, $\text{I}=-1.7~\text{nA}$. (b) The simulated dI/dV patterns at a constant-height condition after considering the s-p hybridized components of the tip. The pattern is deforming and moving as changing the ratio and direction of p states. The weight and direction of p components is labeled (in the white box)

As regards the imaging mechanism of STM, while the prominent Tersoff–Hamann model which only considers the s-wave tip states obviously cannot explain our observation since it shows a simplified result that the STM images approximately resemble the local density of states (LDOS), the Chen’s derivative rule points out that by expanding the vacuum tail of the tip wavefunctions in terms of spherical harmonics, the p- or d-wave tip states image the derivatives of surficial LDOS [27]. As a result, for a general tip with inevitable anisotropic geometry of the apex which also contain spherical harmonic components beyond s-wave, the lateral gradient of surficial LDOS usually contributes a lot in the STM image. Later, we will show that the relative electronic pattern deviation between UHB and LHB can be explained under Chen’s mechanism, which also reveals the different orbital character of the LHB and UHB beyond the general $\mathrm{t}_{2\mathrm{g}}$ description, that is, the further crystal field splitting of $\mathrm{t}_{2\mathrm{g}}$ manifold. By comparing our following analysis and simulations to the inventory STM images, vast visual details can be explained from a phenomenal point of view.

In single-particle approximation, $\mathrm{T} = 0~\text{K}$ and a small bias voltage regime, the differential conductance dI/dV could be expressed by

$$ \begin{aligned} \frac{{dI}}{{dV}} &= \frac{4\pi e}{\hbar} n^{T} \bigl( E_{F}^{T} \bigr) n^{\mathrm{Sam}} \bigl( E_{F}^{\mathrm{Sam}} +eV \bigr) \\ &\quad {}\times \bigl\vert M \bigl( E_{F}^{\mathrm{Sam}} +eV, E_{F}^{T} \bigr) \bigr\vert ^{2}, \end{aligned} $$

(3)

where $n^{T}$ and $n^{\mathrm{Sam}}$ are the density of states of tip and sample as a function of energy, $E_{F}^{\mathrm{Sam}}$ and $E_{F}^{T}$ are the Fermi level of sample and tip respectively. M is the tunneling matrix determined by the eigenstates of the tip and sample: $M= \langle \psi ^{T} \vert U_{T} \vert \psi ^{\mathrm{Sam}} \rangle$, where $\vert \psi ^{T} \rangle$ denotes the tip state, $\vert \psi ^{\mathrm{Sam}} \rangle$ denotes the sample state, and $U_{T}$ is the tip potential. We use the Chen’s spatial series expansion to describe the anisotropic tip states [27]. Only s and the in-plane p components of tip states will be considered since that the $\mathrm{p}_{\mathrm{z}}$ state behaves similarly to an s state in STM imaging and d states are much more localized. The images obtained by tunneling to s- and p-tip states have been discussed in Leo Gross et al.’s work [29]. In their work, the STM images of molecular orbitals were perfectly explained in terms of tunneling to the s-wave tip state $\vert M_{s} \vert ^{2} =| \langle \psi _{s}^{\mathrm{T}} \vert U_{T} \vert \psi ^{\mathrm{Sam}} \rangle \vert ^{2} \propto \vert \psi ^{\mathrm{Sam}} \vert ^{2}$ and to the p-wave tip state $\vert M_{p} \vert ^{2} =| \langle \psi _{p}^{\mathrm{T}} \vert U_{T} \vert \psi ^{\mathrm{Sam}} \rangle \vert ^{2} \propto | \frac{\partial \psi ^{\mathrm{Sam}}}{ \partial x} \vert ^{2}$ (assuming the lobes of p states are along the x direction), where the subscripts denote the orbital type of tip states. It has been clearly suggested in their work that all simulations solely involving the terms of $\vert M_{s} \vert ^{2}$ and $\vert M_{p} \vert ^{2}$ do not alter the positions of symmetry centers, while the former term $\vert M_{s} \vert ^{2}$ faithfully reflects the LDOS distribution but the latter term $\vert M_{p} \vert ^{2}$ highlights the lateral gradient of LDOS. In order to explain our observations where the symmetry center moves with energy, we need to introduce the s-p interference tunneling term $M_{s} M_{p}$ coming from the interference between tunnel-into-s channel and tunnel-into-p channel. Practically, the $M_{s} M_{p}$ term is nonzero when the inversion symmetry of tip state is broken (which is the most likely case for a general tip). In other words, the s-p interference tunneling is a direct consequence of the s-p orbital hybridization. To learn the orbital-interfere tunneling, we model a fully s-p hybridized tip wave function $\vert \psi ^{T} \rangle =a \vert \psi _{s}^{T} \rangle +b \vert \psi _{\vec{p}}^{T} \rangle$, where a and b are real numbers and p⃗ is a p state with the nodal plane perpendicular to the surface plane. Thus, the differential conductance is proportional to the tunneling matrix:

$$ \begin{aligned} \frac{{dI}}{{dV}} &=\sigma \propto \bigl\vert a \bigl\langle \psi _{s}^{T} \bigr\vert U_{T} \bigl\vert \psi ^{\mathrm{Sam}} \bigr\rangle +b \bigl\langle \psi _{\vec{p}}^{T} \bigr\vert U_{T} \bigl\vert \psi ^{\mathrm{Sam}} \bigr\rangle \bigr\vert ^{2} \\ &= \vert a M_{s} +b M_{\vec{p}} \vert ^{2} . \end{aligned} $$

(4)

Using Chen’s derivative rule [27], the differential conductance map could be expressed by:

$$ \sigma (V,x,y)\propto \biggl\vert a \psi ^{\mathrm{Sam}} (V,x,y)+b \frac{\partial}{ \partial \vec{p}^{0}} \psi ^{\mathrm{Sam}} (V,x,y) \biggr\vert ^{2}. $$

(5)

Following Eq. (5), we simulated the dI/dV maps with in-plane rotating directions (denoted as $\vec{p}^{0}$) of the p-wave tip states and a fixed s-to-p weight ratio of 4 [see Fig. 2(b) for the results], based on the calculated DOS extracted in reference [26]. After introducing the s-p hybridized components, due to the presence of a derivative term of $M_{s} M_{p} \propto \psi \frac{\partial \psi}{\partial \vec{\mathrm{p}}^{0}}$, the bright features are shifted away from the positions imaged by a pure s-wave tip (green dots) by rotating the direction of p-component.

In this sense, above analysis indicates that similar orbital-alternation mechanism occurs and causes the abrupt shift of symmetry centers between UHB’s and LHB’s electronic patterns. There are two necessary conditions to generate the shift of symmetry centers, one is the s-p hybridized tip which can produce the crucial $\psi \frac{\partial \psi}{\partial \vec{\mathrm{p}}^{0}}$ term in the imaging process, another is character alternation of orbitals participating in the tunneling process when changing the bias voltage from UHB to LHB. In the tunneling from an insulating sample to a metallic tip, the differential conductance is only related to the tip states around the Fermi level which contain fixed orbitals. The tip orbital character remains basically invariant during ramping the bias from UHB to LHB, hence the variation of orbital components of the sample is responsible for the relative shift between the UHB and the LHB. Therefore, we have to consider the further trigonal splitting of Ru-$\mathrm{t}_{2\mathrm{g}}$ orbitals, i.e., the $\mathrm{a}_{1\mathrm{g}}$ singlet and the $\mathrm{e}_{\mathrm{g}}'$ doublet, which can produce different orbital character of LHB and UHB. The $\mathrm{e}_{\mathrm{g}}'$ orbital has a vertical nodal line and an in-plane wavefunction distribution, which makes it nearly invisible to the tip state. Noting that the $\mathrm{a}_{1\mathrm{g}}$ orbital has a vertical lobe sticking out between the three Cl atoms into the vacuum, the different proportions of $\mathrm{a}_{1\mathrm{g}}$ component in UHB and LHB is supposed to be the exclusive reason for the large shift in Fig. 2(a).

To generalize this effect to a multiband case, the tunneling conductance can be formulated as:

$$ \begin{aligned} \sigma ( V,x,y ) \propto{}& \sum_{\nu} \bigl\vert a \bigl\langle \psi _{s}^{T} \bigr\vert U_{T} \bigl\vert \psi _{\nu}^{\mathrm{Sam}} ( V,x,y ) \bigr\rangle \\ &{}+b \bigl\langle \psi _{\vec{p}}^{T} \bigr\vert U_{T} \bigl\vert \psi _{\nu}^{\mathrm{Sam}} ( V,x,y ) \bigr\rangle \bigr\vert ^{2}, \end{aligned} $$

(6)

where ν denotes the indices of involved orbitals in the sample. The interference between different bands vanishes here, because spin-orbit coupling is ignored here for convenience [30].

Near the Fermi level, based on the aforementioned orbital geometry and STM sensitivity, the differential conductance map could be formulated as:

$$ \sigma ( V,x,y ) = \sigma _{p} ( V,x,y ) + \sigma _{a_{1g}} (V,x,y), $$

(7)

where $\sigma _{p} ( V,x,y )$ and $\sigma _{a_{1g}} (V,x,y)$ are proportional to the LDOS and tunneling matrixes of p and $\mathrm{a}_{1\mathrm{g}}$ orbitals, respectively. The relative weight of the two terms on the right side of Eq. (7) gives a freedom of relative shift between the images of the UHB the LHB. So far, we can reach the conclusion that the relative pattern shift is generated by different ratio of $\sigma _{a_{1g}} / \sigma _{p}$ for LHB and UHB. This indicates the existence of a trigonal field splitting and ensuing orbital polarization, otherwise the LHB and UHB would hold the same octahedral $\mathrm{t}_{2\mathrm{g}}$ symmetry with equal $\sigma _{a_{1g}} / \sigma _{p}$ ratio and show identical electronic pattern without the abrupt shift. Although whether the trigonal field is compressive or elongational is not validated yet at the current step of analysis, the trigonal field orbital polarization is revealed.

4 The inhomogeneous trigonal field compression

The trigonal crystal field is tuned by the strain applied by the graphite substrate, however, this effect is found to be nonuniform. The inhomogeneity is signified by the quasiperiodic modulations with peak-to-peak distance of $1\sim 3~\text{nm}$ as shown in Fig. 3(b–g). The quasiperiodic modulations are the electronic signals responding to tiny structural modifications instead of the structural deformation of crystal, because the high-bias-voltage images [Fig. 3(a, h)] show insignificant corrugations and the lattice constants are basically uniform without obvious local structural squeezing or stretch. The trigonal field is expected to be a globally compressive one while its magnitude fluctuates spatially in response to the tiny structural modulation. Near the Mott gap, the topographic corrugation is stronger for negative bias voltages [Fig. 3(d, e)] compared to positive ones [Fig. 3(b, c)]. This phenomenon is in analogy to the rigid shift of the symmetry center in Fig. 2(a) from the UHB to the LHB, indicating a similar orbital-related mechanism.

Figure 3

STM images of different bias. All the maps are adapted to the same color scale. Setup current: 200 pA, and the corresponding setup bias is indicated in each map

To understand the reason and impact of the spatial variation of trigonal crystal field (denoted as Δ) on LDOS, we presume the spatial fluctuation in Δ as δΔ. Following the conventional definition and Eqs. (1), (2), the $\mathrm{a}_{1\mathrm{g}}$ singlet is below the $\mathrm{e}_{\mathrm{g}}'$ doublet in energy when $\Delta >0$, i.e., the case of trigonal compression. As the trigonal crystal field is normally not strong and its strength is smaller than or comparable to the strength of spin-orbit coupling [28], $\mathrm{a}_{1\mathrm{g}}$ and $\mathrm{e}_{\mathrm{g}}'$ orbitals are highly mixed. Near the band edge, the spatial fluctuation of LDOS (denoted as $\Delta \rho _{\nu}$) is roughly proportional to δΔ, which writes $\delta \Delta \propto - \delta \rho _{a_{1 g}} \propto \delta \rho _{e_{g} '}$. Since the LDOS of Cl-p orbital hybridizes with $\mathrm{a}_{1\mathrm{g}}$ and $\mathrm{e}_{\mathrm{g}}'$ orbitals in a fair strength because the DOS of p states is just a copy of that of $\mathrm{t}_{2\mathrm{g}}$ [26], we consider the Cl-$\mathrm{p}_{\pi}$ state as an indirect access to the overall Ru-$\mathrm{t}_{2\mathrm{g}}$ state. In this sense, both the LDOS of $\mathrm{a}_{1\mathrm{g}}$ and $\mathrm{e}_{\mathrm{g}}'$ orbitals fluctuates in space under the heterogenous modulation of trigonal field δΔ giving rise to the undulation map of $\delta \rho _{a_{1 g}} ( x,y )$ and $\delta \rho _{e_{g} '} (x,y)$. The spatial fluctuation of $\mathrm{a}_{1\mathrm{g}}$ orbitals $\delta \rho _{a_{1 g}} ( x,y )$ would produce much stronger topographic corrugation in STM image noting that the out-of-plane distributed $\mathrm{a}_{1\mathrm{g}}$ orbitals would be detected by the STM in a larger probability than the in-plane distributed $\mathrm{e}_{\mathrm{g}}'$ orbitals. The energy band with large $\mathrm{a}_{1\mathrm{g}}$ proportion holds larger $\delta \rho _{a_{1 g}} ( x,y )$. Therefore, the observation that the stronger topographic corrugation lies in LHB indicates the larger weight of $\mathrm{a}_{1\mathrm{g}}$ orbital component of LHB. Therefore, the energy configuration of lower $\mathrm{a}_{1\mathrm{g}}$ singlet and higher $\mathrm{e}_{\mathrm{g}}'$ doublet evinces the global trigonal field compression [as described in Fig. 1(b)] which is also consistent with our experimental finding of 2.67% lattice expansion of the α-RuCl₃ monolayer on graphite.

5 Conclusions

We investigated the energy-resolved dI/dV maps on strained α-RuCl₃ monolayer supported by graphite substrate. The combined observations, including the unusual deviation between the electronic pattern of UHB and LHB and the stronger electronic corrugations of LHB subjected to the heterogeneous strain, reveal the discrepant orbital character of UHB and LHB as well as the further compressive trigonal field splitting of the $\mathrm{t}_{2\mathrm{g}}$ manifold with the lower $\mathrm{a}_{1\mathrm{g}}$ and higher $\mathrm{e}_{\mathrm{g}}'$ orbitals. The trigonal field compression would definitely make the spin-orbital structure and inherent magnetic interactions in the strained monolayer differ from the bulk system. Besides the stretched lattice which may be advantageous for the advancement toward Kitaev limit, the hole state of each $\mathrm{Ru}^{3+}$ ion occupies the in-plane distributed $\mathrm{e}_{\mathrm{g}}'$ doublet where the spin-orbit coupling entangles states within the double-degenerate manifold [31] and favors an out-of-plane easy axis in terms of the magnetic anisotropy [32]. Thus, it’s intriguing and worth of more studies towards the exact magnetic ground order of the orbital polarized monolayer system, and the methodological technique of investigating the lateral gradient imaging effect with an anisotropic tip can be applied to the general STM field to unravel the underlying orbital nature and excavate in-depth electronic information.