1 Introduction

In a certain crystal, electron state with two-fold rotational (C2) symmetry emerges despite its lattice obeys C3 or C4 symmetries. The electron state holding independent rotational symmetry, which differs from the symmetry of crystal structure, is named as rotational symmetry breaking phase. It is one of the important novel phenomena arising in many quantum materials, unconventional superconductors and strong correlated systems. Scientists have discovered many examples. Stripe order phase in cuprates is reported, where hidden-order phase is supposed to be the reason [1, 2]. Anisotropic transition driven by nematicity is found in iron arsenide [3–6], paving the way to find the true nature of charge order in this class of superconductors. Rotational symmetry breaking order in twisted graphene systems [7, 8] and nematic superconductivity in Bi2Se3 based topological superconductor candidates [9–11], make themselves good platforms for explaining strong correlation induced quantum phenomena. Those observations guarantee an important role that rotational symmetry breaking plays in quantum materials, for every discovery of rotational symmetry breaking phase in a condensed matter may open avenue toward new physics.

The Mn-Bi-Te class of materials are recently discovered topological insulators with inherent magnetic orderings that are predicted to host a variety of unusual phenomena [12–32]. Among them, the quantum anomalous Hall effect and axion insulator state probably have the highest significance and attract enormous research attentions. Furthermore, many physicists believe the Mn-Bi-Te class of material is an ideal platform for magnetic topology research because the theoretical description is simple and sufficient. For example, theorists found that band theory together with strong spin-orbit coupling and Zeeman exchange field, i.e. a model within single-particle picture, is enough to capture the major physics of the material.

Scanning tunneling microscopy (STM) is a powerful technique to investigate topological materials [33–36], here we apply low temperature STM to examine the electronic states on ultra-high vacuum (UHV) cleaved MnBi2Te4 and MnBi4Te7 crystals. Surprisingly, our quasiparticle interference (QPI) patterns show a clearly violation of rotational symmetry of the crystal lattice, as well as a nematic-like electronic state at certain energies. We believe that understanding the facts necessitates a theory that goes beyond the single-particle picture.

2 Results and discussions

The simplest instance in the Mn-Bi-Te class of the magnetic topological insulators is MnBi2Te4. Its crystal is composed of layers of MnBi2Te4 that are stacked along the c axis (Fig. 1(a)). The spins in Mn atoms are believed to form ferromagnetic order in each MnBi2Te4 layer, and out-of-plane A-type antiferromagnetic (AFM) order between adjacent layers [12, 13]. Notably, the MnBi2Te4 crystal structure possess an out-of-plane C3 rotational symmetry. It is also visible in our atomic resolution STM image (Fig. 1(b)). In the Fast Fourier Transforms (FFTs) of our STM images, we show that there are no distortion, reconstruction or specially shaped defects, which can break crystal’s symmetry (Fig. S2, see Additional file 1). They are consistent with former findings [37–39]. Aside from the crystal lattice symmetry, electronic standing wave on the surface (Fig. 1(c)), which is resulted from the scattering of electronic state by local defects, establish QPI. QPI pattern collects the scattering vectors and display a C6 rotational symmetry on our C3 MnBi2Te4 crystal (Fig. 1(d)).

Figure 1
figure 1

Surface electronic structure of MnBi2Te4. (a) Crystal structure of MnBi2Te4. (b) Typical topography of cleaved MnBi2Te4 surface with atomic resolution (10 nm × 10 nm). (c) LDOS maps on a flat MnBi2Te4 terrace (225 mV, 100 nm × 100 nm) (d) FFT of (c) which showing the QPI pattern of MnBi2Te4. (e) and (f) averaged dI/dV spectra measured on the same area, with energy range from 225 mV to −180 mV for e, and from −200 mV to −500 mV for f. Black arrow points to the gap at the Fermi level. Blue arrow indicates the position of surface Dirac node (−320 mV). (g) Energy-dependent QPI line-cut along Γ-M direction (indicated by the dashed line in panel (d))

Full size image

In order to gain insight of the electronic structure on the MnBi2Te4 surface, we take the dI/dV spectra (Fig. 1(e) and (f)). Based on the “V” shape of the dI/dV curve, we are able to find the energy of the surface Dirac point at −320 mV (according to the bottom of the “V”). Combining with our previous results [27], we can estimate the valence band maximum and conduction band minimum to be located at −450 mV and −220 mV respectively. In addition, we resolve a dip in dI/dV curve at the Fermi level. After we perform a line cut on QPI patterns and plot the scattering vector-energy diagram in Fig. 1(g), one can find that the QPI signal diminishes at the 0 mV, indicating that a gap opens at the Fermi level (since we can identify QPI signal at other adjacent energy, other origins that prevent existed states from scattering can be excluded). It represents an overlooked feature of the MnBi2Te4 crystal.

A more intriguing phenomenon occurs when we undertake a comprehensive QPI examination (Fig. 2). We observe “normal” QPI patterns with six-folds rotational symmetry in the energy from 200 meV to −50 mV, which lays in the bulk conduction band of MnBi2Te4. However, in the energy range of the surface state and the high energy part of valence band, our data unambiguously show that the QPI patterns take a nematic-like shape, obeying C2 symmetry instead of C6 (rotational symmetry of high energy QPI obeys). Since QPI patterns originate from the scatterings between electron states, we can conclude that the electron state breaks the C3 rotational symmetry of the underline crystal. Interestingly, the C6 symmetric QPI recovers when we approach the electronic state deeply in the valence band.

Figure 2
figure 2

Rotational symmetry breaking QPIs of MnBi2Te4 surface. (a)-(p) a set of various energy layers of a dI/dV grid. (grid set point: −0.6 V 500 pA) clearly demonstrating the C6 and C2 symmetric QPI at different energies. For example, C6 QPI patterns are shown in (a), (b) and (p), C2 patterns can be remarkably observed in (h) to (m)

Full size image

We also perform a systemic QPI study on its cousin material MnBi4Te7 as a control experiment. In contrast to the fact that MnBi2Te4 contains magnetic atoms in every layer, MnBi4Te7 has both magnetic MnBi2Te4 layers and non-magnetic Bi2Te3 layers in its bulk structure (Fig. 3(a)). Our dI/dV spectra on surfaces of both layers of MnBi4Te7 (Fig. S1) agree with previous findings [37, 38]. But unlike MnBi2Te4, we don’t find any gaps at the Fermi levels of MnBi4Te7, which indicate the feature is not universal in the entire Mn-Bi-Te class. On the other hand, we do see a common unusual characteristic. Our QPI results show C6 symmetric patterns on the magnetic layer surface of MnBi4Te7 in both higher positive and negative voltages, but a C2 pattern at −400 mV (Fig. 3(b)-(e)). Notably, the QPI data in real space (Fig. 3(d)) display a distinct striped structure, which leads to the C2 pattern in reciprocal space. Surprisingly, on the non-magnetic layer surface of MnBi4Te7, we also find the rotational symmetry breaking QPI in a particular energy range (Fig. 3(f)-(i)). Once more, the C2 patterns are apparent in both real and reciprocal spaces (Fig. 3(h)). We combine the preceding results [40, 41] and our dI/dV spectra (Fig. S1) to determine the energies of bulk conduction and valence bands. We establish that the C2 electronic structure in MnBi4Te7 also appears in the energy range where surface state and upper part of the bulk valence band are situated, after carefully reviewing the systematic voltage-dependent QPI data in Figs. S3 and S4. We emphasis that our MnBi2Te4 and MnBi4Te7 samples were measured by two different STM apparatuses (see method section). It thus rules out the instrumental artificial effect as an interpretation of the C2 symmetry QPIs.

Figure 3
figure 3

C6 symmetry breaking QPIs on MnBi4Te7. (a) three-dimensional crystal structure of MnBi4Te7. Two different van de Waals layers can be distinguished, i.e. the upper layer containing Mn elements indicates MnBi2Te4 layer, the lower one without Mn atoms indicates Bi2Te3 layer. (b)-(e) LODS maps at indicated energies on MnBi2Te4 surface. Insets are the corresponding FFT. The C2 QPI patterns are clearly revealed at −400 mV in both real and reciprocal spaces. (f)-(i) same as (b)-(e) but measured on Bi2Te3 surface, the C6 symmetry breaking is discerned at energy of −320 mV

Full size image

Moreover, we carry out magnetic field dependent experiments. During the QPI measurement on MnBi2Te4, we have applied either 8T out-of-plane field or 2T in-plane field to the sample but find no noticeable difference in the C2 symmetric patterns (Fig. 4(a)-(c)). Previous result suggests that magnetic field of up to 7T is capable of aligning all spins out-of-plane [27], thus our field is strong enough to eradicate the possibility that the spin alignment may take part in breaking C3 symmetry. However, our QPI data still takes a nematic-like shape in the presence of Bz = 8T. Furthermore, we take the QPI measurements on the non-magnetic layer of MnBi4Te7 with applying in-plane fields along three Γ-M directions (Fig. 4(d)-(f)). The three cases show almost same C2 QPI images. If the QPI’s C2 relate to magnetic field (magnetic field can align all spin in plane, which can also induce C2 symmetry), the C2 axis should rotate along with magnetic field direction, since it doesn’t rotate, we thus believe that C2 symmetry is not caused by magnetic structure.

Figure 4
figure 4

QPI patterns under applied magnetic field. (a) QPI on a MnBi2Te4 sample at −240 mV, displaying C2 rather than C6 symmetry. (b) and (c) same as a but under an 8T out of plane and 2T in plane magnetic field, respectively. (d)-(f) QPI patterns on the Bi2Te3 termination of a MnBi4Te7 sample (−0.3 V 500 pA). 1T in plane magnetic fields are applied along three Γ-M directions. The directions of the C2 patterns do not change with magnetic fields

Full size image

Despite the fact that the Mn-Bi-Te class of materials has been viewed as a simple and ideal magnetic topological insulator, unexpected observations continue to emerge. A notable example is the experimental detected gapless Dirac cones on MnBi2Te4 and MnBi4Te7 surfaces, which should be gapped due to the time reversal symmetry breaking in these compounds. This issue deserves interests of the researchers. Indeed, theorists start using band theories together with various attempted magnetic configurations to solve the problem [16, 25, 28–31]. Some researchers speculate that whereas MnBi2Te4 and MnBi4Te7 bulks exhibit out-of-plane A-type AFM, the magnetic ordering on their surfaces may differ from the ideal case, i.e. the surface magnetization has an in-plane component [16, 37]. Up to our knowledge, we are not aware of any report that predicts or detects similar rotational symmetry breaking electronic state in the Mn-Bi-Te class of magnetic topological insulators. Following the logic of previous argument on the gap issue, it is natural to speculate that the suggested complex magnetism is the plausible driver of the C2 symmetric QPIs. However, the magnetism-driven electronic nematicity exhibits a reaction to an external applied field [42], which is contrast to our field dependent experimental results. It is also possible that the magnetic field fail to control nematic states, which is mainly because that the state energy is far from Fermi level. Within the experimental evidence, it is difficult to relate our C2 state to the magnetism of MnBi2Te4 and MnBi4Te7 samples. The physics of MnBi2Te4 family is still far from being understood even when we consider magnetic interaction. As a result, a theory that takes into account various interactions in these crystals is required.

Next, we want to discuss the nature of nematic-like states although they emerge in the energy range which includes a part below bulk VBMs of MnBi2Te4 and MnBi4Te7. According to the single-particle band simulation, we know that the surface state does not immediately merge into the valence band at the energy below the bulk VBM [13–16]. They stay away from each other in a large energy range. On the other hand, QPI will be dominated by surface standing wave even when surface state and bulk state coexist. It becomes possible that our uncovered C2 state mainly comes from the surface state. Therefore, it is reasonable to expect that a thorough solution of C2 electronic state can also provide a hitherto missed hint to the surface Dirac gap issue in MnBi2Te4 and MnBi4Te7.

As a summary, we discover a rotational symmetry breaking state in the magnetic topological insulators MnBi2Te4 and MnBi4Te7. C2 symmetric QPI patterns are clearly resolved at the energy range where surface state and high energy part of the bulk valence band are located. We find that these electronic states are resistant to external magnetic fields, proving that they are generated by a non-magnetic mechanism. Our findings suggest that the interaction and interplay in spin, orbit, charge and lattice degrees of freedom may play a substantial role in the magnetic topological insulators MnBi2Te4 and MnBi4Te7. Exotic emergent phenomena and innovative quantum devices are made conceivable by the coexistence of topological band and symmetry breaking order in a same sample.

3 Methods

The single crystalline samples of MnBi2Te4 and MnBi4Te7 are grown by flux-assisted method. The reactants MnTe and Bi2Te3 are mixed in the molar ratio MnTe: Bi2Te3 = 1:5.8 and 1:7.0 for MnBi2Te4 and MnBi4Te7 respectively, are placed into an alumina crucible and sealed by a quartz tube. We then put the sealed quartz tube into a muffle furnace and heat them up to 950°C in 24 hours. We keep this temperature for 12 hours to ensure that the materials melt homogenic. Finally, the sealed quartz tube was slowly cooled down to 578°C and 575°C for MnBi2Te4 and MnBi4Te7 respectively in 24 hours. We take the quartz tubes out from the muffle furnace rapidly and decant them with centrifuge to separate the shining plate-like crystals from the excess Bi2Te3 flux.

MnBi2Te4 samples are measured by a commercial STM (Unisoku 1300) in Suzhou Institute of Nano-Tech and Nano-Bionics. Our samples are cleaved in UHV circumstance (better than \(2\times 10^{-10}\text{ Torr}\)) at 80K. After cleaving, the sample is transferred to measurement stage within 20 mins, and then cooled down to 4.5K. All measurements are conducted at 4.5K, in UHV. Chemical etched Pt/Ir wire is used as STM tip, which is prepared by electron beam heating. Lock-in amplifier is applied to detect dI/dV signals, with modulation of 5 mV and 991 Hz. MnBi4Te7 samples are measured by Unisoku 1600 STM in Shanghai Jiao Tong university. The samples are cleaved in UHV (better than \(1\times 10^{-9}\text{ Torr}\)), but at room temperature. After being transferred to STM head, the samples are cooled down to 4.5K in 25 mins for STM measurements. The STM tips are etched tungsten wire followed by electron beam annealing. dI/dV data are acquired with lock-in amplifier with 5 mV and 991 Hz modulation.