INTRODUCTION

Investigations of the recently discovered antiferromagnetic (AFM) topological insulator (TI) MnBi2Te4 [1]—the first in a series of materials combining an intrinsic magnetic order and a nontrivial topology of the electronic band structure—have significantly advanced and enriched our understanding of quantum effects in a solid [27]. The crystal structure of this unique material is a sequence of septuple-layer blocks (SBs) Te–Bi–Te–Mn–Te–Bi–Te. In the ground state of MnBi2Te4, the magnetic moments on manganese atoms align up in the long-range AFM order of A-type [1]. The combination of time reversal symmetry and translation symmetry of the magnetic lattice makes it possible to classify MnBi2Te4 as a topologically nontrivial material with a three-dimensional invariant Z2 = 1 [8]. The violation of the combined symmetry on the (0001) surface of MnBi2Te4 should lead to the energy gap opening in the spectrum of the topological electronic state. The compatibility of the intrinsic magnetic order with the band topology in MnBi2Te4 serves as a basis for the realization of a number of phenomena in thin films of this material such as the quantum anomalous Hall effect (QAHE) [4, 5], the axion insulator phase (AI) [6] and other effects potentially important from the point of view of spintronic applications [27]. Moreover, in films with an even number of SBs placed in an external magnetic field, a transition from the AI state to the QAHE state is observed [9, 10].

The essential prerequisite for achieving the quantized conductivity regime is the presence of the exchange gap in the surface spectrum of the topological states, with the chemical potential fixed in the gap [2]. The first principles calculations predict a quite significant gap size up to 88 meV [1]. However, photoemission (ARPES) measurements of the surface states spectra in MnBi2Te4 give contradictory results. Some groups observe a gap in the surface electronic structure, although of different size in the range of several tens of meV [1113], some others see a gapless state [1418]. Many researchers are inclined to believe that the exchange gap modulation over wide range is associated with the structural imperfection of a surface area of the MnBi2Te4 samples under study [1113, 19, 20]. According to experimental and theoretical data [2123], antisite defects are present in this material: Mn atoms occupying sites in the Bi layers (MnBi), and Bi atoms replacing Mn atoms (BiMn) in the SB middle. In such a paradigm, a decrease in the gap size or a gap absence may be related to an increase in the concentration of BiMn and MnBi defects [11, 20, 24], which is determined by the conditions of sample preparation.

Another specific feature of the AFM TI of the MnBi2Te4 type is the relatively weak AFM coupling between the ferromagnetic (FM) layers of neighboring SBs and not strongly pronounced magneto-crystalline anisotropy along the easy axis perpendicular to the basic plane (0001). Therefore, an external magnetic field of moderate magnitude H < 5 T is able to rearrange the magnetic order, often provoking the formation of complex noncollinear textures both on the surface of thick/bulk samples [25] and in thin films [2630]. By studying the topological transition in an external field between the QAHE regime and the AI one in MnBi2Te4 films with a thickness of 4 to 8 SBs, the authors [10] found an inversion of the spectral gap in the region of the noncollinear phase. In [31], magneto-resistance fluctuations in MnBi2Te4 thin films are observed to increase under the metamagnetic transition between the AFM phase and the phase with a canted AFM order. Recently [32], the existence of bound electron states on MnBi antisite defects in the subsurface SB of MnBi2Te4 was shown by means of scanning tunneling microscopy and spectroscopy. It is noteworthy that, in the magnetization reversal process of samples, the local density of states near the MnBi defect changes drastically when the magnetic moments experience a spin-flop transition.

Understanding the behavior of topological electronic states in an exchange field with a complex orientation and spatial configuration is an important topic that has been developed in a number of theoretical investigations. In the articles [33, 34], the states induced by collinear magnetic domain walls with different textures on the TI surface were described in detail. In [35], the study of bound states was generalized to the case of noncollinear domain walls. The question of the spectral properties of surface and edge states, as well as domain wall states, in an intrinsic AFM TI film which is subject to transition from the AI regime to the QAHE one, was considered in [36]. In this article, we analytically explore the behavior of bound states formed near a short-range magnetic defect on the surface of AFM TI undergoing transformation from the AFM phase to the FM one under the influence of an external magnetic field. Such a task is very relevant, since it combines two aspects that are fundamental for the electronic properties of AFM TI of type MnBi2Te4—the noncollinear magnetization and intrinsic defects.

MODEL FOR THE SURFACE STATES OF THE INTRINSIC AFM TI

The crystal structure of the MnBi2Te4 compound along the growth direction ez is made up of SBs Te(1)–Bi(1)–Te(2)–Mn–Te(3)–Bi(2)–Te(4), between which weak van der Waals forces act. The topological properties are determined by four low-energy states formed of the \({{p}_{z}}\)-orbitals of atoms in the outer layers of SB: bonding \({\text{|Bi}}(1,2),\sigma \rangle \) and anti-bonding \({\text{|Te}}(1,4),\sigma \rangle \) combinations, where the index \(\sigma = \, \uparrow \downarrow \) denotes the projection of spin on the quantization axis ez. The matrix element A of the velocity operator mixes states with opposite parity and spin projection. In the basis \({{u}_{\Gamma }}\) formed of these four states, in the vicinity of the Γ point of the Brillouin zone, the kp Hamiltonian is given by [37]:

$${{{\text{H}}}_{t}}({\mathbf{k}}) = (\Xi - {\text{B}}{{k}^{2}}){{\tau }_{z}} \otimes {{\sigma }_{0}} + {\text{A}}{{\tau }_{x}} \otimes ({\boldsymbol{\sigma }} \cdot {\mathbf{k}}),$$
(1)

where σα and τβ (α, β = 0, x, y, z) denote the Pauli matrices in spin and orbital space, respectively, Ξ and B determine the energy gap and the band curvature in the spectrum of bulk states at k = 0. We use an isotropic version of the model, keeping the quadratic terms in momentum k = (kx, ky, kz). With a large enough spin–orbit coupling, the energy levels of the states \({\text{|Bi}}(1,2),\sigma \rangle \) and \({\text{|Te}}(1,4),\sigma \rangle \) are inverted near the \(\Gamma \) point, which is expressed via a condition \(\Xi {\text{B}} > 0\) corresponding to a nontrivial invariant \({{Z}_{2}} = 1\).

In the inner layers Te(2)–Mn–Te(3) of each SB, the moments of the \(3d\)-atoms are arranged in a ferromagnetic (FM) order due to super-exchange coupling. The magnetizations m(n) in neighboring SBs are directed antiparallel to each other: m(n) = –m(n ± 1), n is a SB index. The long-range AFM order of A-type with polarization orthogonal to the base plane (x, y), m(n) = (0, 0, \(m_{z}^{{(n)}}\)), \(m_{z}^{{(n)}}\) ~ (–1)n, is observed in the MnBi2Te4 bulk below the Néel temperature TN ≈ 24 K [1]. In known AFM TIs, the energy of magneto-crystalline anisotropy is relatively small, therefore an external magnetic field of moderate magnitude is able to reorient magnetic moments relative to the easy axis ez, sometimes provoking the formation of complex noncollinear textures [25]. We will consider only the -magnetization configurations homogeneous in the plane (x, y), where the components m(n) = (\(m_{x}^{{(n)}}\), \(m_{y}^{{(n)}}\), \(m_{z}^{{(n)}}\)) change from one SB to adjacent one. In addition to the Hamiltonian (1), we introduce the magnetic term Hex as the sum of the exchange energies of all SBs expressed in the basis \({{u}_{\Gamma }}\):

$${{{\text{H}}}_{{{\text{ex}}}}}(z) = c\sum\limits_n \left( {\begin{array}{*{20}{c}} {M_{z}^{{(n)}}}&{M_{ - }^{{(n)}}} \\ {M_{ + }^{{(n)}}}&{ - M_{z}^{{(n)}}} \end{array}} \right)\delta (z - {{z}_{n}}),$$
(2)

where \(M_{z}^{{(n)}} = {\text{diag}}({{J}_{1}},{{J}_{2}})m_{z}^{{(n)}}\), \(M_{ \pm }^{{(n)}} = {\text{diag}}({{J}_{3}},{{J}_{4}})m_{ \pm }^{{(n)}}\), \(m_{ \pm }^{{(n)}} = m_{x}^{{(n)}} \pm im_{y}^{{(n)}}\), δ(z) is the delta function, z is the spatial coordinate along the layer growth direction. The exchange potential is considered to be localized in the SB middle at \({{z}_{n}} = nc - {{z}_{0}}\), c is the SB thickness. Furthermore, we further imply that the magnetization amplitude is included in the matrix elements of the exchange integrals J1, 2, 3, 4, and is fixed, |m(n)| = 1. Thus, the vector m(n) determines only the orientation of the moments in nth SB.

Let’s consider a sample of the intrinsic AFM TI occupying a three-dimensional half-space z > 0. The topological electrons defined in (1) are influenced by the electrostatic surface potential (SP) Hb(r) [20, 38, 39], in addition to the exchange field (2). Taking into account the foregoing, we write the full functional of the electronic energy in the form

$$\Omega = \int {d{\mathbf{r}}{{\Theta }^{ + }}} ({\mathbf{r}})[{{H}_{t}}( - i\nabla ) + {{H}_{b}}({\mathbf{r}}) + {{{\text{H}}}_{{{\text{ex}}}}}(z)]h(z)\Theta ({\mathbf{r}}),$$
(3)

where spinor envelope functions Θ(r) describe low-energy states, h(z) is the Heaviside function, r = (xy, z). Assuming that SP is localized near the boundary of the sample over a characteristic length ~d, which is significantly shorter than the spatial scale of the decay of the envelope wavefunction Θ(r) ~ exp(–qz), dq \( \ll \) 1, one can use an approximation, Hb(r) = U(x, y)dδ(z). The SP spatial variations along the surface are associated with fluctuations in the c-oncentration of antisite defects [20, 24]. Besides we present the SP matrix in the diagonal form U = diag{U1, U2, U1, U2}. Summation in Hex(z) (2) is carried out over the SB index n = 1, 2, 3, …, where the count starts from the uppermost SB. We assume \({{z}_{0}} = c{\text{/}}2\), it means that the FM layer closest to the surface is located at a distance \(z = {{z}_{1}} = c{\text{/}}2\) from it.

SURFACE STATES OF AFM TI WITH NON-COLLINEAR MAGNETIZATION TEXTURE

Initially, we discuss the situation with an infinite and homogeneous surface in the plane (x, y), in other words, a surface with an averaged SP U = 〈U(x, y)〉 = const. In this case, varying the functional (3), we arrive at one-dimensional boundary problem:

$$[{{H}_{t}}(\boldsymbol{\kappa} , - i{{\partial }_{z}}) + {{{\text{H}}}_{{{\text{ex}}}}}(z) - E]\Theta (\boldsymbol{\kappa} ,z) = 0,$$
(4)
$$\left[ {\frac{{\delta {{H}_{t}}(\boldsymbol{\kappa} , - i{{\partial }_{z}})}}{{\delta ( - i{{\partial }_{z}})}} - 2dU} \right]\Theta (\boldsymbol{\kappa} ,z) = 0,$$
(5)

where momentum \(\boldsymbol{\kappa} = ({{k}_{x}},{{k}_{y}})\) plays the role of a parameter, \({{\partial }_{z}} = \partial {\text{/}}\partial z\). We are interested in the eigenstates of the problem given by Eqs. (4) and (5) that decay away from the surface, \(\Theta (\boldsymbol{\kappa} ,z \to \infty ) = 0\), and have energy inside the bulk band gap, \({{E}_{S}}(\boldsymbol{\kappa} ) < \Xi \). In general, it is hardly possible to find an exact solution to this problem. Therefore, we apply the perturbation theory procedure on a reduced basis [20, 38, 39], which allows us to transfer from the initial three-dimensional model of AFM TI in half-space to an effective two-dimensional Hamiltonian \({{H}_{S}}(\boldsymbol{\kappa} )\). The details of the procedure are described in [20]. Here we highlight the main points.

In the absence of the exchange field, Hex(z) = 0, Eqs. (4) and (5) admit an exact solution for \(\kappa = 0\). To simplify the calculations, we choose the following relation between the SP matrix elements: U1 = –U2 = U [20]. In this case, it is easy to determine the energy, \({{E}_{0}} = 0\), and the envelope wavefunction, Θ0(z) = \({{(1,i,1, - i)}^{t}}{{\theta }_{0}}(z)\), where t means the transpose operation. The spatial profile of the surface state

$${{\theta }_{0}}(z) = D\left[ {\exp ( - {{q}_{1}}z) + \frac{{\sqrt {\lambda - 1} - \tilde {U}}}{{\sqrt {\lambda - 1} + \tilde {U}}}\exp ( - {{q}_{2}}z)} \right],$$
(6)

significantly depends on the magnitude and sign of SP, where \(\tilde {U} = \frac{{d{{q}_{0}}U}}{\Xi }\), \({{q}_{0}} = \sqrt {\frac{\Xi }{{\text{B}}}} \), \(\lambda = \frac{{{{{\text{A}}}^{2}}}}{{4{\text{B}}\Xi }}\). In turn, the envelope decay scale ~\(q_{{1,2}}^{{ - 1}}\) is determined by the parameters of the Hamiltonian (1) as \({{q}_{{1,2}}} = {{q}_{0}}[\sqrt \lambda \pm \sqrt {\lambda - 1} ]\). The normalization constant D satisfies the condition \(2\int\limits_0^\infty dz\theta _{0}^{2}(z) = 1\). Without limiting the generality of consideration, we further focus on the case \(\lambda > 1\).

It is not difficult to make sure that the pair of spinors \({{\Phi }^{ \uparrow }}(z) = (1,{\kern 1pt} i,{\kern 1pt} 0,{\kern 1pt} {{0)}^{t}}{{\theta }_{0}}(z)\) and \({{\Phi }^{ \downarrow }}(z)\) = (0, 0, 1, \( - i{{)}^{t}}{{\theta }_{0}}(z)\) forms the orthonormal basis [20]. Calculating the matrix elements for the perturbations ~Hex and ~κ in this basis, we obtain the surface Hamiltonian

$$\begin{gathered} {{H}_{S}}(\boldsymbol{\kappa} ) = {\text{A}}({{k}_{x}}{{\sigma }_{y}} - {{k}_{y}}{{\sigma }_{x}}) \\ \, + {{J}_{{||}}}({{Q}_{x}}{{\sigma }_{x}} + {{Q}_{y}}{{\sigma }_{y}}) + {{J}_{z}}{{Q}_{z}}{{\sigma }_{z}}, \\ \end{gathered} $$
(7)

where longitudinal, Qx, y, and transverse, Qz, Q-factors are introduced as

$${{Q}_{{x,y,z}}} = 2c\sum\limits_{n = 1}^\infty m_{{x,y,z}}^{{(n)}}\theta _{0}^{2}({{z}_{n}}),$$
(8)

\({{J}_{z}} = \frac{{{{J}_{1}} + {{J}_{2}}}}{2}\), J|| = \(\frac{{{{J}_{3}} - {{J}_{4}}}}{2}\). The surface spectrum \(E_{S}^{ \pm }(\boldsymbol{\kappa} )\) = ±\(\sqrt {J_{z}^{2}Q_{z}^{2} + {{{({\text{A}}{{k}_{y}} - {{J}_{{||}}}{{Q}_{x}})}}^{2}} + {{{({\text{A}}{{k}_{x}} + {{J}_{{||}}}{{Q}_{y}})}}^{2}}} \) corresponding to the Hamiltonian (7) has the shape of the Dirac cone, which is shifted by the wave vector κ0 = \(\frac{{{{J}_{{||}}}}}{{\text{A}}}\)(–Qy, Qx) from the center of the Brillouin zone and has the energy gap \(2\Delta = 2{{J}_{z}}{{Q}_{z}}\) at the point \(\boldsymbol{\kappa} = {{\boldsymbol{\kappa} }_{0}}\). The gap \(2\Delta \) and displacement κ0 are of an exchange nature, however their magnitude and sign are determined by electrostatic conditions on the AFM TI surface according to the explicit functional dependence of the envelope function (6) on the SP strength U. Note that \(2\Delta \) and κ0 are directly related to the orientation of the magnetization in SBs, m(n), which can change in an external magnetic field.

For the prototype AFM TI MnBi2Te4, the penetration depth of the wavefunction of the topological surface state into the bulk material, \(\sim {\kern 1pt} q_{{1,2}}^{{ - 1}}\), is one or two SBs adjacent to the surface [11, 12, 19]. In the face of this fact, one may break the summation over n in (8) by holding the first two terms of the series. The contribution of the subsequent SBs with \(n \geqslant 3\) is negligible due to the exponential dependence (6). Assuming for certainty that \(m_{y}^{{(n)}} = 0\) and introducing the azimuth angle \({{\vartheta }_{n}}\) so that \(m_{x}^{{(n)}} = \sin {{\vartheta }_{n}}\) and \(m_{z}^{{(n)}} = \cos {{\vartheta }_{n}}\), we rewrite Eq. (8) in the reduced form

$$\begin{gathered} {{Q}_{{x,z}}}({{\vartheta }_{1}},{{\vartheta }_{2}}) \\ = 2c\left[ {\left\{ {\begin{array}{*{20}{c}} {\sin {{\vartheta }_{1}}} \\ {\cos {{\vartheta }_{1}}} \end{array}} \right\}\theta _{0}^{2}\left( {\frac{c}{2}} \right) + \left\{ {\begin{array}{*{20}{c}} {\sin {{\vartheta }_{2}}} \\ {\cos {{\vartheta }_{2}}} \end{array}} \right\}\theta _{0}^{2}\left( {\frac{{3c}}{2}} \right)} \right], \\ \end{gathered} $$
(9)

where the upper/lower line gives the orientation dependence of the longitudinal/transverse Q-factor. From a physical point of view, Eq. (9) correctly describes how the spin degree of freedom of the AFM TI surface state responds to the magnetization deviation from the collinear configuration. For each pair of angles \({{\vartheta }_{1}}\) and \({{\vartheta }_{2}}\), there are the energy exchange gap \(2\Delta = 2{{J}_{z}}{{Q}_{z}}({{\vartheta }_{1}},{{\vartheta }_{2}})\) and the momentum exchange displacement κ0 = ey\(\frac{{{{J}_{{||}}}}}{{\text{A}}}\)Qx1, ϑ2) in the surface state spectrum \(E_{S}^{ \pm }(\boldsymbol{\kappa} )\).

An external magnetic field H of a relatively small amplitude, that is directed along the easy axis of AFM TI, distorts weakly the fully compensated collinear texture of A-type inherent to the ground state. With growing field, at a critical value \(H = {{H}_{f}}\) a spin-flop transition occurs with a sharp turn of the sublattice moments. Then the moments are gradually oriented along the field direction H and finally are aligned strictly parallel when the saturation field Hs is reached. If the field H is directed perpendicular to the easy axis, the sublattice moments change orientation from AFM to FM gradually with the growth of the field in the entire range from 0 to Hs. This behavior of m(n)(H) has been experimentally found both in MnBi2Te4 with the easy axis ez [28, 30, 32, 40, 41] and in the planar AFM EuIn2As2, which has the easy plane (x, y) and is therefore considered as a candidate for AI [42].

Figure 1 presents the dependence of the transverse Q-factor on the orientation of the moments in the surface SBs, \({{Q}_{z}}({{\vartheta }_{1}},{{\vartheta }_{2}})\), for a special case with characteristic parameter values \(\tilde {U} = - 1\), \(\lambda = 1.1\) and c = 1. As the field H increases, the magnetic subsystem of AFM TI transforms from the equilibrium state at \(H = 0\) to the strictly FM state at \(H \geqslant {{H}_{s}}\), passing through a sequence of noncollinear textures \(\{ {{\vartheta }_{1}}(H),{{\vartheta }_{2}}(H)\} \). The optimal trajectory of such a transition, {ϑ1(H), \({{\vartheta }_{2}}(H{{)\} }_{O}}\), is determined by the energy minimum of the magnetic subsystem in space of possible configurations. Knowing the optimal trajectory, one can estimate the dependence of the gap on the external field, \(2\Delta (H)\).

Fig. 1.
figure 1

(Color online) Transverse Q-factor versus the magnetization orientation in the near-surface SBs of the intrinsic AFM TI with uniaxial anisotropy, presented by equidistant contour lines along which \({{Q}_{z}}({{\vartheta }_{1}},{{\vartheta }_{2}})\) takes constant values: (black lines) 0 and (green lines) ±0.2, ±0.4, ±0.6, and ±0.8. The function \({{Q}_{z}}({{\vartheta }_{1}},{{\vartheta }_{2}})\) increases monotonically as angle |ϑ1, 2| decreases and reaches a maximum in the FM phase at \({{\vartheta }_{{1,2}}} = 0\). The four optimal trajectories \({{\{ {{\vartheta }_{1}}(H),{{\vartheta }_{2}}(H)\} }_{O}}\) of the transition from the AFM phase to the FM phase with growing field H = Hez (H > 0) are shown in red: a continuous straight line \({{\vartheta }_{1}} = - {{\vartheta }_{2}}\) corresponds to the CAFM configuration existing in the interval \({{H}_{f}} < H < {{H}_{s}}\); dashed lines qualitatively depict the change in the magnetization texture in a weak field H < \({{H}_{f}}\); crosses indicate the spin-flop transitions. The two optimal trajectories of the transition from the AFM phase to the FM phase with \({{\vartheta }_{1}} = {{\vartheta }_{2}} = \frac{\pi }{2}\) in the field H = Hex (H > 0) are presented by blue and cyan lines. The dimensionless parameters are used: \(\tilde {U} = \frac{{dU}}{{\sqrt {{\text{B}}\Xi } }} = - 1\), \(\tilde {c} = c\sqrt {\frac{\Xi }{{\text{B}}}} \) = 1 and \(\lambda = 1.1\).

Let us consider in detail the most interesting case when an AFM with uniaxial anisotropy is exposed to an external field H = Hez. Magnetization measurements in the MnBi2Te4 samples [28, 30, 32, 40, 41, 43] show a sharp spin-flop transition at \(H = {{H}_{f}}\). Micromagnetic simulations for the MnBi2Te4 films performed in [28, 32] reproduce in the interval Hf < H < \({{H}_{s}}\) a magnetic phase with a canted texture of sublattice moments, for which we will use the abbreviation CAFM. Such a texture has uncompensated polarization along the external field, reaching saturation at \(H = {{H}_{s}}\). In our approach, it can be described as \({{\vartheta }_{1}}(H) = - {{\vartheta }_{2}}(H)\), where |sinϑ1(H)| ~ HsH. Figure 2 presents the dependence of the transverse Q-factor on the SP strength, \({{Q}_{z}}(U)\), for different values of the angle \({{\vartheta }_{1}}\), which sets the deviation of the moments from the direction of the external field in the CAFM phase. One can see that the gap, \(2\Delta (H)\), increases noticeably with decreasing angle \({{\vartheta }_{1}}\), although at the same time it can be either larger or smaller than the amplitude of the equilibrium gap \(2{\text{|}}\Delta (H = 0){\text{|}}\). In turn, the latter, \(2\Delta (H = 0)\), changes sign at the critical SP strength \(U = {{U}_{0}}\) (\({{\tilde {U}}_{0}} \approx 1.37\) under the specified parameters in Fig. 2), i.e., \({{Q}_{z}}({{U}_{0}})\). Thus, starting from the ground AFM state \(\{ {{\vartheta }_{1}} = 0,{\kern 1pt} {{\vartheta }_{2}} = \pm \pi \} \), if \(U > {{U}_{0}}\) (or {ϑ1 = \( \pm \pi ,{{\vartheta }_{2}}\) = 0}, if U < U0), in the region of weak fields at a certain value \(H_{t}^{{(1)}} < H_{f}^{{(1)}}\) (or \(H_{t}^{{(2)}} < H_{f}^{{(2)}}\)), the system passes through a noncollinear configuration \({{\{ {{\vartheta }_{1}}(H_{t}^{{(1)}}),{{\vartheta }_{2}}(H_{t}^{{(1)}})\} }_{O}}\) (or \({{\{ {{\vartheta }_{1}}(H_{t}^{{(2)}}),{{\vartheta }_{2}}(H_{t}^{{(2)}})\} }_{O}}\)), which corresponds to the gapless state of surface electrons with \(\Delta (H_{t}^{{(1)}})\) = 0 (or \(\Delta (H_{t}^{{(2)}} = 0\))). Obviously, \(H_{f}^{{(1)}} \ne H_{f}^{{(2)}}\) and \(H_{t}^{{(1)}} \ne H_{t}^{{(2)}}\). In other words, when in the ground state of AFM TI, the moments on the surface with \(U > {{U}_{0}}\) are parallel to the external field, m(1) (H = 0) \( \uparrow \uparrow \) H, or when \(U < {{U}_{0}}\) and the moments are antiparallel to the field, m(1) (H = 0) \( \downarrow \uparrow \) H, a surface topological transition can be observed, in which the Chern number, having the form \(C = {\text{sgn}}(\Delta ){\text{/}}2\) within the Hamiltonian HS(κ) (7), performs a quantized jump of a unit value. Indeed, in Fig. 1, two of the four optimal trajectories cross the line \({{Q}_{z}} = 0\) between the AFM phase and CAFM one (this part of the trajectories is indicated by dashed lines). The precise determination of the critical values \(H_{f}^{{(1,2)}}\) and \(H_{t}^{{(1,2)}}\), as well as configuration \({{\{ {{\vartheta }_{1}}(H_{t}^{{(1,2)}}),{\kern 1pt} {{\vartheta }_{2}}(H_{t}^{{(1,2)}})\} }_{O}}\), is a separate difficult task. Note that due to the degeneracy in energy, a pair of mirror-symmetric with respect to the axis ez trajectories corresponding to the textures \({{\{ {{\vartheta }_{1}}(H),{\kern 1pt} {{\vartheta }_{2}}(H)\} }_{O}}\) and \({{\{ - {{\vartheta }_{1}}(H),{\kern 1pt} - {{\vartheta }_{2}}(H)\} }_{O}}\) is possible. The formation of bound states of surface electrons is possible at the domain wall separating such textures [35, 36].

Fig. 2.
figure 2

(Color online) Evolution of the transverse \(Q\)-factor for AFM TI with uniaxial anisotropy caused by variation of the noncollinear magnetization texture from the AFM phase to the fully saturated FM phase. The angle \({{\vartheta }_{1}}\) for the canted magnetic texture takes the values \(\frac{\pi }{3}\) (brown line), \(\frac{\pi }{4}\) (green), \(\frac{\pi }{6}\) (blue) at fixed parameters: \(\lambda = 1.1\) and \(\tilde {c} = c\sqrt {\frac{\Xi }{{\text{B}}}} = 1\). In the ground state, the magnetization in SB (with n = 1) adjacent to the surface is directed parallel, \(m_{z}^{{(1)}} = 1\) (red line), or antiparallel, \(m_{z}^{{(1)}} = - 1\) (orange), to the applied external field H = Hez (Н > 0). The dimensionless variable \(\tilde {U} = \frac{{dU}}{{\sqrt {{\text{B}}\Xi } }}\) is deposited along the horizontal axis.

Now consider a situation where the planar AFM TI is placed in the field H = Hex. In the ground state, when \({{\vartheta }_{1}} = - {{\vartheta }_{2}} = \pm \pi {\text{/}}2\), and also in the collinear FM state, when \({{\vartheta }_{1}} = {{\vartheta }_{2}} = \pi {\text{/}}2\), the exchange gap is absent, \(\Delta = 0\). However, the external field in the interval \(0 < H < {{H}_{s}}\) provokes the formation of a noncollinear configuration with the output of magnetic moments from the base plane (x, y) and, accordingly, opens a spectral gap of finite size \(2{\text{|}}\Delta (H){\text{|}}\). As a result of the spin-flop transition, the CAFM phase occurs in the interval \({{H}_{f}} < H < {{H}_{s}}\), in which the magnetization can be described as \({{\vartheta }_{2}}(H) = \pi - {{\vartheta }_{1}}(H)\), where |cosϑ1(H)| ~ HsH. Using such a relationship between the angles, we describe in Fig. 3 the dependence \({{Q}_{z}}(U)\) for the canted magnetization texture at different angle values \({{\vartheta }_{1}}\). One can see that the exchange gap size \(2{\text{|}}\Delta (H){\text{|}}\) decreases with the field, but it can change the sign depending on the SP strength. Note also that there are two optimal trajectories with textures \({{\{ {{\vartheta }_{1}}(H),\pi - {{\vartheta }_{2}}(H)\} }_{O}}\) and \({{\{ \pi - {{\vartheta }_{1}}(H),{{\vartheta }_{2}}(H)\} }_{O}}\), which are mirror-symmetric with respect to the axis ex.

Fig. 3.
figure 3

(Color online) Evolution of the transverse \(Q\)-factor for AFM TI with an easy magnetic plane when the noncollinear magnetization texture varies from the AFM phase to the fully saturated FM phase. The angle \({{\vartheta }_{1}}\) for the canted magnetic texture takes the values π/3 (brown line), π/4 (green), π/6 (blue) at fixed parameters: \(\lambda = 1.1\) and \(\tilde {c} = c\sqrt {\frac{\Xi }{{\text{B}}}} = 1\). In both AFM and FM phases \({{Q}_{z}} = 0\) (red horizontal line). The dimensionless variable \(\tilde {U} = \frac{{dU}}{{\sqrt {{\text{B}}\Xi } }}\) is deposited along the horizontal axis.

The behavior of the Q-factor with the SP strength in the phase with the CAFM order, which occurs at \(0 < H < {{H}_{s}}\) when the uniaxial AFM TI is placed in the field H = Hex (optimal trajectories are shown in Fig. 1), or when the planar AFM TI is placed in the field H = Hez, can easily be understood from Fig. 3 or Fig. 2, respectively. It is also not difficult to estimate the momentum shift of the spectrum \(E_{S}^{ \pm }(\boldsymbol{\kappa} )\) as a function of the SP strength, κ0(U), by paying attention to the fact that \({{Q}_{x}}({{\vartheta }_{1}},{{\vartheta }_{2}}) = {{Q}_{z}}({{\vartheta }_{1}} - \pi {\text{/}}2,{{\vartheta }_{2}} - \pi {\text{/}}2)\) according to equation (9).

A SHORT-RANGE DEFECT ON THE SURFACE OF THE AFM TI WITH A NON-COLLINEAR MAGNETIZATION TEXTURE

We proceed to the study of how a defect forms a bound electronic state on the surface of AFM TI with a noncollinear magnetization texture. To do this, one has to solve the equation

$$\begin{gathered} \left[ {{{H}_{S}}\left( {{{k}_{x}} \to - i\frac{\partial }{{\partial x}},{{k}_{y}} \to - i\frac{\partial }{{\partial y}}} \right) + {{H}_{D}}(\boldsymbol{\rho} )} \right] \\ \times \;\left( \begin{gathered} {{f}_{1}} \hfill \\ {{f}_{2}} \hfill \\ \end{gathered} \right) = {{E}_{D}}\left( \begin{gathered} {{f}_{1}} \hfill \\ {{f}_{2}} \hfill \\ \end{gathered} \right), \\ \end{gathered} $$
(10)

where, in addition to the surface Hamiltonian HS(κ) (7), a term HD(ρ) describing the perturbation of SP generated by a defect in a small neighborhood of the point ρ = (x, y) = 0 is introduced. The solution of Eq. (10) for the spinor function f1((ρ), f2(ρ))t must decay away from the defect, f1, 2(ρ) → 0 at \(\rho \to \infty \). The corresponding energy level ED should lie inside the exchange gap, \({\text{|}}{{E}_{D}}{\text{|}} < {\text{|}}\Delta {\text{|}}\).

In order to adequately take into account the presence of the antisite defect MnBi or BiMn in the surface region of the AFM TI MnBi2Te4, which is associated with the redistribution of charge and spin density, we model the interaction of topological electrons with the defect in the form HD(ρ) = [\({v}\)σ0 + (μσ)]g(ρ). This interaction is assumed to be isotropic in the plane (xy), g(ρ) = g(ρ). The component ~\({v}\) is related to the potential scattering of electron by a single defect. Another spin-dependent component implies the presence of magnetic moment μ = (μx, μy, μz), and the strength of the exchange coupling with the defect is included in the magnitude μ = |μ|. The specificity of our consideration is to preserve the finite radius ρ0 of the defect action [44, 45]. In a two-dimensional system, the strength of the potential, which is modeled using the delta function (i.e., when \({{\rho }_{0}} \to 0\)), is not enough to bind the particle. Following [44, 45], we use the Gaussian \(g(\rho ) = \exp \left( { - \frac{{{{\rho }^{2}}}}{{\rho _{0}^{2}}}} \right)\) for the spatial distribution of the perturbation.

One supposes that the scale of change of the bound state significantly exceeds the radius \({{\rho }_{0}}\). Within the framework of the accepted assumptions, we obtain explicitly (we omit cumbersome calculations) the envelope wavefunction outside the area of the defect action, that is, when \(\rho > {{\rho }_{0}}\):

$$\begin{gathered} {{f}_{{1,2}}}(\boldsymbol{\rho} ) = \frac{{\rho _{0}^{2}}}{{2{{{\text{A}}}^{2}}}}\{ [({{E}_{D}} \pm \Delta )({v} \pm {{\mu }_{z}}){{K}_{0}}({\text{P}}) \\ + \;i\sqrt {{{\Delta }^{2}} - E_{D}^{2}} {{K}_{1}}({\text{P}}){{e}^{{ \pm i\eta }}}{{\mu }_{ \pm }}]{{f}_{{1,2}}}(0) \\ \, + [({{E}_{D}} \pm \Delta ){{\mu }_{ \mp }}{{K}_{0}}({\text{P}}) + i\sqrt {{{\Delta }^{2}} - E_{D}^{2}} {{K}_{1}}({\text{P}}){\kern 1pt} {{e}^{{ \pm i\eta }}}({v} \mp {{\mu }_{z}})] \\ \times \;{{f}_{{2,1}}}(0)\} \exp (i{{\kappa }_{0}}y), \\ \end{gathered} $$
(11)

where the upper sign + or – on the right side of the equation correlates with the spinor component f1 or f2, respectively; K0(P) and K1(P) are the modified zero- and first-order Bessel functions [46] of the dimensionless argument \({\text{P}} = \frac{{\rho \sqrt {{{\Delta }^{2}} - E_{D}^{2}} }}{{\text{A}}}\), κ0 = \(\frac{{{{J}_{{||}}}{{Q}_{x}}}}{{\text{A}}}\) (here it means that Qy = 0), \({{\mu }_{ \pm }} = {{\mu }_{x}} \pm i{{\mu }_{y}}\), \(\tan \eta = x{\text{/}}y\). In the plane (x, y), the function f1, 2(ρ) experiences oscillation with a period \(2\pi /{{\kappa }_{0}}\) along the direction orthogonal to the longitudinal component of the resulting exchange field, and decreases as P–1exp(–P) with moving away from the defect, where P \( \gg \) 1. The envelope amplitude on the defect \({{f}_{{1,2}}}(0)\) is determined by the normalization condition, and the relationship between \({{f}_{1}}(0)\) and \({{f}_{2}}(0)\) stems from the system of equations:

$$\begin{gathered} {{f}_{{1,2}}}(0) = \mp [({v} \pm {{\mu }_{z}}){{f}_{{1,2}}}(0) + {{\mu }_{ \mp }}{{f}_{{2,1}}}(0)] \\ \times \;\frac{{\Delta + {{E}_{D}}}}{{4{{{\text{A}}}^{2}}}}\rho _{0}^{2}\ln \left( {\frac{{{{\Delta }^{2}} - E_{D}^{2}}}{{4{{{\text{A}}}^{2}}}}\rho _{0}^{2}} \right). \\ \end{gathered} $$
(12)

The equations (12) are written basing on the applicability of the proposed approach when the condition \(\frac{{\rho _{0}^{2}}}{4}\frac{{{{\Delta }^{2}} - E_{D}^{2}}}{{{{A}^{2}}}} \ll 1\) is fulfilled. The secular equation arising from (12) determines the energy levels of the bound states ED depending on \({v}\), μ and \(\Delta \). If the magnetic moment of the defect has only a normal component, μ = ezμz, the system (12) decomposes into two independent equations. The first of them gives a so-lution f1(ρ) ≠ 0 and f2(ρ) = 0 with energy \(E_{D}^{{(1)}}\), existing under the condition \(({v} + {{\mu }_{z}})\Delta > 0\). The second one gives a solution f1(ρ) = 0 and f2(ρ) ≠ 0 with  energy \(E_{D}^{{(2)}}\) if \(({v} - {{\mu }_{z}})\Delta < 0\). The energies \(E_{D}^{{(1)}}\) and \(E_{D}^{{(2)}}\) as functions of the defect potential \({v}\) transform into each other being inverted relative to the origin, \(E_{D}^{{(2)}}({v}) \leftrightarrow - E_{D}^{{(1)}}( - {v})\). In the case of a non-magnetic defect (μz = 0) for all values of \({v}\), there is only one bound state inside the gap. Figure 4 shows the dependence \(E_{D}^{{(1)}}({v})\) = ED(\({v}\)) in the case \({{\mu }_{z}} = 0\) and \({v} > 0\) for some values \(\Delta > 0\). One can see that the bound state energy ED(\({v}\)) varies greatly with the exchange gap size. Figure 5 demonstrates the dependence \({{E}_{D}}({v})\) when the magnetic moment on the defect is switched on. The component \({{\mu }_{z}}\) shifts branches \(E_{D}^{{(1)}}\) and \(E_{D}^{{(2)}}\) along the horizontal axis in opposite directions without changing their shape. Thus: when \({{\mu }_{z}} < 0\), any bound state is missing in the interval \({\text{|}}{v}{\text{|}} < {\text{|}}{{\mu }_{z}}{\text{|}}\); in turn, when \({{\mu }_{z}} > 0\) a pair of the levels of the same depth and opposite sign are present in the interval \({\text{|}}{v}{\text{|}} < {\text{|}}{{\mu }_{z}}{\text{|}}\). If there is an in-plane projection of the moment, \({{\mu }_{x}} \ne 0\), the system (12) has two solutions with a pair of constituents f1(ρ) ≠ 0 and f2(ρ) ≠ 0, which correspond to two branches of the bound states. As follows from Fig. 5, a relatively large strength of the spin-dependent scattering ~μx, z is required to noticeably affect the energy levels inside the gap.

Fig. 4.
figure 4

(Color online) Binding energy of the electronic state versus the potential scattering strength by the non-magnetic short-range defect at different values of the exchange gap in the surface state spectrum of AFM TI. The dimensionless values are used: \({{\tilde {E}}_{D}} = \frac{{{{E}_{D}}{{\rho }_{0}}}}{{2{\text{A}}}}\), \({\tilde {v}} = \frac{{{v}{{\rho }_{0}}}}{{2{\text{A}}}}\), \(\tilde {\Delta } = \frac{{\Delta {{\rho }_{0}}}}{{2{\text{A}}}}\), where \(\tilde {\Delta }\) = (black line) 0.2, (red line) 0.1, (blue line) 0.05, (brown line) 0.02, and (green line) 0.01.

Fig. 5.
figure 5

(Color online) Binding energy of the electronic state versus the potential scattering strength \({v}\) at different orientations of the magnetic moment \({\boldsymbol{\mu }} = ({{\mu }_{x}},0,{{\mu }_{z}})\) of a short-range defect on the AFM TI surface, namely: (green lines) \({{\tilde {\mu }}_{x}} = 0\), \({{\tilde {\mu }}_{z}} = 3\); (brown lines) \({{\tilde {\mu }}_{x}} = 0\), \({{\tilde {\mu }}_{z}} = - 3\); (blue lines) \({\text{|}}{{\tilde {\mu }}_{x}}{\text{|}} = 1\), \({\text{|}}{{\tilde {\mu }}_{z}}{\text{|}} = \sqrt 8 \); and (red lines) \({\text{|}}{{\tilde {\mu }}_{x}}{\text{|}} = 3\), \({\text{|}}{{\tilde {\mu }}_{z}}{\text{|}} = 0\). For comparison, the black lines present the energy levels for a non-magnetic defect with \({{\mu }_{x}} = {{\mu }_{z}} = 0\). The dimensionless values are used: \({{\tilde {E}}_{D}} = \frac{{{{E}_{D}}{{\rho }_{0}}}}{{2{\text{A}}}}\), \({\tilde {v}} = \frac{{{v}{{\rho }_{0}}}}{{2{\text{A}}}}\), \({{\tilde {\mu }}_{{x,z}}} = \frac{{{{\mu }_{{x,z}}}{{\rho }_{0}}}}{{2{\text{A}}}}\), \(\tilde {\Delta } = \frac{{\Delta {{\rho }_{0}}}}{{2{\text{A}}}}\), where \(\tilde {\Delta } = 0.1\).

If the localization radius of the bound state, ~A/\(\sqrt {{{\Delta }^{2}} - E_{D}^{2}} \), exceeds the average distance between the nearest defects, the single defect approximation becomes incorrect. Based on experimental estimations [24], the MnBi antisite defects on the MnBi2Te4 surface can be considered as non-interacting at concentrations below 4%.

DISCUSSION

According to the above theoretical arguments, the variation of the position of the bound state level on the MnBi antisite defect in the AFM TI MnBi2Te4 under the influence of an external magnetic field H can be represented as

$$\frac{{d{{E}_{D}}}}{{d{\mathbf{H}}}} = \frac{{\partial {{E}_{D}}}}{{\partial {{{\mathbf{m}}}^{{(n)}}}}}\frac{{d{{{\mathbf{m}}}^{{(n)}}}}}{{d{\mathbf{H}}}} + \frac{{\partial {{E}_{D}}}}{{\partial {\boldsymbol{\mu }}}}\frac{{d{\boldsymbol{\mu }}}}{{d{\mathbf{H}}}}.$$
(13)

Here, the first term arises due to the rearrangement of the magnetic order m(n), the second term is due to the reorientation of the moment on the MnBi defect. In a field of moderate magnitude (up to ~10 T), these two processes correlate with each other, \(\frac{{d{{{\mathbf{m}}}^{{(n)}}}}}{{d{\mathbf{H}}}}\)\(\frac{{d{\boldsymbol{\mu }}}}{{d{\mathbf{H}}}}\), since in every SB the moments on the MnBi, ~μ, defects exhibit a strong AFM coupling with the magnetization m(n) of the central layer, that is, μ \( \uparrow \downarrow \) m(n) [43]. Moreover, the Mn moments in the middle of the SB show actually a ferrimagnetic order due to magnetic vacancies on the MnBi defects, therefore, a reduced average moment from \(2.66{{\mu }_{B}}\) [32] to \(3.9{{\mu }_{B}}\) [43] per Mn atom is observed in magnetization measurements. It is the samples with ferrimagnetic layers in the SBs that are subject to spin-flop transition at \({{H}_{f}} \approx 3{-} 4{\kern 1pt} \) T and parallel alignment at \({{H}_{s}} \approx 6{-} 9\) T [1, 10, 28, 30, 40, 43]. Full polarization in MnBi2Te4, when all moments— both on Mn atoms and on MnBi defects—are directed along the field and show the average value \(4.6{{\mu }_{B}}\) per Mn atom, is achieved in very large fields \(H \approx 60{\kern 1pt} \) T [43]. The level position \({{E}_{D}}\), as described above (Fig. 5), responds weakly to a change in the moment direction μ. In addition, the amplitude μ, as a rule, does not exceed the potential scattering strength \({v}\). On the other hand, we have shown that the magnetization reorientation in surface SBs (at fixed \({v}\) and μ) can significantly affect the exchange gap size (Fig. 2). In turn, \(\frac{{\partial {{E}_{B}}}}{{\partial {{{\mathbf{m}}}^{{(n)}}}}}\) = \(\frac{{\partial {{E}_{D}}}}{{\partial \Delta }}\frac{{d\Delta }}{{d{{{\mathbf{m}}}^{{(n)}}}}}\), where the change in the energy of the state on the defect is directly related to the change in the gap size, \({\text{|}}{{E}_{D}}{\text{|}} < \Delta \) (Fig. 4). Thus, the main contribution to \(\frac{{d{{E}_{D}}}}{{d{\mathbf{H}}}}\) is made by the change in the moment orientation in the central layers of the SBs m(n) (the first term in (13)), which is most pronounced in the noncollinear phase in the vicinity of the critical field Hf. At the same time, the value of the spin-flop transition field is determined by the trajectory of the transition from the AFM phase to the FM phase (or vice versa) and therefore can take different values, \(H_{f}^{{(1)}}\) or \(H_{f}^{{(2)}}\), depending on the mutual orientation of the sublattices moments m(n) at H = 0 and the applied field H = Hez.

The bound state generates a sharp peak in the density of states centered at energy ED inside the gap, which should manifest itself in tunneling conductivity near the Fermi level. In [32], low-temperature scanning microscopy and spectroscopy on the (0001) surface of the AFM TI MnBi2Te4 below \({{T}_{N}}\) revealed the formation of the bound electronic states localized at a scale of ~2.0 nm near the MnBi antisite defects. The authors studied the evolution of the local density of states near the defect with increasing external magnetic field perpendicular to the surface from –8 to 8 T and found a drastic change in the tunneling spectra in a narrow neighborhood (width of δH ~ 2 T) of two critical fields –2.0 and 4.0 T, which are associated with the surface spin-flop transitions. In the rest of the research area, which is occupied by the collinear FM and AFM phases, the change in tunnel spectra turned out to be insignificant. The theoretical approach developed above essentially suggests the physical mechanism for the non-trivial phenomenon discovered in [32] and reproduces its main features on a qualitative level.

CONCLUSIONS

In this work, we analytically investigated not only the surface topological states, but also the bound states on the defect inherent in the intrinsic AFM TI with a noncollinear magnetization texture. Our approach is based on the fact, specific to the MnBi2Te4-type intrinsic AFM TIs, that the length of the penetration of the surface topological state into the material is commensurate with the AFM order period, and the profile of the electron and spin density depends on the surface electrostatic potential. Therefore, the surface spin-flop transition in the magnetic subsystem, provoked by an external field, manifests in the principal characteristics of the topological state, in particular, in the exchange gap. We have shown that the electronic response of the system to a short-range disturbance of the surface potential displays itself as the formation of the bound state, the energy level of which is determined by the exchange gap size and the disturbance strength. Our results make it possible not only to interpret recent experiments in a consistent manner, but also reveal new aspects of the relationship between magnetic order and nontrivial electronic states in AFM TIs.