Three-Dirac-fermion approach to unexpected universal gapless surface states in van der Waals magnetic topological insulators

Layered van der Waals (vdW) topological materials, especially the recently discovered MnBi 2 Te 4 -family magnetic topological insulators (TIs), have aroused great attention. However, there has been a serious debate about whether the surface states are gapped or gapless for antiferromagnetic (AFM) TI MnBi 2 Te 4 , which is crucial to the prospect of various magnetic topological phenomena. Here, a minimal three-Dirac-fermion approach is developed to generally describe topological surface states of non-magnetic / magnetic vdW TIs under the modulation of the interlayer vdW gap. In particular, this approach is applied to address the controversial issues concerning the surface states of vdW AFM TIs. Remarkably, topologically protected gapless Dirac-cone surface states are found to arise due to a small expansion of the interlayer vdW gap on the surface, when the Chern number equals zero for the surface ferromagnetic layer; while the surface states remain gapped in all other cases. These results are further conﬁrmed by our ﬁrst-principles calculations on AFM TI MnBi 2 Te 4 . The theorectically discovered gapless Dirac-cone states provide a unique mechanism for understanding the puzzle of the experimentally observed gapless surface states in MnBi 2 Te 4 . This work also provides a promising way for experiments to realize the intrinsic magnetic quantum anomalous Hall e ﬀ ect in MnBi 2 Te 4 ﬁlms with a large energy gap. magnetic topological insulator, van der Waals gap, Dirac-fermion model, gapless surface state


I. INTRODUCTION
Three-dimensional topological insulators (TIs) are characterized by gapless Dirac-cone surface states in the bulk band gap [1,2].However, breaking the time-reversal symmetry (TRS) leads to an energy gap at the Dirac point of these surface states [3].Such gapped Diraccone surface states play a crucial role in a series of exotic topological quantum phenomena [4], including the topological magnetoelectric effect [5,6] and the quantum anomalous Hall effect (QAHE) [7,8].While magnetic doping has conventionally been used to induce gapped Dirac-cone surface states in TIs [8], this method is inevitably subject to complex and detrimental effects, such as magnetic inhomogeneity and disorder effects.Recently, MnBi 2 Te 4 -family intrinsic magnetic TIs were discovered , which offers a promising alternative playground.These materials not only increase the temperature of the QAHE [46], but also enable the study of axion electrodynamics [47][48][49][50][51][52].However, there is still a significant controversy on the surface states of MnBi 2 Te 4 , as both gapped and gapless surface states were experimentally observed [10,[53][54][55][56][57][58][59].
Meanwhile, most nonmagnetic/magnetic TIs are layered materials with an interlayer van der Waals (vdW) gap, such as Bi 2 Se 3 -family TIs [60] and MnBi 2 Te 4 -family magnetic TIs [9][10][11][12].These materials are composed of covalent-bonding layers held together by weak vdW forces.The interlayer vdW gap plays a crucial role in the electronic structures of both bulk and surface states of these materials.However, the vdW gap is sensitive to impurities or intercalated atoms in fabrication processes, and even a small concentration of impurities can significantly expand the vdW gap.Moreover, the surfaceinduced symmetry breaking may also cause a considerable expansion of the topmost vdW gap.Previous studies have shown that an expansion of the topmost vdW gap can lead to a relocation of surface states in Bi 2 Se 3 -family TIs [61].
In this work, we develop a three-Dirac-fermion model that can generally describe the surface states of layered nonmagnetic/magnetic TIs under the modulation of the interlayer vdW gap.We apply this model to investigate the evolution of the surface states of layered Atype AFM TIs [62], as the topmost vdW gap expands from the bulk value (d = d 0 ) to the surface layer decoupled limit (d = ∞), as illustrated in Fig. 1.The surface layer can be regarded as either an effective ferromagnetic (FM) block consisting of an odd number of septuplelayers (SLs) of MnBi 2 Te 4 [11], as schematically shown in Fig. 1(i,ii), or an effective AFM block consisting of an even number of SLs of MnBi 2 Te 4 , as schematically shown FIG.1: Evolution of surface states of an AFM TI with an expansion of the topmost vdW gap.The AFM TI consists of FM or AFM block layers (colored in blue) held together by weak vdW forces.In the unexpanded case (d0), the AFM TI has gapped Dirac-cone surface states.With increasing the topmost vdW gap (d = d0 + δd) between a surface layer (colored in pink) and the below bulk (colored in blue), if the surface layer is FM, two cases will occur, depending on its Chern number C: (i) When |C| = 1, the surface states remain gapped.(ii) When C = 0, the energy gap of surface states has a topologically protected gapless transition point.In contrast, (iii) if the surface layer is AFM, the surface states always keep gapped.
in Fig. 1(iii).For the FM surface layer, we find that if the surface layer has a zero Chern number, unexpected topologically protected gapless Dirac-cone surface states will arise at the topological transition point [see (ii) in Fig. 1], while the gapped surface states persist if the surface layer has a nonzero Chern number [see (i) in Fig. 1].On the other hand, for the AFM surface layer, the energy gap of surface states is always maintained [see (iii) in Fig. 1].These findings are further confirmed by our first-principles calculations on AFM TI MnBi 2 Te 4 , which may solve the puzzle of the angle resolved photoelectron spectroscopy (ARPES) observations of both gapless and gapped surface states of MnBi 2 Te 4 .Notably, the three-Dirac-fermion approach provides a unified description of surface states of layered TIs, indicating that engineering the interlayer vdW gap can provide a new route for developing the applications of nonmagnetic/magnetic TIs.

II. EFFECTIVE MODEL ANALYSIS
The surface states of the nonmagnetic/magnetic TIs under an expansion of the interlayer vdW gap can be captured by a simple low-energy effective model composed of three helical Dirac fermions.As illustrated in Fig. 2(a), two of the Dirac fermions, labelled as D 1 and D 2 , re-spectively, come from top and bottom surface states of the surface layer.The third Dirac fermion, labelled D 3 , comes from the top surface state of the below bulk.Normally, D 1 , D 2 , and D 3 are local Dirac states with a typical spread (1 ∼ 2 nm).With magnetic moments along the out-of-plane direction (the z direction), each Dirac fermion can be described by the low-energy Hamiltonian where i = 1, 2, 3, v is the Fermi velocity, s = +1(−1) denotes the helicity of D 1 and D 3 (D 2 ), the Pauli matrices act on the spin subspace, and h i indicates the Zeeman coupling.We further consider two couplings due to finite wavefunction overlaps: ∆ where σ 0 is 2 × 2 identity matrix in the spin subspace.
We first consider nonmagnetic TIs to see how surface states change with an expansion of the topmost vdW gap (d).With increasing d from the pristine bulk value (d = d 0 ), ∆ 23 gradually decreases to zero, while ∆ 12 remains unchanged.Therefore, we treat ∆ 12 as the energy unit hereinafter.Without the Zeeman term, the nonmagnetic three-Dirac-fermion model exhibits two linear bands with dispersions ±vk, where k = (k 2 x + k 2 y ).Regardless of ∆ 23 /∆ 12 , the two linear bands always meet at Γ, forming a gapless Dirac cone, as reflected from the doubly degenerate energy level at shown in Fig. 2(b).However, the location of the gapless Dirac cone gradually changes from the top of the surface layer to the top of the below bulk [see Fig. 2(a)].The persistent gapless Dirac cone in the nonmagnetic case can be understood from the destructive interference behavior in a three-level system.
When A-type AFM order is present, each of the three Dirac fermions becomes gapped due to a mass term induced by the Zeeman coupling.Interestingly, the competition between the Zeeman coupling h and the intralayer coupling ∆ 12 leads to distinct results.We first consider the surface layer as one FM block with h 1 = h 2 = −h 3 = h (where h denotes the Zeeman coupling strength assumed to be positive).Intriguingly, if the condition ∆ 12 > h is satisfied, for example, for a sufficiently thin surface layer, there is always an energy-gap-closing-andreopening process with a gapless Dirac-cone surface state, when the interlayer coupling ∆ 23 is tuned by modulating the vdW gap, as shown in Fig. 2  ) correspond to the top and bottom surface states of the surface layer, and D3 comes from the top surface states of the below bulk.∆12 is the intralayer coupling between D1 and D2, while ∆23 is the interlayer coupling between D2 and D3.(b,c,e,f) Bands at Γ as a function of ∆23/∆12, corresponding to increasing the interlayer vdW gap, for the nonmagnetic case (b) and the AFM case hosting an AFM surface layer (c) and an FM surface layer with ∆12 > h (e) and with ∆12 < h (f).The h represents the Zeeman coupling strength.Fig. 2(f).We will show that the gapless surface state is topologically protected by the Chern number transition of the three-Dirac-fermion system from the weakly coupled limit of ∆ 23 → 0 to the strongly coupled limit of ∆ 23 ≫ ∆ 12 .
In the weakly coupled limit of ∆ 23 → 0, only D 1 and D 2 are coupled through ∆ 12 , while D 3 is nearly isolated with the Chern number , and correspondingly, the total Chern number is C = C 12 + C 3 = −1/2 (C = 1/2).In the strongly coupled limit with ∆ 23 ≫ ∆ 12 , and ∆ 23 ≫ h, D 1 becomes nearly isolated, and only D 2 and D 3 are coupled.Because of the P T symmetry combining the spatial-inversion operation (P ) and the time-reversal operation (T ) [see the supplementary material (SM) [63] for details], the coupled D 2 and D 3 give a zero Chern number.Therefore, the total Chern number in the strongly coupled limit is equivalent to that of D 1 , given by C 1 = sgn(h 1 )/2 = 1/2.It follows that if ∆ 12 > h is satisfied, the total Chern number of the three-Dirac-fermion system changes by |∆C| = 1 from the weakly coupled limit to the strongly coupled limit in the vdW gap expansion process, ensuring the existence of a topological transition shown in Fig. 2(e).Differently, for ∆ 12 < h, the total Chern number remains unchanged and the energy gap of surface states stays open [see Fig. 2(f)].Therefore, we can conclude that the gapless surface state is topologically protected and arises from the competition between the Zeeman coupling and the Dirac fermion couplings.
Secondly, we consider an AFM surface layer with h 1 = −h 2 = h 3 = h.In this case, there is no gapless transition with the interlayer vdW gap expansion, shown in Fig. 2(c).Instead, the energy gap of the surface state remains an almost constant magnitude 2h, irrespective of ∆ 23 /∆ 12 .Moreover, the total Chern number remains unchanged as C = 1/2, which can be obtained through similar arguments as above by taking two limits of ∆ 23 ≪ ∆ 12 and ∆ 23 ≫ ∆ 12 into account.
Furthermore, it is worth mentioning that the three-Dirac-fermion model can also be used to describe the FM TIs, for example, by setting h 1 = h 2 = h 3 = h.However, in contrast to the A-type AFM TIs, in the FM case, a topological transition with a gapless surface state appears only when ∆ 12 < h is satisfied (see SM for more details [63]).MnBi 2 Te 4 .It is composed of FM SLs that are coupled to each other through the vdW force, and it exhibits an A-type AFM order in the magnetic ground state, with out-of-plane FM coupling within each SL and AFM coupling between neighboring SLs [11].However, the existence of unavoidable Mn Bi and Bi Mn antisite defects and Mn vacancies might lead to significant changes of the vdW gap [18,30,64,65].The mechanical cleavage and exfoliation processes of the sample preparation, and the symmetry suddenly breaking on the surface may also result in a small expansion of the topmost interlayer vdW gap [64].Now, based on first-principles calculations, we investigate the expansion effects of the topmost vdW gap of MnBi 2 Te 4 .

III. MATERIAL REALIZATION
We start from the nonmagnetic case of CaBi 2 Te 4 where Mn atoms in MnBi 2 Te 4 are replaced with Ca atoms.For our calculations, we take a (7 + 1)-SL CaBi 2 Te 4 thin film with gradually expanding the topmost vdW gap between the first and second SL.The evolution of energy bands at Γ with increasing δd = d − d 0 is presented in Fig. 3(a), where the bands locating at the first and second SLs are highlighted in blue.The blue bands at the Fermi level are doubly degenerate, indicating the persistent existence of the gapless Dirac-cone surface states in the first and second SLs, which is well consistent with the model analysis in Fig. 2(b).
We now investigate the AFM TI MnBi 2 Te 4 .We take (7 + 1)-SL and (7 + 3)-SL MnBi 2 Te 4 films for our calcula-tions.Here, 1 SL in (7+1)-SL film and 3 SLs in (7+3)-SL film are effectively considered as the surface layer, while the remaining 7 SLs are considered as the below bulk.We make these choices based on the following considerations.For the thickness of MnBi 2 Te 4 thin film smaller (greater) than 3 SLs, the coupling ∆ 12 between the two Dirac-cone surface states (D 1 and D 2 ) is expected to be greater (smaller) than the Zeeman coupling strength h, thus leading to a Chern number of 0 (1) for the 1-SL (3-SL) MnBi 2 Te 4 film.In Figs.3(b) and 3(d), we plot the energy bands at Γ with increasing the vdW gap expansion δd for (7 + 1)-SL and (7 + 3)-SL MnBi 2 Te 4 , respectively, where red and green bands denote the bands from the surface layer.As shown in Fig. 3(b), the energy gap closes at a small expansion (δd = 0.18 Å) and then reopens (red lines) in Fig. 3(e) with increasing δd.It is worth mentioning that this critical value of δd = 0.18 Å is stable and not sensitive with the different functionals of first-principles calculations (see SM [63]).In Fig. 3(f), we explicitly plot the surface local density of states (LDOS) at the critical value of δd = 0.18 Å, confirming the emergence of gapless Dirac-cone surface state.In contrast, for the case of the (7 + 3)-SL MnBi 2 Te 4 film, the band gap decreases with increasing δd, as shown by green lines in Figs.3(d) and 3(e), but it tends to saturate for large δd and never closes.We also present corresponding results for (7 + 2)-SL MnBi 2 Te 4 in Figs.3(c) and 3(e) (orange lines), where the energy gap of the surface states remains almost un-

IV. DISCUSSION AND CONCLUSION
For the surface states of MnBi 2 Te 4 , there was a problem of experimental and theoretical incompatibility.Unexpected gapless surface states were observed by many ARPES experiments [54][55][56], contrary to gapped surface states predicted by theories [11,42].Very encouragingly, the mechanism of the vdW gap expansion, which we proposed, can solve this problem.Taking a small expansion of the topmost vdW gap (e.g., δd = 0.18 Å) for MnBi 2 Te 4 , the gapless Dirac-cone surface states of MnBi 2 Te 4 have been obtained by using the present three-Dirac-fermion approach and first-principles calculations.
A previously proposed explanation for the gapless Dirac-cone surface states is the reconstruction of the surface magnetization, but it will break the three-fold rotation symmetry, resulting in the loss of three-fold rotation symmetry of the surface states [56,67].In contrast, under the present mechanism of the vdW gap expansion, the gapless surface states retain the original three-fold rotation symmetry.In Fig. 4, we plot the calculated result for the surface LDOS and Fermi surfaces (FSs) at two selected energy levels for MnBi 2 Te 4 with δd = 0.18 Å (first and second columns).One can see that the surface states are gapless and their FSs preserve the three-fold rotation symmetry but break the six-fold rotation symmetry due to the broken TRS.We also calculate the surface states and FSs of CaBi 2 Te 4 to exclude the possibility of gapless surface states induced from nonmagnetic TIs, as shown in Fig. 4 (third and fourth columns).Though the surface states of CaBi 2 Te 4 are gapless, they preserve the six-fold rotation symmetry, which is essentially different from the three-fold rotation symmetry in AFM TI MnBi 2 Te 4 .It is worth mentioning that the gapped surface states were observed in the recent point contact tunneling spectroscopy on MnBi 2 Te 4 [57], indicating that a moderate pressure on the surface can reduce an expansion of the vdW gap to obtain gapped surface states.Therefore, it is expected that a moderate pressure will provide a promising way to realize the QAHE with a large band gap in MnBi 2 Te 4 films.
Furthermore, we have applied the three-Dirac-fermion model to other vdW magnetic TIs in the MnBi 2 Te 4 family, such as MnBi 4 Te 7 , MnBi 6 Te 10 , and MnSb 4 Te 7 (see SM [63]).It is found that the gapless surface states universally exist in all these materials under the expansion of the topmost vdW gap, which is not only confirmed by our first-principles calculations but also consistent with most experimental observations [28,[68][69][70][71].
Supplemental Materials for "Three-Dirac-fermion approach to unexpected universal gapless surface states in van der Waals magnetic topological insulators" Dinghui Wang 1 , Huaiqiang Wang Here, we show the existence of the combined symmetry of inversion P and time-reversal T in a coupled two-Diracfermion system, where the two Dirac fermions have opposite T-breaking mass terms (induced by opposite magnetic moments) and opposite helicities.For the three-Dirac-fermion model with h 1 = h 2 = −h 3 = h in the main text, when ∆ 12 = 0, D 1 is isolated from D 2 and D 3 , and the coupled D 2 and D 3 subsystem can be exactly described by the above model.In the ordered bases of (|D 2↑ ⟩,|D 2↓ ⟩,|D 3↑ ⟩,|D 3↓ ⟩), the four-band model Hamiltonian is given by where k ± = k x ± ik y , v is the Fermi velocity, h represents the Zeeman coupling strength, and the Pauli matrices τ i and σ i act in the orbital and spin subspaces.The inversion and time-reversal operators are given by τ x and iσ y K (K mean complex conjugate), respectively.The combined P T operator is then simply obtained as iτ x σ y K.It is straightforward to show that the above Hamiltonian satisfies the P T symmetry as

P T H(k)(P T )
Since the antiunitary P T operator satisfies (P T ) 2 = −1, Kramers degeneracy is ensured at every momentum, leading to doubly degenerate band structures with vanishing total Berry curvatures and thus zero Chern number for the occupied bands.

II. FERROMAGNETIC CASE BY THREE-DIRAC-FERMION MODEL
Here, we present the detailed analysis of the ferromagnetic (FM) counterpart of three-Dirac-fermion model, where the gapless transition point appears only when ∆ 12 < h is satisfied.For the coupled three-Dirac-fermion model with FM moments, i.e., h 1 = h 2 = h 3 = h (h > 0 is assumed), the Hamiltonian is given by The energy spectrum can be easily obtained as Obviously, the energy gap closes at the Γ point when ∆ 2 12 + ∆ 2 23 = h is satisfied.In the detaching process of the surface system with the van der Waals gap distance between surface and bulk layers changing from d 0 to ∞, as shown in Fig. 2(a) in the main text, ∆ 12 remains unchanged, while ∆ 23 /∆ 12 gradually decreases from a large value to zero.It follows that, when ∆ 12 < h, there exists a gap closing-and-reopening process at a critical value of ∆ 23 /∆ 12 , as shown in Fig. S1(a).This is accompanied by a change of the total Chern number from 1/2 to 3/2.In contrast, when ∆ 12 > h, the gap always keeps open, as shown in Fig. S1(b), and the Chern number keeps unchanged as 1/2.Note that the condition for the mergence of gap closing in the FM case is opposite to that in the antiferromagnetic (AFM) case in the main text.
To confirm the above analysis based on the three-Dirac-fermion model, we have carried out first-principles calculations of (5+1)-SL and (5+3)-SL FM MnBi 2 Te 4 with out-of-plane magnetic moments, where the topmost 1 SL and 3SL are gradually detached from the the 5SL bulk.The evolutions of the surface gap size in the expansion process of the vdW gap distance between the detached layer and the bulk are shown in Fig. S1(c), where a gapless point emerges for the (5+3)-SL case with ∆ 12 < h, while the gap remains open for the (5+1)-SL case with ∆ 12 > h.These results are consistent with the theoretical analysis of the FM case by the three-Dirac fermion model.

III. SURFACE LDOS AND FERMI CONTOURS BY THREE-DIRAC-FERMION MODEL
In this section, to further show the functionality of the three-Dirac model, we will use it to calculate the surface local density of states (LDOS) and Fermi contours in the vdW gap expansion process for both the nonmagnetic case and AFM case exhibiting a gapless transition point, corresponding to the (7+1)-SL nonmagnetic CaBi 2 Te 4 and AFM MnBi 2 Te 4 films in the main text.
To reveal the trigonal or hexagonal structures of the surface states, we need to include symmetry-allowed higherorder corrections up to third-order terms of k in the Hamiltonian of Eq. ( 2) in the main text.A nonmagnetic Bi 2 Te 3 -type surface state satisfies the following three symmetries, namely, the time-reversal symmetry T , three-fold rotation symmetry C 3z , and mirror symmetry M x .In the presence of the out-of-plane magnetic order, both T and M x are broken, whereas the C 3z symmetry and combined time-reversal and mirror symmetries M x T are preserved.Through the method of invariant in the k • p theory, all symmetry allowed terms up to third order in k can be obtained (see, e.g., Ref. [1]).For simplicity, we consider the following symmetry allowed third-order correction for each of the three Dirac fermions where k ± = k x ± ik y .It is worth mentioning that this third-order correction term for Dirac surface states of TIs has also been derived in Ref. [2].The full Hamiltonian H ′ of the coupled three-Dirac-fermion system is then obtained by combining the Hamiltonian in Eq. ( 2) of the main text with the above terms.
To calculate the LDOS and Fermi contours of the gradually detached surface composed of the two Dirac fermions, D 1 and D 2 , we resort to the Green's function method, and the surface LDOS is given by where the retarded Green's function G r (ω) is given by G r (ω In Fig. S2, we present the results of surface LDOS (first column) and three representative Fermi contours in ascending order of the energy cuts in the LDOS for the nonmagnetic layered topological insulators (TIs) (last row) and three stages in the interlayer vdW gap expansion process of layered AFM TIs, namely, without vdW expansion (first row with a representative ∆ 23 /∆ 12 = 2), at the critical gap expansion value exhibiting the gapless Dirac cone (second row with ∆ 23 /∆ 12 ≈ 0.95 for h/∆ 12 = 0.3), and, in the surface-detached limit with ∆ 23 /∆ 12 = 0 (third row).From the third column [Figs.S2(c, g, k)], it can be clearly seen that, with increasing vdW gap expansion, the Fermi contours gradually change from trigonal shape to hexagonal shape, which results from the gradually recovered inversion symmetry of the detached surface system.It should be emphasized that the results predicted by the effective three-Dirac-fermion model are in qualitative consistence with the first-principles results in Fig. 4 in the main text.
We remark that, in real materials, there also exist Rashba-like bands on the surface, e.g., due to the broken inversion symmetry on the surface.However, since we focus on the Dirac surface states originating from nontrivial bulk topology, which suffice to explain the trigonal or hexagonal warping effect in magnetic TIs, we did not include these Rashba surface bands in the above theoretical discussion by three-Dirac-fermion model.

IV. RESULTS FROM DIFFERENT VDW FUNCTIONALS
To further validate the emergence of a gapless point in the expansion process of the topmost interlayer vdW gap of a (7 + 1)-SL MnBi 2 Te 4 , we have chosen different vdW functionals to plot the evolution of the surface gap, as shown in Fig. S3.Obviously, the gapless point stably emerges in all cases.

V. RESULTS FOR OTHER VDW MAGNETIC TOPOLOGICAL INSULATORS
In this section, to further show the functionality and universality of the proposed three-Dirac-fermion model, we apply it to some other representative examples of vdW magnetic TIs in the MnBi 2 Te 4 family, namely, MnBi 4 Te 7 , MnBi 6 Te 10 , MnSb 2 Te 4 and MnSb 4 Te 7 .Since the basic physics is similar, we here focus on MnBi 4 Te 7 and present results from both first-principles calculations and three-Dirac-fermion model analysis, which are consistent with each other.Whereas, we only present the first-principles results for the rest three materials.

A. MnBi4Te7
As shown in Fig. S4(a), MnBi 4 Te 7 consists of one SL MnBi 2 Te 4 and one quintuple-layer (QL) Bi 2 Te 3 as the building block.Similar to MnBi 2 Te 4 , the magnetic ground state features out-of-plane FM couplings within each SL, and AFM couplings between adjacent SLs.MnBi 4 Te 7 is proposed to be a nontrivial Z 2 AFM TI, which is expected to exhibit gapped Dirac surface states for boundaries along the out-of-plane (z-) direction.However, experimental observations show that [3][4][5][6], depending on the surface terminations, there exist two types of topological surface states for MnBi 4 Te 7 , namely, gapless (gapped) surface states for SL (QL) termination.The gapped surface states for the QL termination case have been explained to result from the hybridization between surface and bulk bands.However, the gapless surface states for the SL termination still remain elusive.
Here, we will take advantage of the three-Dirac-fermion model to show that in the vdW expansion process of the topmost vdW gap distance, a gapless transition point in the surface band structure can emerge only for the SLtermination case, whereas the surface bands always remain gapped for the QL termination.It should be emphasized that our results are in consistence with the experimental observations, and can provide further insight into the properties of the surface state in MnBi 4 Te 7 .

SL termination case
We first consider the SL termination case, where the three-Dirac-fermion models consists of two Dirac surface states from the topmost SL and the other from the top surface state from the QL below, as shown in Fig. S4(a).The model Hamiltonian is then given by Here the Zeeman term h SL results from the magnetic moments within the SL, while the Zeeman term h QL comes from the FM proximity effect on the QL from neighboring SLs.Note that h SL can be treated as a constant, whereas h QL depends on the interlayer distance between the QL and neighboring SLs.Since a QL has two neighboring SLs, h QL in Eq. (S6) consists of two contributions, h I QL and h II QL , from the upper and lower SLs, respectively, which depend on the distance between the QL and neighboring SLs.For simplicity and without loss of generality, in the unexpanded case, we assume the proximity-induced Zeeman term for the top surface state of the QL to be h I QL = 0.5h SL (h II QL = −0.3hSL ) from the upper (lower) SL.In the expansion process of the topmost interlayer vdW gap between the QL and the SL above, h II QL obviously keeps unchanged, while h I QL gradually decreases until it vanishes in the decoupled limit, which can be assumed to be proportional to the distance-dependent coupling ∆ 23 between the QL and the SL.As a result, h QL is then given by With the above assumptions, the evolution of the SL-terminated surface band structure at the Γ point in the topmost vdW gap expansion process can be obtained, as shown in Fig. S4(b), where ∆ 12 is chosen as the energy unit and the other parameters are representatively set to be h SL /∆ 12 = 0.5 and ∆ 0 23 /∆ 12 = 2.A gap closing-and-reopening process is discernable with a gapless point separating two phases with C = 1/2 and C = −1/2, respectively.Similar to the discussion in the main text, the C = 1/2 phase can be understood from the ∆ 23 ≫ ∆ 12 limit, where the total Chern number is reduced to that of the gapped Dirac fermion D 1 on top of the SL with C = (1/2) sgn(h SL ) = 1/2.Whereas C = −1/2 comes from the other limit of ∆ 23 = 0, where the total Chern number is given as the sum of the isolated SL and the top surface state D3 of the QL.For the isolated SL with h SL < ∆ 12 , its Chern number equals zero, so the total Chern number is reduced to that of D3 with C = (1/2) sgn(h II QL ) = (1/2) sgn(−h SL ) = −1/2.Consequently, such a gapless point is topologically protected and thus inevitable for the SL termination case in the vdW gap expansion process.
As a further evidence, in Fig. S4(c), we plot the surface band gap evolution from first-principles calculations of a AFM MnBi 4 Te 7 with 4 (SL+QL) layers and SL termination, where the topmost 1 SL is gradually detached from the bulk.The emergence of the gap closing point in Fig. S4(c) is consistent with the above three-Dirac-fermion model analysis.

QL termination case
For the QL termination case, as shown in Fig. S4( Here, h t QL and h b QL represent the proximity-effect-induced Zeeman term of the top and bottom surface states of the topmost QL from the SL below.Similar to the discussion above, both h t QL and h b QL depend on the interlayer distance between the topmost QL and the below SL, and they are also proportional to the interlayer coupling ∆ 23 .For simplicity, h t QL and h b QL are reasonably assumed to be 0.3h SL (∆ 23 /∆ 0 23 ) and 0.5h SL (∆ 23 /∆ 0 23 ), respectively.The evolution of the band structure at Γ in the expansion process of the topmost interlayer vdW gap distance can then be obtained, shown in Fig. S4(e).Obviously, the surface band gap remains open and the Chern number keeps unchanged as 1/2.This result can also be understood by inspecting the two limits of ∆ 23 /∆ 12 → ∞ and ∆ 23 /∆ 12 → 0. In the ∆ 23 /∆ 12 → ∞ (∆ 23 /∆ 12 → 0) limit, the total Chern number is reduced to that of D1 (D3) on top of the QL (SL), which is simply given as   nations to plot the surface band gap as a function of the expansion δd of the topmost interlayer vdW gap.It can be seen that similar to MnBi 4 Te 7 , only the SL termination of MnBi 6 Te 10 exhibit a gapless point, while the gap remains for both QL1 and QL2 terminations.
Figure S5(c) presents the results of the band gap evolution for both the AFM and FM cases of a (5+1)-SL MnSb 2 Te 4 .Since MnSb 2 Te 4 is topologically trivial without any Dirac surface states, the band gap persists in the topmost vdW gap expansion process for both cases.
In fig.S5(d), we plot the evolutions of the surface band gap in the topmost vdW gap expansion process of a 4-(QL+SL) AFM MnSb 4 Te 7 films with SL and QL terminations, respectively.Because MnSb 2 Te 4 is topologically nontrivial, analogous to MnBi 4 Te 7 , the gapless point appears only in the SL termination case.

VI. WAVEFUNCTION PROFILE OF TOPOLOGICAL SURFACE STATES
To further support the validity of the three-Dirac-fermion model in the description of layered TIs with vdW gap modulation, we plot the wavefunction profile of the topological surface state in the vdW gap expansion process.The wavefunction of the Dirac state at the Γ point is simply assumed as ψ i (z) ∝ e −λi|z−zi| (1/λ i characterizes the material-dependent decay length and λ i = 1 is chosen for simplicity in numerical calculations), with i = 1, 2, and 3 corresponding to D 1 , D 2 , and D 3 Dirac states, respectively, and z i denotes the position of the peak of each Dirac state.When there exist finite overlaps between wavefunctions of these Dirac states, e.g., when the surface layer is sufficiently thin (e.g., a few SLs for MnBi 2 Te 4 ), finite couplings will be induced between them.This underlies the coupling terms of ∆ 12 between D 1 and D 2 , and ∆ 23 between D 2 and D 3 in the construction of the coupled three-Dirac-fermion model, shown in Fig. S6(a).From this coupling picture, it is obvious that the final topological surface state should be formed by the hybridization of all the three Dirac states.
Here, we take the case of ∆ 12 > h as an example, corresponding to Fig. 2(e) in the main text, where the coupling between D 1 and D 2 is significantly larger than the magnetic Zeeman coupling strength.We plot the wavefunction profile of the final topological surface state (red lines) for three representative cases, namely, ∆ , at different expansion distances of the interlayer vdW gap between the surface layer (shaded in pink) and blow bulk (shaded in blue), corresponding to the pristine case with strongly-coupled surface and bulk systems, the critical case near the gapless point with moderate expansion, and the significantly expanded case with weakly-coupled surface and bulk systems.As expected, in the strongly-coupled (weakly-coupled) case, the wavefunction distribution of the topological surface state locates mainly within the surface layer (around the top of the below bulk).Whereas for the moderate expansion case, the topological surface state exhibit similar weights on the surface layer and the top of the below bulk.The overall feature of the above topological surface state is consistent with previous works on MnBi 2 Te 4 .

VII. FIRST-PRINCIPLES CALCULATION DETAILS
First-principles calculations were performed using the Vienna ab initio simulation package (VASP) [7].The Perdew-Burke-Ernzerhof (PBE) functional [8], within the projector augmented wave (PAW) [9] are used to describe the exchange-correlation potential and energy.Lattice constants are adopted from experimental data, a = 4.334 Å and c = 13.637Å for each septuple layer.Then we relax atoms positions.A Hubbard-like U = 5 eV is used to account for strongly localized Mn 3d orbitals.A kinetic cutoff energy of 410 eV and a 10×10×1 Γ-centered k points mesh is used in a self-consistent field.The unit cells are stacked along c direction to generate multi-septuple layers films with a 20 Å vacuum space for the thick slab system.In all calculations, we keep G-type AFM with a magnetic axis along the c direction.By replacing Mn in MnBi 2 Te 4 with Ca, and then relaxing the atoms positions, we simulate the non-magnetic case in CaBi 2 Te 4 .To account for the surface layer vdW expansion effect, we construct the maximally localized Wannier function (MLWF) [10] from first-principles calculations with Mn d, Bi p, and Te p orbitals.The tight-binding Hamiltonian is divided into surface and bulk parts.The spectral functions and Fermi surface are calculated with the surface Green's functions of the semi-infinite system.Both spectral functions and the Fermi surface are projected to the upmost septuple layer.

12 FIG. 2 :
FIG.2:The three-Dirac-fermion model.(a,d) Schematic of the three-Dirac-fermion model of the nonmagnetic case (a), and the AFM case (d), where the Dirac states (D1 and D2) correspond to the top and bottom surface states of the surface layer, and D3 comes from the top surface states of the below bulk.∆12 is the intralayer coupling between D1 and D2, while ∆23 is the interlayer coupling between D2 and D3.(b,c,e,f) Bands at Γ as a function of ∆23/∆12, corresponding to increasing the interlayer vdW gap, for the nonmagnetic case (b) and the AFM case hosting an AFM surface layer (c) and an FM surface layer with ∆12 > h (e) and with ∆12 < h (f).The h represents the Zeeman coupling strength.

FIG. 4 :
FIG. 4: Surface LDOSs and Fermi surfaces (FSs) of AFM MnBi2Te4 and nonmagnetic CaBi2Te4.The surface LDOS of MnBi2Te4 (a,b) and FSs at energy levels E1 (e,f) and E2 (i,j) with the topmost vdW gap expansion δd = 0.18 Å.The surface states are gapless in (a,b).Surface FSs show the three-fold rotation symmetry due to the broken TRS, while bulk FSs (e.g.kz = 1.2π) interestingly show a six-fold-like rotation symmetry, because of the weak kz dependence of bulk bands due to the P T symmetry [66].The surface LDOS (c,d) and FSs at E1 (g,h) and E2 (k,l) of CaBi2Te4.The surface FSs show a six-fold rotation symmetry, while bulk FSs (e.g.kz = 1.2π) clearly show a three-fold rotation symmetry.Moreover, notably, there are a set of Rashba-type surface states at E2 due to breaking the inversion symmetry on the surface.

Fig. S 1 :
Fig.S1: Bands at the Γ point of the coupled three-Dirac model with FM moments and (a) ∆12 < h, (b) ∆12 > h in the expansion process of the van der Waals gap between surface and layers, where ∆23/∆12 gradually decreases to zero in the surface-detached limit.The gap closing-and-reopening process appears only when ∆12 < h, which is accompanied by a unit change of the Chern number (C).In the numerical calculation, h/∆12 is chosen to be 1.2 and 0.8 in (a) and (b), respectively.(c) Surface gap evolution from first-principles calculation of a (5+1)-SL (red lines) and (5+3)-SL (blue lines) FM MnBi2Te4 with out-of-plane FM moments.

Fig. S 2 :
Fig. S 2: Surface LDOS (first column) and corresponding Fermi contours in ascending order of the three energy cuts.The upper three rows correspond to the three stages in the interlayer vdW gap expansion process of a layered AFM topological insulator, which presents the gapless transition point in the expansion process.(First row) the unexpanded case with a representative value of ∆23/∆12 = 2. (Second row) at the critical gap expansion value of ∆23/∆12 ≈ 0.95.(Third row) the surface-detached limit with ∆23/∆12 = 0.The last row describes the nonmagnetic case with a representative ∆23/∆12 = 2.In the numerical calculations, the Zeeman field strength is set as h/∆12 = 0.3, and the coefficient λ of the third-order terms in Eq. (S4) is chosen as λ1 = −λ2 = λ3 with λ/∆12 = 0.2.

Fig. S 3 :
Fig. S 3: The evolution of the surface band gap in the expansion process of the topmost interlayer vdW gap of a (7+1)-SL MnBi2Te4 with different vdW functionals.All curves exhibit a gapless point around 0.18 Å in the expansion process.

2 Fig. S 4 :
Fig. S 4: Schematic of the three-Dirac model of the SL termination case (a), and QL termination case (d) of AFM MnBi4Te7.Bands at Γ as a function of ∆23/∆12, corresponding to increasing the interlayer vdW gap from the three-Dirac-fermion model for the SL termination case (b) and the QL termination case (e).First-principles results of the surface gap evolution in the topmost vdW expansion process with increasing δd for a 4-(SL+QL) MnBi4Te7 with the SL termination case (c) and the QL termination case (f).Gapless point only appears for the SL termination case.

Fig. S 5 :
Fig. S 5: (a) Schematic of MnBi6Te10 with three different kinds of surface terminations.Surface band gap evolutions in the expansion process of the topmost interlayer vdW gap from first-principles calculations for 3-(SL+QL+QL) AFM MnBi6Te10 with the three surface terminations (b), (5+1)-SL MnSb2Te4 with both FM and AFM configurations (c), and 4-(SL+QL) AFM MnSb4Te7 with SL and QL terminations, respectively (d).
follows that the band gap should remain open with no gapless transition point.This is also further confirmed by the first-principles calculation of the surface gap evolution of a 4 (SL+QL) MnBi 4 Te 7 with QL termination, as shown in Fig. S4(f).

2 Fig. S 6 :
Fig. S 6: For the case of ∆12 > h in the three-Dirac fermion model illustrated in (a), wavefunction profile at Γ point of the surface states for three representative cases, ∆23/∆12 = 2 (b), ∆23/∆12 = 0.92 (c), and ∆23/∆12 = 0.2 (d), in the vdW gap expansion process with gradually increased d and decreased ∆23/∆12, corresponding to the the pristine case with strongly coupled surface and bulk systems, the critical case near the gapless point with moderate expansion, and the significantly expanded case with weakly coupled surface and bulk systems.h = 0.4 is chosen in the numerical calculations.