1 Introduction

In condensed matter physics and materials science, the band theory serves as the cornerstone for understanding and designing solid-state materials with the structure-property relationship. This relationship highlights that the physical properties of materials are determined by atomic connectivity [1]. One prototypical example is the two-dimensional (2D) carbon allotrope graphene [2], which is characterized by \(sp^{2}\) hybridization of carbon that leaves \(p_{z}\) orbital in a hexagonal lattice, so that the conduction and valence bands are linearly touched at the Brillouin zone boundary, giving rise to massless Dirac fermions with unique electronic properties, such as semi-metallicity, ultrahigh carrier mobility and nontrivial topological properties [3–6]. During the past years, tremendous efforts have been made to the exploration of other 2D Archimedean lattices for new phenomena and diverse functionalities, including but not limited to triangular, kagome, Lieb, dice and square-octagon lattices [7–13].

Amongst them, triangular lattice is one most common and generic network in the 2D limit. With each site coordinating with six neighbors, the triangular lattice has attracted considerable interest due to its ideal realization of spin frustration that may lead to quantum spin liquid states [14, 15]. It is also an attractive model to achieve unconventional superconductivity, as observed in the atomic Sn layer deposited on Si(111)-\(R30^{\circ}( \sqrt {3} \times \sqrt {3})\) surface [16, 17]. Conventional semiconductors, such as Si and Ge, have reconstructed surfaces with triangular arrangement in (111) orientation. From the simplest tight-binding (TB) model, a single orbital within triangular lattice gives a single energy band [18], conforming to the angle-resolved photoemission spectroscopy (ARPES) and density functional theory (DFT) results of the surfaces [19]. Recent progress of orbital physics, utilizing the orbital degree of freedom in addition to the charge and spin of electrons, opens up new opportunities to realize richer extraordinary properties, including metal-insulator transition, flat band and nontrivial topology [7, 11, 20–27]. An interesting example is the \((p_{x}, p_{y})\)-orbital hexagonal lattice model. With the coexistence of flat and Dirac bands, this model possesses spin-orbit coupling (SOC) arising from the on-site spin-orbit interaction, so that the SOC strength can be much enhanced [22, 23]. Indeed, our earlier proposal of epitaxially growing heavy atoms (such as Bi) on a semiconductor substrate demonstrated quantum spin Hall (QSH) states with a large gap of 0.8 eV, where the \((p_{x}, p_{y})\)-orbital hexagonal model is realized via an intriguing “substrate-orbital-filtering” effect [23]. Later experiments confirmed the proposal by scanning tunneling microscopy/spectroscopy (STM/STS) and ARPES measurements [28]. Development of topological phases based on semiconductor substrate is important in a way that it not only facilitates experimental characterization and manipulation of the topological edge states, but also allows potential applications due to the intrinsic compatibility with mature semiconductor technology. In this regard, it is tempting to investigate the triangular lattice with multiple orbitals and design new topological materials supported on semiconductor substrate.

In this work, we theoretically explore the electronic properties of triangular lattice with multiple \((p_{x}, p_{y}, p_{z})\) orbitals. For heavy main-group elements, such as In, Pb and Bi, \(p_{x}\), \(p_{y}\) and \(p_{z}\) orbitals are normally treated as the valence states since the s orbital is located relatively deep in energy. By systematically comparing the lattice with a single p orbital, \((p_{x}, p_{y})\) and \((p_{x}, p_{y}, p_{z})\) orbitals within TB modeling, we show the rich electronic structures, including type I/II Dirac point, quadratic nodal point, nodal-loops. QSH states can be achieved by regulating the horizontal mirror symmetry, SOC magnitude and relative position of orbitals. Combined with DFT based first-principles calculations, we demonstrate the realization of QSH phases by epitaxial growth of In atoms on a series of semiconductor substrates, including cubic C/Si/Ge(111), 4H-SiC(0001) and InAs(111) surfaces, to form a multi-\((p_{x}, p_{y}, p_{z})\) orbital triangular lattice. The topological gap can reach ∼0.2 eV, feasible for room-temperature characterization. Importantly, the DFT calculated results capture well the salient features of the mapped topological phase diagram in the TB parameter space. As the proposed systems are experimentally available [29], we expect the present study to stimulate immediate attention to regulate the properties of semiconductor-supported topological materials via carrier doping, external strain, electrical and magnetic fields.

2 Methods

2.1 First-principles calculations

DFT-based first-principles calculations were performed by using the Vienna ab initio simulation package (VASP) [30, 31]. The projected augmented wave (PAW) method was employed for the ion-electron interactions [32]. The exchange-correlation energy was modeled by the generalized gradient approximations (GGA) in the form of Perdew-Burke-Ernzerhof (PBE) formulism [33]. An energy cutoff of 500 eV was adopted for the plane-wave basis. The convergence threshold of energy and force was set to \(1\times 10^{-6}\text{ eV}\) and 0.01 eV/Å, respectively. SOC was treated as a perturbation term [34]. The Wannier90 software was used to obtain the maximally-localised Wannier functions (MLWFs) and topological properties [35–37]. To simulate the substrate supported systems, a surface slab of eight atomic layers were included, with the bottom four layers fixed at bulk positions and all other atoms fully relaxed. The Brillouin zone integration was sampled by a \(17\times 17\times 1\) Γ-centered k-mesh. To avoid the interaction between neighboring slabs, a vacuum region of ∼20 Å was adopted perpendicular to the surface plane. STM image was simulated based on the Tersoff-Hamann theory [38]. The formation energy of In monolayer on substrate is defined by, \(E_{\mathrm{{f}}}=(E_{\mathrm{{tot}}}-n\times E_{\mathrm{{In}}}-E_{\mathrm{{sub}}})/n\), where \(E_{\mathrm{{tot}}}\), \(E_{\mathrm{{In}}}\) and \(E_{\mathrm{{sub}}}\) are the energies of the supported system, single In atom and substrate, respectively. n is the number of In atoms.

2.2 Tight-binding modeling

To model the triangular lattice with multiple orbitals, the SK hopping parameters are constructed to describe the NN p-p interactions [39],

$$ \begin{gathered} E_{i,j}= \langle i \left| H \right| j \rangle =-n_{i}n_{j}(V_{pp\pi}-V_{pp\sigma}), \\ E_{i,i}= \langle i \left| H \right| i \rangle =n^{2}_{i}V_{pp \sigma}+\bigl(1-n^{2}_{i} \bigr)V_{pp\pi}, \end{gathered} $$
(1)

where \(V_{pp\pi}\) and \(V_{pp\sigma}\) are SK π/σ hybridization parameters, and \(n_{i,j}\) represents the direction cosine parameters with \(i \neq j\in \{x,y,z\}\). The SK π/σ hybridization parameters can also be written in terms of hopping parameters, \(t_{1}\), \(t_{2}\) and \(t_{z}\). SOC is treated by, \(\langle i \vert \lambda _{\mathrm{{SOC}}}\mathbf{S}\cdot \mathbf{L} \vert j \rangle \), with S (L) denoting the spin (orbital) angular momentum.

3 Results and discussions

3.1 Single and multi-orbital models of triangular lattice

Triangular lattice has only one site in a unit cell and each site coordinates with six neighbors in the 2D plane (see Fig. 1a), which differs significantly from the honeycomb graphene structure. With a \(\mathrm{D}_{\mathrm{{6h}}}\) point group symmetry, the lattice involves two-fold rotation symmetry \(C_{\mathrm{{2z}}}\), three-fold rotation symmetry \(C_{\mathrm{{3z}}}\), inversion symmetry P and in-plane two-fold rotation symmetry \(C_{\mathrm{{2xy}}}\). As the simplest case, we first consider a single orbital on each site. The symmetry restricted TB Hamiltonian can be written as, \(\mathcal{H}_{1} = 2t_{1}[\cos (k_{1})+\cos (k_{2})+\cos (k_{1}+k_{2})]\), with \(t_{1}\) denoting the nearest-neighbor (NN) hopping. This Hamiltonian produces a single band within the Brillouin zone, and partial-filling will give rise to a metallic band structure (Fig. 1b). With the presence of SOC (the SOC strength is denoted by \(\lambda _{\mathrm{{SOC}}}\)), this single band remains intact, while the inclusion of additional terms in the Hamiltonian, such as correlation and exchange energies, may give rise to unconventional superconductivity and magnetic frustration. This has been well-documented in literature [15–17].

Figure 1
figure 1

Single and multi-orbitals in a 2D triangular lattice. (a) Schematic of a 2D triangular lattice, the Brillouin zone with high-symmetry points, and the \(p_{x}\), \(p_{y}\) and \(p_{z}\) orbitals. Unit cell is indicated by red arrows. (b) Band structure of a single p-orbital triangular lattice with \(\mathrm{D}_{\mathrm{{6h}}}\) symmetry. (c) Band structure of a \((p_{x}, p_{y})\)-orbital triangular lattice with \(\mathrm{D}_{\mathrm{{6h}}}\) symmetry. The Dirac point at K and quadratic point at Γ can be gapped by SOC. The resulting \(\mathrm{Z}_{2}\) invariants are 0 for each band. (d) Band structure of a \((p_{x}, p_{y}, p_{z})\)-orbital triangular lattice with \(\mathrm{D}_{\mathrm{{6h}}}\) symmetry. \(p_{z}\) band introduces a type-II Dirac point (green circle) and two nodal-loops circling K and Γ. With SOC, crossing points are gapped and \(\mathrm{Z}_{2}\) invariants are 0. (e) Same as (d) but with \(\mathrm{C}_{\mathrm{{6v}}}\) symmetry (without SOC). The dashed line denotes the energy of the Dirac point. (f) Same as (b) but with \(\mathrm{C}_{\mathrm{{6v}}}\) symmetry. \(\lambda _{\mathrm{{R}}}\) varies from 0 to 0.8. (g) Same as (c) but with \(\mathrm{C}_{\mathrm{{6v}}}\) symmetry. \(\lambda _{\mathrm{{SOC}}}\) varies from 0 to 2.8. The resulting \(\mathrm{Z}_{2}\) invariants are 0. (h) Band structure of a \((p_{x}, p_{y}, p_{z})\)-orbital triangular lattice with \(\mathrm{C}_{ \mathrm{{6v}}}\) symmetry. \(\lambda _{\mathrm{{SOC}}}\) varies from 0 to 2.8, and the calculated \(\mathrm{Z}_{2}\) invariants are 1, −1, 0 for the bottom, middle and top band, respectively. For (b) and (f), \(t_{1}=-0.3\). For (c) and (g), \(t_{1}=1.55\), \(t_{2}=0.78\). For (d), (e) and (h) \(t_{z}=-0.6\) while \(t_{1}\) and \(t_{2}\) are the same as that in (c) and (g). \(\lambda _{\mathrm{{SOC}}}=0.8\) in (c) and (d)

Full size image

Next, we consider the \((p_{x}, p_{y})\)-orbital triangular lattice. With \(\mathrm{D}_{\mathrm{{6h}}}\) symmetry, the Hamiltonian can be expressed as,

$$ \begin{gathered} \mathcal{H}_{xy}= \mathcal{E}_{xy}\otimes \tau _{0}+ \begin{bmatrix} E_{xx} & E_{xy} \\ E_{xy}^{*} & E_{yy} \end{bmatrix}, \end{gathered} $$
(2)

in which \(\tau _{0}\) is the identity Pauli matrix operating on the orbital degree of freedom. \(\mathcal{E}_{xy}\) is the onsite energy of \((p_{x}, p_{y})\)-manifold. The off-diagonal term, describing the coupling between \(p_{x}\) and \(p_{y}\) orbitals, can be written as, \(E_{xy}=-2t_{2}\cos (k_{1})+2t_{2}\cos (k_{1}+k_{2})\), where \(t_{2}\) is the \((p_{x}, p_{y})\) NN hopping parameter. The diagonal integrals, denoting the single band Hamiltonians, are modified based on \(\mathcal{H}_{1}\) [see more details in Additional file 1 Note 1]. Diagonalization of Eq. (2) gives two energy bands that cross each other, forming a Dirac point at K and a quadratic nodal point at Γ (Fig. 1c). SOC splits the two points, and the two gapped bands are topologically trivial, with the topological \(\mathrm{Z}_{2}\) invariant calculated to be 0 (Fig. S1).

Further inclusion of \(p_{z}\) orbital in the model leads to the \((p_{x}, p_{y}, p_{z})\)-orbital triangular lattice, for which the block diagonal Hamiltonian is constructed as,

$$ \begin{gathered} \mathcal{H}_{xyz}= \mathcal{E}_{xy}\otimes \tau _{0} \oplus \mathcal{E}_{zz} + \begin{bmatrix} \mathcal{H}_{xy} & 0 \\ 0 & E_{zz} \end{bmatrix}, \end{gathered} $$
(3)

where \(\mathcal{E}_{zz}\) is the onsite energy and \(E_{zz}\) is the single-band Hamiltonian of the \(p_{z}\) orbital. The \(E_{zz}\) integral can be written as, \(E_{zz} = 2t_{z}\cos(k_{1})+2t_{z}\cos(k_{2})+2t_{z}\cos(k_{1}+k_{2})\) wherein \(t_{z}\) describes the NN hopping of \(p_{z}\) orbital. As shown in Fig. 1d, compared to the \((p_{x}, p_{y})\)-orbital triangular lattice, here \(p_{z}\) crosses \((p_{x}, p_{y})\) bands, leading to the formation of a type-II Dirac point with large tilted conical dispersion. Interestingly, the emergent Dirac points result in two nodal loops circling K and Γ points (also see Fig. S2). With SOC, these crossed points are split, and the calculated \(\mathrm{Z}_{2}\) is 0 for all bands, so that the system remains topologically trivial.

The above results suggest that in weak SOC limit, multi-orbital triangular lattice can exhibit many interesting band structures, including type I/II Dirac point, quadratic nodal point and nodal loops. These properties may lead to a variety of intriguing physical phenomena, such as large magnetoresistance, Klein tunneling, drumhead states, and even high transition temperature superconductors [40–42]. However, the SOC induced band gap opening does not contribute to a topological phase, and the multi-orbital triangular lattice with \(\mathrm{D}_{\mathrm{{6h}}}\) symmetry remains topologically trivial.

3.2 Triangular lattice with broken horizontal mirror symmetry

With Z-direction confined to (sub)nanometer thick, 2D materials always require a certain substrate during growth, characterization or device setup. Therefore, the horizontal mirror symmetry (\(M_{\mathrm{{h}}}\)) will be broken since the two surfaces of the structure become asymmetric when supported on substrate. The original point group symmetry is \(\mathrm{D}_{\mathrm{{6h}}}\), where the horizontal mirror symmetry can be defined as, \(M_{\mathrm{{h}}} = C_{\mathrm{{2z}}}P\). When \(M_{\mathrm{{h}}}\) is removed, the point group will be reduced to \(\mathrm{C}_{\mathrm{{6v}}}\). As a result, the TB Hamiltonians need to be modified, especially when SOC is taken into account. For the single-orbital triangular lattice, the Hamiltonian should be revised in the form of \(\mathcal{H}_{1} \otimes \sigma _{0} +\mathcal{H}'(\lambda _{\mathrm{{R}}}) \otimes \sigma _{y}+\mathcal{H}''(\lambda _{\mathrm{{R}}})\otimes \sigma _{x}\), with σ acting on the spin space. The Hamiltonians \(\mathcal{H}'(\lambda _{\mathrm{{R}}})\) and \(\mathcal{H}''(\lambda _{\mathrm{{R}}})\) denote the additional interactions determined by effective SOC \(\lambda _{\mathrm{{R}}}\). Here, the off-diagonal integrals are extrinsic rather than intrinsic SOC. The calculated band structures are shown in Fig. 1f. Clearly, the spinful bands become split with SOC, while at high-symmetry points, they remain doubly degenerate. This type of splitting originates from the well-known Rashba effect, which has been widely discussed in inversion symmetry-breaking systems, such as surface structures, heterostructures and gated 2D materials [43–45]. It should also be noted that Rashba splitting tends to have detrimental effect on the topological phase. However, if the Rashba effect is not so strong compared to intrinsic SOC, the topological phase can be maintained, as revealed in the hexagonal Bi layer grown on semiconductor surface [23, 28]. Therefore, we will mainly focus on the intrinsic SOC effect in the following discussions.

For the \((p_{x}, p_{y})\)-orbital triangular lattice, breaking of \(M_{\mathrm{{h}}}\) symmetry respects the original Hamiltonian \(\mathcal{H}_{xy}\) (Eq. (2)), so that the band structure with \(\mathrm{C}_{\mathrm{{6v}}}\) symmetry retains the same as that with \(\mathrm{D}_{\mathrm{{6h}}}\), as indicated in Figs. 1c and 1g. Note that although SOC splits the Dirac point at K and the quadratic nodal point at Γ, and the size of opened gaps increases with larger SOC strength, the bands are still topologically trivial with \(\mathrm{Z}_{2}=0\). This could be understood by the fact that separately either the Dirac point or the quadratic nodal point can give rise to nontrivial topology induced by SOC; however, counting the two points together, the total effect leads to a trivial phase. Interestingly, further reducing \(\mathrm{C}_{\mathrm{{6v}}}\) to \(\mathrm{C}_{\mathrm{{3v}}}\) symmetry destroys the Dirac point at K point, and a topological phase with \(\mathrm{Z}_{2}=1\) can be achieved by introducing SOC that splits the quadratic nodal point at Γ [20, 46].

Now, we turn to discuss the case of \((p_{x}, p_{y}, p_{z})\)-orbital triangular lattice. With \(\mathrm{D}_{\mathrm{{6h}}}\) symmetry, the Hamiltonian (Eq. (3)) is block diagonal, so that the \((p_{x}, p_{y})\)-manifold does not hybridize with \(p_{z}\) orbital. However, with \(\mathrm{C}_{\mathrm{{6v}}}\) symmetry, hybridization between them will occur, and the resulting band structure differs significantly from that with \(\mathrm{D}_{\mathrm{{6h}}}\) symmetry. As shown in Fig. 1e, the nodal-loops disappear and a clear Dirac point is left at K point. SOC opens an energy gap between the two Dirac bands, and the gap size increases with stronger SOC. Importantly, the calculated \(\mathrm{Z}_{2}\) invariants of the two bands are ±1, indicative of a topologically nontrivial phase. Thus, \(M_{\mathrm{{h}}}\) symmetry breaking triggers a topological phase transition in the \((p_{x}, p_{y}, p_{z})\)-orbital triangular lattice.

3.3 Topological phase diagram and edge states in \((p_{x}, p_{y}, p_{z})\)-orbital model

As \(M_{\mathrm{{h}}}\) symmetry has significant influence on the band properties of \((p_{x}, p_{y}, p_{z})\)-orbital triangular lattice, we then delve into the fundamental mechanisms by analysing the TB Hamiltonian. To describe the hybridization between \(p_{z}\) and \((p_{x}, p_{y})\) orbitals, we introduce two terms, \(E_{xz}\) and \(E_{yz}\), which depict the interactions between \(p_{x}\) and \(p_{z}\), \(p_{y}\) and \(p_{z}\) orbitals respectively, in the Hamiltonian \(\mathcal{H}_{xyz}\) (Eq. (3)). As such, the revised Hamiltonian with \(\mathrm{C}_{\mathrm{{6v}}}\) symmetry reads,

$$ \begin{gathered} \mathcal{H}^{\mathrm{{C_{6v}}}}_{xyz}= \mathcal{E}_{xy} \otimes \tau _{0}\oplus \mathcal{E}_{zz} + \begin{bmatrix} E_{xx} & E_{xy} & E_{xz} \\ E_{xy}^{*} & E_{yy}& E_{yz} \\ E_{xz}^{*}& E_{yz}^{*}& E_{zz} \end{bmatrix}, \end{gathered} $$
(4)

in which the introduced integrals can be derived from the modified Slater-Koster (SK) hopping parameters [39], based on SK π/σ hybridization integrals \(V_{pp\pi}\) and \(V_{pp\sigma}\). Thus, \(E_{xz}\) and \(E_{yz}\) can be written as, \(E_{xz}= i\frac{\zeta _{\mathrm{{xyz}}}}{2}(V_{pp\sigma}-V_{pp\pi})[2\sin (k_{1})- \sin (k_{2})+\sin (k_{1}+k_{2})]\) and \(E_{yz}= i\frac{\sqrt {3}\zeta _{\mathrm{{xyz}}}}{2} (V_{pp\sigma}-V_{pp\pi})[ \sin (k_{1}+k_{2})+\sin (k_{2})]\), respectively. Here, the parameter \(\zeta _{\mathrm{{xyz}}}\) denotes the hybridization strength. In practice, it is related to the binding between 2D material and the substrate, which can be infinitesimal or very large, corresponding to weak van der Waals interaction and strong chemical bonding, respectively. Moreover, when supported on a substrate, the energy of \(p_{z}\) orbital may also change its relative position with respect to that of \((p_{x}, p_{y})\) orbitals, i.e. \(\mathcal{E}_{zz} \neq \mathcal{E}_{xy}\). Without losing generality, we fix \(V_{pp\sigma}\) and \(V_{pp\pi}\) integrals and focus on the evolution of band structures against the hybridization strength and SOC.

Figure 2a presents the evolution of band structures with increasing \(\zeta _{\mathrm{{xyz}}}\). Clearly, hybridization between \(p_{z}\) and \((p_{x}, p_{y})\) orbitals splits the nodal points and destroys the nodal-loops with appearance of energy gaps, so that a clean Dirac point can be obtained at K point. Note that the gap opening derives from symmetry breaking rather than SOC. With the inclusion of SOC, both the Dirac point at K and the quadratic point at Γ are opened that leads to a QSH phase (Fig. 1h). Interestingly, with the increase of \(\zeta _{\mathrm{{xyz}}}\), there is a phase transition from one topological phase (QSH I) with \(\mathrm{Z}_{2}\) invariants of 1, −1, 0 for the three gapped bands (from bottom to top), to another topological phase (QSH II) with \(\mathrm{Z}_{2}\) invariants of 1, 0, −1 for the three bands, respectively (Fig. 2b). We can map out the topological phase diagram by adjusting \(\zeta _{\mathrm{{xyz}}}\) and \(\lambda _{\mathrm{{SOC}}}\), which is shown in Fig. 2c. With finite \(\zeta _{\mathrm{{xyz}}}\) and a small value of \(\lambda _{\mathrm{{SOC}}}\), the system is topologically nontrivial. Increase of SOC first closes the nodal loop and then reopens a gap, inducing a transition from nontrivial to trivial phase (see more details in Fig. S3). Increase of \(\zeta _{\mathrm{{xyz}}}\) initiates a trivial to nontrivial phase transition (QSH I). Further increase of \(\zeta _{\mathrm{{xyz}}}\) changes QSH I to QSH II (Fig. S4). To confirm the topology, we construct a semi-infinite nanoribbon with the 2D \((p_{x}, p_{y}, p_{z})\)-orbital triangular lattice and calculate the topological edge states by iterative Green’s function approach [47]. The edge states for the QSH I and QSH II phases are shown in Fig. 2c and Fig. S5, respectively. The edge-resolved bands could be clearly seen, which intersect at Γ̄ and connect the conduction and valence bands.

Figure 2
figure 2

Symmetry breaking and topological phase diagram of \((p_{x}, p_{y}, p_{z})\)-orbital triangular lattice. (a) Band structure evolution with different \(\zeta _{\mathrm{{xyz}}}\). Nodal loops are broken by non-zero \(\zeta _{\mathrm{{xyz}}}\) (squares). (b) Band structures with different SOC (\(\zeta _{\mathrm{{xyz}}}=0.9\)). The \(\mathrm{Z}_{2}\) invariants are also indicated. (c) Topological phase diagram with different \(\zeta _{\mathrm{{xyz}}}\) and \(\lambda _{\mathrm{{SOC}}}\), including the trivial, QSH I and QSH II phases. (d) Topological edge states corresponding to (b) (\(\lambda _{\mathrm{{SOC}}}=0.8\)). \(t_{1}=1.55\), \(t_{2}=0.78\), \(t_{z}=-0.4\). All bands are shifted with reference to the energy of the Dirac point (\(\mathrm{E}_{\mathrm{{DP}}}\))

Full size image

Then we consider changing the relative position of \(p_{z}\) orbital with reference to that of \((p_{x}, p_{y})\) and explore the band evolution. When \(p_{z}\) is lower than the Dirac point formed by the \((p_{x}, p_{y})\) bands at K (Fig. 3a), increase of \(\zeta _{\mathrm{{xyz}}}\) splits the nodal-loop around Γ and opens an energy gap. Along M - K, a tilted Dirac point is formed that cannot be gapped by \(\zeta _{\mathrm{{xyz}}}\). With the presence of SOC, a topological phase arises with \(\mathrm{Z}_{2}\) invariants of 1, −1, and 0 for the three bands (Fig. 3b). The nontrivial band topology is also confirmed by the edge state calculations (Fig. 3c). When \(p_{z}\) is shifted below \((p_{x}, p_{y})\) bands, the system becomes topologically trivial (see Fig. S6), since no overlap occurs between \(p_{z}\) and \((p_{x}, p_{y})\) states. On the other hand, if we shift \(p_{z}\) band to higher enegies, there will be a phase transition from QSH I to QSH II due to the band inversion at Brillouin zone center (Fig. S7). Further up-shift of \(p_{z}\) above \((p_{x}, p_{y})\) bands will result in a trivial phase, since there is no overlap between \(p_{z}\) and \((p_{x}, p_{y})\) states again. The detailed topological phase diagram is plotted in Fig. 3d.

Figure 3
figure 3

Topological phase diagram with modified energy position of \(p_{z}\). (a) Band evolution with different \(\zeta _{\mathrm{{xyz}}}\), so that \(\mathrm{D}_{\mathrm{{6h}}}\) symmetry is reduced to \(\mathrm{C}_{ \mathrm{{6v}}}\). Nodal loops are broken by non-zero \(\zeta _{\mathrm{{xyz}}}\) (squares). (b) Band structure with SOC, with \(\mathrm{Z}_{2}\) invariants indicated. (c) Topological edge states. (d) Topological phase diagram with different \(\mathcal{E}_{zz}\) and \(\lambda _{\mathrm{{SOC}}}\), including the trivial, QSH I and QSH II phases. \(t_{1}=1.55\), \(t_{2}=0.78\), \(t_{z}=-0.4\), \(\zeta _{\mathrm{{xyz}}}=0.5\) and \(\lambda _{\mathrm{{SOC}}}=0.4\). For (a)–(c), \(\mathcal{E}_{xy}=0\), and \(\mathcal{E}_{zz}=-5\)

Full size image

3.4 Realization of QSH states on semiconductor substrates

The above TB analyses suggest the possibility of obtaining nontrivial topology in \((p_{x}, p_{y}, p_{z})\)-orbital triangular lattice by regulating the horizontal mirror symmetry, onsite energy of related orbitals and SOC. Here, we demonstrate real material systems that can fully capture this multi-orbital model and exhibit large-gap QSH states. Especially, recent experiments demonstrated the successful growth of In monolayer on SiC surface, which has a triangular lattice and shows nontrivial topological properties, as observed by STM/STS and ARPES [29]. We thus resort to the epitaxial growth of In monolayer on a series of semiconductor substrates, including diamond, SiC, Si, Ge, and InAs.

We first calculate the band structure of a free-standing In monolayer by DFT, and the results are shown in Fig. 4a. Clearly, the \(p_{z}\) intersects \((p_{x}, p_{y})\) bands, forming nodal points. SOC splits these points, and the calculated \(\mathrm{Z}_{2}\) is 0, suggesting a trivial phase (Fig. S8 and Fig. S9). This is in perfect agreement with the TB results of \((p_{x}, p_{y}, p_{z})\)-orbital triangular lattice with \(\mathrm{D}_{ \mathrm{{6h}}}\) symmetry. Epitaxial growth of In monolayer on a substrate will break the \(M_{\mathrm{{h}}}\) symmetry and leave the structure with \(\mathrm{C}_{\mathrm{{6v}}}\) symmetry (Fig. 4b). We consider the adsorption of In monolayer on the surfaces of C(111), Si-face of 4H-SiC(0001), Si(111), Ge(111) and As-face of InAs(111). As an example, the simulated STM image of In monolayer on 4H-SiC(0001) is shown in Fig. 4c, which shows a perfect triangular arrangement that agrees well with experimental observations [29]. As the lattice constant of semiconductor surface increases from 2.56 Å for C(111), 3.07 Å for SiC(0001), 3.84 Å for Si(111), 4.00 Å for Ge(111), to 4.28 Å for InAs(111), the hopping parameters in the TB Hamiltonian can thus be fully modeled. Also, chemical bonding with charge transfer between In and substrate can reflect the \(\mathrm{D}_{\mathrm{{6h}}}\) symmetry breaking. We compute the charge transfer by using Bader charge analysis approach [48, 49], and find that In atoms lose electrons and substrates gain electrons for these structures. Interestingly, the transferred electrons increase for the systems from In@SiC(0001) to In@InAs(111), indicating stronger \(M_{\mathrm{{h}}}\) symmetry breaking for In@ InAs(111). We also calculate the differential charge density in real space. As shown in Fig. 4e, the density plots have a one-to-one correspondence with the Bader charge analysis.

Figure 4
figure 4

Realization of \((p_{x}, p_{y}, p_{z})\)-orbital triangular lattice in semiconductor supported systems. (a) DFT calculated band structure of a freestanding In monolayer in triangular lattice (with SOC). The lattice constant is fixed to 3.07 Å, i.e., that of SiC(0001). (b) Illustration of epitaxially growing In monolayer on a semiconductor substrate. (c) Optimized atomic structure and simulated STM image of In@SiC(0001) in the constant height mode (1.2 Å higher than In). (d) Lattice constant and charge transfer. (e) Real-space differential charge density plots for In grown on different substrates. The differential charge density is defined as, \(\Delta \rho =\rho _{\mathrm{{tot}}}-(\rho _{\mathrm{{In}}}+\rho _{\mathrm{{sub}}})\), with \(\rho _{\mathrm{{tot}}}\), \(\rho _{\mathrm{{In}}}\) and \(\rho _{\mathrm{{sub}}}\) charge densities of the total system, separated In monolayer and substrate, respectively. Yellow (cyan) color denotes electron accumulation (depletion). \(\text{Isovalue}=0.002~e/\text{Bohr}^{3}\)

Full size image

Next, we investigate the band structures of In monolayer on semiconductor substrates, and the results are presented in Fig. 5. Obviously, the energy bands of \((p_{x}, p_{y}, p_{z})\)-orbital triangular lattice can still be identified around the bulk gap of semiconductor substrates, though the In monolayer now has a reduced \(\mathrm{C}_{\mathrm{{6v}}}\) symmetry. Here, some important features can be identified. First, the p-projected energy bands become less dispersive from In@C(111) to In@InAs(111), which can be attributed to the increased lattice constant that leads to smaller In-In hopping. Also, by examining the relative positions of \(p_{z}\) band with respect to that of \((p_{x}, p_{y})\), we find that they have a clear relationship with the charge transfer between In and substrate. For instance, from In@SiC(0001) to In@InAs(111), the \(p_{z}\) band gradually shifts down due to the increased electrons transferred to the substrate (Figs. 5b-e). Remarkably, for In@C(111) and In@SiC(0001), the SOC split Dirac point can be obviously observed within the bulk gap of the substrate, leading to a nontrivial phase with the topological invariant of occupied states [\(\mathrm{Z}_{2}(\mathrm{occ})\)] calculated to be 1 (Fig. 5a and 5b); For In@Si(111) and In@Ge(111), the \(p_{z}\) band is located below the Dirac point, and SOC also opens a nontrivial gap with \(\mathrm{Z}_{2}(\mathrm{occ})=1\) (Fig. 5c and 5d); For In@InAs(111), however, the \(p_{z}\) band becomes lower than \((p_{x}, p_{y})\), and the calculated \(\mathrm{Z}_{2}(\mathrm{occ})\) is 0, suggesting a trivial phase (Fig. 5e). The calculated topological gaps are 232 and 37.3 meV for In@C(111) and In@SiC(0001) at K, while the gaps for In@Si(111) and In@Ge(111), originated from the tilted Dirac points along M - K, are calculated to be 156 and 164 meV, respectively. Therefore, electronic properties of supported In monolayers can be effectively controlled by the underlying substrates, which fully capture the characteristics of \((p_{x}, p_{y}, p_{z})\)-orbital triangular lattice, and agree well with the TB calculations.

Figure 5
figure 5

Electronic structures and nontrivial band topology. (a)–(e) Band structures of In monolayer grown on C(111), SiC(0001), Si(111), Ge(111) and InAs(111) substrates, respectively. Relative contributions of In \(p_{x}\), \(p_{y}\) and \(p_{z}\) orbitals are shown. \(\mathrm{Z}_{2}\) invariants of the occupied bands, \(\mathrm{Z}_{2}(\mathrm{occ})\), are also indicated. Fermi level is set to zero. The shadow regions represent the energy bands of substrate. (f) Schematic diagram of \(\mathrm{Z}_{2}(\mathrm{occ})\) against the relative position of \(p_{z}\) orbital with reference to that of \((p_{x}, p_{y})\)

Full size image

It is interesting to notice that free-standing In monolayer is topologically trivial, while the substrate supported system can be nontrivial. This originates from the crucial role of substrate in breaking the horizontal mirror symmetry and shifting the relative position of \(p_{z}\) orbital, so that the topological phase can be realized, as predicted in the Hamiltonian \(\mathcal{H}^{\mathrm{{C_{6v}}}}_{xyz}\) (Eq. (4)). This differs from bismuthene grown on SiC(0001), where the \(p_{z}\) orbital is fully filtered via “substrate-orbital-filtering” effect that leaves \((p_{x}, p_{y})\) orbitals in a hexagonal lattice. In both cases, semiconductor substrate converts the otherwise topologically trivial lattice to a nontrivial phase, particularly attractive to design novel quantum materials and manipulate the topological phase transition.

4 Outlooks and conclusions

In experiments, In monolayer can be epitaxially grown on semiconductor substrates via molecular beam epitaxy (MBE) approach, while the morphology and electronic structures can be characterized by STM/STS. In particular, bismuthene and indium monolayers have been successfully prepared on SiC(0001) surface [28, 29] that exhibit QSH states. We have calculated the formation energies of In monolayer on C(111), SiC(0001), Si(111), Ge(111) and InAs(111), and found that the formation energies range from −1.6 to −3.5 eV (Table S1), indicating that the systems are energetically stable. The dynamical stability is also considered by phonon dispersion spectrum (Fig. S10). After sample preparation, properties such as nodal points, nodal loops and topological edge states, can be investigated by ARPES and magnetotransport measurements [29, 50–53]. Importantly, we highlight the mature semiconductor technology that may enable large-scale growth, atomic characterization and efficient manipulation of novel quantum states by chemical doping, external strain, electrical and magnetic fields.

In summary, we have systematically investigated the triangular lattice with multi-\((p_{x}, p_{y}, p_{z})\) orbitals. With \(\mathrm{D}_{\mathrm{{6h}}}\) symmetry, the lattice exhibits type-I/II Dirac point, quadratic nodal point and nodal-loops in the band structure. Breaking the horizontal mirror symmetry reduces \(\mathrm{D}_{\mathrm{{6h}}}\) to \(\mathrm{C}_{\mathrm{{6v}}}\), and nontrivial band topology emerges with the presence of SOC. We map out the topological phase diagram by tuning the symmetry, SOC and relative position of orbitals, and practical materials systems are proposed to achieve QSH phase based on semiconductor substrates. These findings not only shed light on the design of topological materials with orbital physics, but also pave an avenue to integrate novel quantum states with available technology for potential applications in dissipationless transport, optoelectronic and spintronic devices.