Orbital-active Dirac materials from the symmetry principle

Abstract

Dirac materials, starting with graphene, have drawn tremendous research interest in the past decade. Instead of focusing on the $p_{z}$ orbital as in graphene, we move a step further and study orbital-active Dirac materials, where the orbital degrees of freedom transform as a two-dimensional irreducible representation of the lattice point group. Examples of orbital-active Dirac materials occur in a broad class of systems, including transition-metal-oxide heterostructures, transition-metal dichalcogenide monolayers, germanene, stanene, and optical lattices. Different systems are unified based on symmetry principles. The band structure of orbital-active Dirac materials features Dirac cones at $K(K')$ and quadratic band touching points at Γ, regardless of the origin of the orbital degrees of freedom. In the strong anisotropy limit, i.e., when the π-bonding can be neglected, flat bands appear due to the destructive interference. These features make orbital-active Dirac materials an even wider playground for searching for exotic states of matter, such as the Dirac semi-metal, ferromagnetism, Wigner crystallization, quantum spin Hall state, and quantum anomalous Hall state.

1 Introduction

Graphene opened up a new era of topological materials, followed by the discovery of topological insulators, topological superconductors, and semi-metals in both two and three dimensions (see reviews [1–3]). Since then, the interplay between topology and correlation has been the primary focus of condensed matter research. Graphene and its variants, due to its excellent electronic and mechanical properties [4, 5], have become wonderful platforms for hosting exotic phases of matter and also find themselves widely applicable in electric device engineering and material science. The characteristic feature of graphene is the appearance of Dirac cones in the spectrum, tied to the symmetry of the underlying honeycomb lattice. Two sublattices (A and B) of the honeycomb lattice transform into each other under the simplest non-abelian point group $C_{3v}$, which contains 3-fold rotations and in-plane reflections. At the $K (K')$ point of the Brillouin zone, the wavefunctions of A and B sublattices form the two-dimensional (E) irreducible representations (irrep) of the $C_{3v}$, enforcing the Dirac cones. Once there, the Dirac cones are stable as long as time-reversal and inversion symmetries are preserved.

The on-site $p_{z}$ orbital of graphene transforms trivially (it belongs to the $A_{1}$ irrep) under the site symmetry group $C_{3v}$. It is natural to ask what happens if the on-site orbitals form the E-irrep of the point group. The E-irrep features the double degeneracy and anisotropy, which is expected to bring rich orbital physics in graphene-like Dirac materials. Such a situation arises in many distinct systems. It was initially studied in optical lattices, where the two-dimension irrep is realized by the $p_{x}$ and $p_{y}$ orbitals in the harmonic trap [6, 7]. In transition-metal-oxide heterostructures [8–11] and transition-metal-dichalcogenide monolayers [12], the d-orbitals decompose based on the $C_{3v}$-symmetry and are active near the Fermi surface. In the hexagonal monolayers of heavy elements, such as Germanene, Stanene, and Bismuthene, the $(p_{x}, p_{y})$ doublets realize the orbital degrees of freedom. Due to the enriched orbital structure of the Dirac cone, the gap opening, which turns out to be topologically non-trivial, equals to the atomic spin-orbit coupling, hence, it can be very large reaching the order of 1eV [13–18].

Even in simple carbon systems, orbital physics can be realized via lattice engineering, for example, organic framework [19, 20] and graphene-kagome lattice [21]. Remarkably, recently experiments [22–24] on twisted bilayer graphene revealed Mott insulator and superconductivity phases, and it is proposed that the low-lying degrees of freedom are compatible with two orbitals on the honeycomb lattice as well [25–30]. Furthermore, the orbital degrees of freedom do not have to be electronic and can manifest themselves as the polarization modes of polaritons in photonic lattices [31, 32] and phonons in graphene and mechanical structures [33–36].

Given all these interconnected systems and the increasing realizations of orbital-active Dirac materials, this work aims to bridge all the different systems through the symmetry principle. Despite the vastly different origins, the orbital degrees of freedom can be understood as the irreducible representations of the site symmetry of the lattice, which leads to universal properties. We show that the symmetry alone enforces the Dirac cone at $K (K')$ point and the quadratic band touching at the Γ point. Various gap opening mechanisms and interaction effects are discussed, which lead to the quantum spin Hall effect and quantum anomalous Hall effect. In particular, when the $E_{g}$ doublets realize orbital degrees of freedom, the resulting topological insulator states carry octupole order. Finally, the method employed here for studying the doubly-degenerate orbitals in the honeycomb lattice can be readily generalized to orbital degrees of freedom arising from larger lattice point groups.

The rest of the paper is organized as follows. In Sect. 2, the symmetry of the honeycomb lattice and the orbital realization of the on-site irreducible representations are studied by focusing on the d-orbitals. In Sect. 4, the band structure of the orbital-active honeycomb lattice systems is derived from a simple tight-binding model. In Sect. 5, we go beyond the simple tight-binding model and demonstrate that many interesting features of the band structure are solely protected by the lattice symmetry. In Sect. 6, various band gap opening mechanisms are stuided. In Sect. 7, the interplay between band structure and the interaction effects is discussed. Section 8 is left for summary and outlook.

2 The honeycomb lattice and orbital symmetries

We start with reviewing the symmetry of the planar honeycomb lattice. The planar honeycomb lattice, sketched in Fig. 1(a), consists of two sublattices A (blue) and B (red). The three nearest neighbor vectors are labeled as $\hat{e}_{1}\sim \hat{e}_{3}$. The symmetry of the lattice is described by the space group $P6mm$, a direct product of the point group $C_{6v}$ and the translation symmetry of the triangular Bravis lattice.¹ The maximal point group $C_{6v}$ is realized at the centers of the hexagons. On the other hand, the point group symmetry acting on a lattice site, called site symmetry, is a subgroup of the maximal point group. The site symmetry group is important because it affects the orbital part of the wavefunction of the degrees of freedom living on lattice sites (such as electrons, phonons, etc.) The site symmetry of the honeycomb lattice is $C_{3v}$ generated by a 3-fold rotation axis and three vertical reflection planes (e.g., the yz-plane and its symmetry counterparts by rotations of ±120^∘). In contrast, the reflection with respect to the xz-plane and its symmetry counterparts by rotations of ±120^∘ interchange the A and B sublattices and are not included in the site symmetry.

Figure 1

(a) The standard honeycomb lattice with $C_{6v}$ point symmetry group. (b) The buckled honeycomb lattice with the symmetry group is downgraded to $D_{3d}$

The orbital of the onsite degrees of freedom is classified by the irreducible representations of the site symmetry group. The $C_{3v}$ group has three irreducible representations (irrep), including two 1d irreps $A_{1,2}$ and a 2d irrep E as explained in Appendix A. The irreps fully determine the symmetry structure of the onsite degrees of freedom, regardless of their microscopic origins. In this article, we focus on electron atomic orbitals. Taking z axis perpendicular to the lattice plane, the s and $p_{z}$ orbitals realize the $A_{1}$-irrep and lead to the remarkable electronic structure of graphene. In contrast, the $p_{x}$ and $p_{y}$-orbitals realize the two dimensional E-irrep. This doublet can also be organized into the complex basis $p_{x} \pm ip_{y}$ which are eigenstates of the orbital angular momentum $L_{z}$ with eigenvalues ±1, respectively. As to the 5-fold d-orbitals, the $d_{r^{2}-3z^{2}}$ falls into the $A_{1}$ irrep. The remaining four form two E irreps: the $(d_{xz}, d_{yz})$ doublet and the $(d_{xy}, d_{x^{2}-y^{2}})$ doublet. The complex orbitals $d_{xz}\pm i d_{yz}$, and $d_{xy}\pm d_{x^{2}-y^{2}}$ carry orbital angular momentum numbers ±1 and ∓2, respectively. Since the site symmetry group only has one 2d irrep, the three doublets, $(p_{x}, p_{y})$, $(d_{xz}, d_{yz})$ and $(d_{xy}, d_{x^{2}-y^{2}})$ are equivalent as far as the symmetry is concerned. One can explicitly check that the group elements of the site symmetry $C_{3v}$ have the same matrix representation of the E-irrep.

A closely related lattice structure sketched in Fig. 1(b) is dubbed buckled honeycomb lattice, which can be viewed as a bilayer of sites taken from a cubic lattice in the $(1,1,1)$ direction. The blue and red dots form a honeycomb lattice when projecting into the $(1,1,1)$ plane. Compared to the planner honeycomb lattice, the point group symmetry of the buckled lattice downgrades from $d_{6h}$ to $d_{3d}$, where the six-fold rotation becomes a rotoreflection. On the other hand, the site symmetry remains the same, described by $C_{3v}$. As a result, based on previous analysis, the realizations of the E-irrep in the buckled lattice must be equivalent to the $(p_{x}, p_{y})$ doublet in the planar case. Here we focus on the d-orbitals and establish this equivalence. The buckled honeycomb lattice originates from the cubic lattice. Taking the z-axis along the $(0,0,1)$-direction, the 5-fold d-orbitals split into a $T_{2g}$ triplet $(d_{yz}, d_{zx}, d_{xy})$ and an $E_{g}$ doublet $(d_{x^{2}-y^{2}}, d_{r^{2}-3z^{2}})$, which are irreps of the $O_{h}$ point group. The site symmetry of the buckled lattice $C_{3v}$ is a subgroup of $O_{h}$. The $E_{g}$ doublet falls into the only 2d irrep E of $C_{3v}$, while the $T_{2g}$ triplet further splits into the 1d irrep $A_{1}$ and the 2d irrep E.

To make the connection between the $E_{g}$ doublet and the orbital realization of the E irrep in the planar case more explicit, we rotate the frame of the buckled lattice so that the z-axis is along the 3-fold axis $(1,1,1)$. Then the $E_{g}$ doublet becomes

$$ \begin{aligned} &d_{x^{2}-y^{2}}\rightarrow \frac{1}{\sqrt{3}}(d_{xy}+ \sqrt{2}d_{xz}), \\ &d_{r^{2}-3z^{2}}\rightarrow \frac{1}{\sqrt{3} }(d_{x^{2}-y^{2}}+ \sqrt{2} d_{yz}). \end{aligned} $$

(1)

Hence, the $E_{g}$ orbitals are a superposition of two E doublets $(d_{xy}, d_{x^{2}-y^{2}})$ and $(d_{xz}, d_{yz})$ in the planar case. Therefore, as far as the site symmetry $C_{3v}$ is concerned, the $e_{g}$ doublet is equivalent to the $(p_{x}, p_{y})$ doublet in the planar case. In fact, the mapping can be made explicit as

$$ \begin{aligned} d_{x^{2}-y^{2}} \leftrightarrow p_{x}, \qquad d_{r^{2}-3z^{2}} \leftrightarrow p_{y}. \end{aligned} $$

(2)

For completeness, the decomposition of the five d-orbitals into two E irreps and one $A_{1}$ irrep of the $C_{3v}$ group is presented as follows,

$$ \begin{aligned} \textstyle\begin{cases} (d_{x^{2}-y^{2}}, d_{r^{2}-3z^{2}}) & E, \\ (\frac{1}{\sqrt{2}}(d_{yz} -d_{zx}), \frac{1}{\sqrt{6}} (d_{yz}+d_{zx}-2d_{xy}) ) & E, \\ \frac{1}{\sqrt{3}}(d_{xy}+d_{yz}+d_{zx}) &A_{1}, \end{cases}\displaystyle \end{aligned} $$

(3)

choosing $(1,1,1)$ as the rotation axis. In addition to the $E_{g}$ orbitals which become an E-representation, the $T_{2g}$-orbitals split into one E irrep and one $A_{1}$ irrep. In principle, the two E-representations froming the $E_{g}$ and $T_{2g}$ orbitals can mix. In transition-metal-oxides where the buckled lattice is relevant, there is often an oxygen octahedron around each transition metal ion. The octahedron introduces a large crystal field that splits the $E_{g}$ and $T_{2g}$ orbitals. Hence, the mixing between the E irrep of $C_{3v}$ derived from the $E_{g}$ orbitals and that from the $T_{2g}$ orbitals is weak.

3 Magnetic octupole moment of the $E_{g}$ doublet

Although all realizations of E-irrep of $C_{3v}$ are equivalent from the symmetry consideration. The $E_{g}$ orbitals are special physically and worth special attention. The key difference lies in the angular momentum of the complex combination of the doublets. In the case of $(p_{x}, p_{y})$, the complex combination $p_{x} \pm i p_{y}$ takes the form $\exp (\pm i \theta )$, and thus carries angular momentum ±1 along the rotating axis. The same applies to the $(d_{xz}, d_{yz})$ doublet. In the case of $(d_{xy}, d_{x^{2}-y^{2}})$, the complex combination $d_{xy} \pm i d_{x^{2}-y^{2}}$ takes the form $\exp (\mp i 2\theta )$ and thus carries angular momentum ∓2. In contrast, the angular momentum of the complex combination of the $E_{g}$ orbitals $d_{x^{2}-y^{2}}\pm d_{r^{2}-3z^{2}}$ vanishes. From Eq. (1), the complex combination of $E_{g}$ doublet can be viewed as the weighted superposition of the complex combinations of the $(d_{xz}, d_{yz})$ and $(d_{xy}, d_{x^{2}-y^{2}})$. The angular momentum of the two doublets cancel each other, leading to the zero angular momentum of the $E_{g}$ doublet.

Instead of the angular momentum, the complex $E_{g}$ orbitals carry higher rank magnetic moment, measured by spherical tensor operators $Y_{lm}$. A list of spherical tensor operators constructed from the angular momentum operator L⃗ in the d orbital space can be found in Appendix B. Going through all the higher rank tensor operators, we find that the leading non-vanishing spherical tensor operators projecting into $E_{g}$ orbital is

$$ \begin{aligned} P_{E_{g}} Y_{3,\pm 2}P_{E_{g}}=\mp 3\sqrt{\frac{5}{2}} i\sigma _{2}, \end{aligned} $$

(4)

where $P_{E_{g}}$ is the projection operator. Two non-vanishing components of the rank-3 spherical tensor operators can be grouped into a single cubic harmonic tensor $\hat{f}_{xyz}=\frac{i}{\sqrt {2}}(Y_{2,-2}-Y_{2,2})$. It is projected into $E_{g}$ orbital space as

$$ \begin{aligned} P_{E_{g}}\hat{f}_{xyz} P_{E_{g}}=-3\sqrt {5} \sigma _{2}, \end{aligned} $$

(5)

where $\hat{f}_{xyz}$ corresponds to the octupole magnetic moment. Therefore, the complex combinations of the $E_{g}$ orbital, instead of carrying angular momentum, carry octupole magnetic moment, which was proposed to be the “hidden order” in certain strongly-correlated electronic systems [37–41].

4 The band structure of the orbital active honeycomb lattice

In this section, we study the band structure of the orbital active honeycomb lattice, including the planer and the buckled ones. To be concrete, we first introduce a simple nearest neighboring tight-binding model before presenting more general scenarios in the next section.

4.1 Constructing the tight-binding model

In the orbital active honeycomb lattice, the hopping between neighboring sites occurs between different components of the E-irrep and thus is more complicated than the orbital inactive case. There are two kinds of hopping processes allowed by symmetry on a bond. In terms of the chemistry convention, the amplitude of the σ-bonding is much larger than that of the π-bonding. The difference in the hopping amplitude arises from the anisotropy of the orbital wavefunction. The bonding direction of the σ (π)-bond is along the direction of the maximal (minimal) angular distribution of orbital wavefunctions. The σ-bond and the π-bond for all orbital realizations of the E-irrep are shown in Fig. 2 for one of nearest neighboring bond $\hat{e}_{3}$. In the planar case, the σ-bonding orbital is $p_{y}$, $d_{yz}$ or $d_{x^{2}-y^{2}}$, and in the buckled case, the σ bonding orbital is $d_{r^{2}-3z^{2}}$. The σ bonding orbitals along other nearest neighboring bonds are linear combinations of the two orbitals in the E irrep, obtained from applying 3-fold rotation on $p_{y}$, $d_{yz}$, $d_{x^{2}-y^{2}}$ or $d_{r^{2}-3z^{2}}$.

Figure 2

The orbital configuration of the σ-bonding on the $\hat{e}_{3}$ bond for (a) $(p_{x}, p_{y})$ doublet; (b) $(d_{xy}, d_{x^{2}-y^{2}})$ doublet; (c) $E_{g}$ doublets

Since all the different doublets form the same irrep of the $C_{3v}$ group, they transform in the same way under rotation. To unify the notation, we use $\gamma _{x,y}$ to represent the two states in the E irrep for different orbital realizations, where $\gamma _{x}$ stands for $p_{x}$, $d_{xy}$ or $d_{x^{2}-y^{2}}$, and $\gamma _{y}$ stands for $p_{y}$, $d_{x^{2}-y^{2}}$ or $d_{r^{2}-3z^{2}}$, correspondingly. $\gamma _{1}$, $\gamma _{2}$, $\gamma _{3}$ are defined to be the σ-bonding orbitals along the three nearest neighboring bonds $\hat{e}_{1}$, $\hat{e}_{2}$, $\hat{e}_{3}$, respectively,

$$ \begin{aligned} &\gamma _{1} = \frac{\sqrt{3}}{2} \gamma _{x} +\frac{1}{2} \gamma _{y}, \qquad \gamma _{2} =- \frac{\sqrt{3}}{2} \gamma _{x} +\frac{1}{2} \gamma _{y}, \\ & \gamma _{3} = - \gamma _{y}. \end{aligned} $$

(6)

Since the σ bonding is much stronger than the π bonding, we neglect the π bonding and construct the single particle Hamiltonian of the nearest neighboring σ bonding. Using $\gamma _{1}\sim \gamma _{3}$, the Hamiltonian can be conveniently written as

$$ \begin{aligned} H_{0}=t_{\parallel }\sum _{\vec{r}\in A,j=1,2,3} \bigl\{ \gamma _{j}^{ \dagger }(\vec{r}+\hat{e}_{j}) \gamma _{j}(\vec{r}) +h.c. \bigr\} , \end{aligned} $$

(7)

where the summation over r⃗ is only on the A sublattice and $\hat{e}_{1}\sim \hat{e}_{3}$ are the unit vectors pointing from A site to its three nearest neighboring B sites on the planar honeycomb lattice

$$ \begin{aligned} &\hat{e}_{1}=\frac{\sqrt{3}}{2} \hat{e}_{x} +\frac{1}{2} \hat{e}_{y}, \qquad \hat{e}_{2} =-\frac{\sqrt{3}}{2} \hat{e}_{x} + \frac{1}{2}\hat{e}_{y}, \\ &\hat{e}_{3}=-\hat{e}_{y}. \end{aligned} $$

(8)

The nearest neighboring distance is set to 1. In the case of the buckled honeycomb lattice, the three nearest neighboring vectors are the same as in the planar case when the coordinates are projected onto the $(1,1,1)$ plane.

The Hamiltonian has the same form for different realizations of the E-irrep of the site symmetry $C_{3v}$ for both the planar and buckled honeycomb lattice. This demonstrates the power and elegance of the symmetry principle.

4.2 The spectra and wavefunctions

The Hamiltonian Eq. (7) is ready to be diagonalized in momentum space, in which a 4-component spinor $\psi (\vec{k})$ is defined as

$$ \begin{aligned} \psi (\vec{k})= \bigl(\gamma _{x, A}(\vec{k}), \gamma _{y,A}(\vec{k}), \gamma _{x,B}( \vec{k}), \gamma _{y,B}(\vec{k})\bigr)^{T}, \end{aligned} $$

(9)

where A and B refer to the two sublattices. The annihilation operators $\gamma _{x,y}(k)$ is defined as

$$ \begin{aligned} \gamma _{x, y}(\vec{k})&= \frac{1}{\sqrt{N}} \sum_{\vec{r}} \gamma _{x,y}( \vec{r})e^{-i \vec{k}\cdot \vec{r}}. \end{aligned} $$

(10)

The crystal momentum k⃗ is defined in the Brillouin zone shown in Fig. 3(a). In the case of the buckled honeycomb lattice, r⃗ is the projected coordinate in the $(1,1,1)$ plane.

Figure 3

(a) The Brillouin zone of the honeycomb lattice. (b) The band structure of the tight binding model is described in Eq. (7) in the strong anisotropy limit. There are four bands (The spin degrees of freedom add another copy and are omitted). The bottom band and the top band are completely flat, while the middle two bands have the same dispersion relations as in graphene

With this setup, the Hamiltonian takes the following block form:

$$\begin{array}{r}H(k)=\left(\begin{array}{cc}0& H_{AB}\\ H_{AB}^{†}& 0\end{array}\right),\end{array}$$

(11)

where

$$\begin{array}{rl}& H_{AB}\\ & =t_{\parallel }\left(\begin{array}{cc}\frac{3}{4}(e^{i\overrightarrow{k}\cdot {\hat{e}}_1}+e^{i\overrightarrow{k}\cdot {\hat{e}}_2})& \frac{\sqrt{3}}{4}(e^{i\overrightarrow{k}\cdot {\hat{e}}_1}+e^{i\overrightarrow{k}\cdot {\hat{e}}_2})\\ \frac{\sqrt{3}}{4}(e^{i\overrightarrow{k}\cdot {\hat{e}}_1}+e^{i\overrightarrow{k}\cdot {\hat{e}}_2})& \frac{1}{4}(e^{i\overrightarrow{k}\cdot {\hat{e}}_1}+e^{i\overrightarrow{k}\cdot {\hat{e}}_2}+2e^{i\overrightarrow{k}\cdot {\hat{e}}_3})\end{array}\right).\end{array}$$

(12)

There are four band. The middle two bands exhibit exactly the same dispersion as that in graphene:

$$ \begin{aligned} E_{2,3}=\mp \frac{t_{\parallel}}{2} \biggl\vert \sum_{i}e^{i \vec{k} \cdot \hat{e}_{i}} \biggr\vert =\mp \frac{t_{\parallel}}{2} \sqrt{3+2 \sum_{i=1}^{3} \cos \vec{k} \cdot \vec{b}_{i}}, \end{aligned} $$

(13)

where $\vec{b}_{i} =\frac{1}{2} \epsilon _{ijk}(\hat{e}_{j}-\hat{e}_{k})$ are the next nearest neighboring vectors. The bands display two Dirac cones at K and $K'$. In addition, Fermi surface nesting and Van Hove singularity occur at $1/4$-filling above and below the Dirac point. The wavefunctions associated with the middle two bands are

(14)

where the phase $\theta =\text{arg}(\sum_{i}e^{i \vec{k}\cdot \hat{e}_{i} })$ and the normalization $N_{0}=6-2 \sum_{i=1}^{3}\cos \vec{k} \cdot \vec{b}_{i}$.

On the other hand, the top and the bottom bands are perfectly flat with the energy,

$$ \begin{aligned} E_{1,4}=\mp \frac{3}{2}t_{\parallel}. \end{aligned} $$

(15)

They connect to the middle two bands at the Γ point. The corresponding wavefunctions are represented as

(16)

and the energy dispersions are plotted in Fig. 3.

4.3 The appearance of the flat-band and the localized state

The existence of the flat bands implies that one can construct local eigenstates of the single-particle Hamiltonian. The flat band has been studied in detail in [6, 7] in the context of the p-orbitals in the honeycomb optical lattice, and plaquette states on a hexagon are constructed as the local basis (Fig. 4(a)). Here we investigate it in the orbital-active Dirac material realized by the $d_{xy}/d_{x^{2}-y^{2}}$ and the $e_{g}$ doublets.

Figure 4

(a) The spatial localized states in the lower flat band of the tight-binding Hamiltonian Eq. (7) for (a) $(p_{x}, p_{y})$ doublet; (b) $(d_{xz}, d_{yz})$ doublet; (c) $E_{g}$ doublet. (d) The spatial localized state Wigner-crystal when the lower flat-band is $1/3$ filled

The localized states can be elegantly constructed from the Bloch wavefunction in Eq. (16),

(17)

where R⃗ are the centers of the hexagons. Each hexagon hosts one localized state from each flat band. The localized states are

(18)

The summation is over the six vertices of the hexagon as shown in Fig. 4(b) for the case of the $d_{xy}/d_{x^{2}-y^{2}}$ doublet. The localized state has the same weight on each site but different orbital configurations, related by $\frac{\pi}{3}$ rotations. On each site, the orbital is projected into the π-bonding along the outward bond away from the hexagon. Due to the destructive interference, electrons in such localized single-particle states cannot leak out the plaquette, rendering the localized states eigenstates of the tight-binding Hamiltonian in Eq. (7) with the same energy. The above analysis can be carried over to the $E_{g}$ doublets on the buckled honeycomb lattice. In this case, the localized state is confined in a buckled hexagon as shown in Fig. 4(c).

4.4 Orbital configurations at high symmetric points

As shown in Fig. 3(b), the spectrum exhibits double degeneracy at the $K(K')$ point and the Γ point in the Brillouin zone. Now we investigate the Bloch wavefunction at these high symmetric points in detail.

4.4.1 Γ point

Around the center of the Brillouin zone $\vec{k}=(0,0)$, the Hamiltonian Eq. (11), in the unit of $t_{\parallel}$, can be expanded as

$$ \begin{aligned}[b] H_{\Gamma }(k) ={}&\frac{3}{2} \biggl(1- \frac{1}{4} \vert k \vert ^{2}\biggr) \tau _{1} \otimes \sigma _{0} \\ &{} - \frac{3}{4} ( k_{x} \tau _{2} \otimes \sigma _{1} +k_{y} \tau _{2} \otimes \sigma _{3} ) \\ &{} - \frac{3}{16} \bigl( \bigl(k_{x}^{2}-k_{y}^{2}\bigr)\tau _{1} \otimes \sigma _{3}+ 2 k_{x} k_{y} \tau _{1} \otimes \sigma _{1} \bigr), \end{aligned} $$

(19)

where the Pauli matrices $\sigma _{0} \sim \sigma _{3}$ ($\sigma _{0}$ represents the identity matrix) describe the orbital degrees of freedom $\gamma _{x,y}$ in the E irrep, and $\tau _{0} \sim \tau _{3}$ act on the space of sublattice $(A, B)$.

To the leading order, the dispersion of the above band structure is

$$ \begin{aligned} &E^{\Gamma}_{1,4}=\mp \frac{3}{2} t_{\parallel }, \\ &E^{\Gamma}_{2,3}=\pm \frac{3}{2}t_{\parallel } \biggl( -1+\frac{1}{4} \vert \Delta k \vert ^{2} \biggr). \end{aligned} $$

(20)

Therefore, the bands touch each other quadratically at both upper and lower degeneracy points. The degenerate wavefunctions at each touching point can be regrouped so that they only contain one of each the orbital component. At the lower touching point, the wavefunctions are

(21)

At the upper touching point, the B sublattice component acquires a minus sign, and the wavefunctions are

(22)

4.4.2 K and $K'$ points

Around $\vec{K}=(\frac{4\pi}{3\sqrt{3}},0)$, the Hamiltonian in Eq. (11) can be expanded as

$$ \begin{aligned} H_{K} ={}&-\frac{4}{3}\Delta k_{x} \tau _{1} \otimes \sigma _{0} + \frac{4}{3}\Delta k_{y} \tau _{2} \otimes \sigma _{0} \\ &{}-\frac{3}{8}(2+\Delta k_{x}) \tau _{1} \otimes \sigma _{0} - \frac{3}{8}\Delta k_{y} \tau _{2} \otimes \sigma _{3} \\ &{}-\frac{3}{8} \Delta k_{y} \tau _{1} \otimes \sigma _{1}- \frac{3}{8} (2- \Delta k_{x})\tau _{2} \otimes \sigma _{1} , \end{aligned} $$

(23)

where $\Delta \vec{k} =\vec{k}- \vec{K}$. The middle two bands touch each other with the dispersion,

$$ \begin{aligned} E_{2,3}= \mp \frac{3}{4}t_{\parallel } \vert \Delta k \vert , \end{aligned} $$

(24)

which demonstrates the Dirac cone. The doubly degenerate wavefunctions can be combined so that each of them only occupies one of the sublattices:

(25)

The orbital states in Eq. (25) on the two sublattices are circularly polarized and exhibit opposite chiralities. Such complex combinations of orbitals exhibit distinct physical properties for different orbital realizations. In the case of the $(p_{x},p_{y})$ doublet as well as the $(d_{xz}, d_{yz})$ doublet, the circularly polarized state carries angular momentum $L_{z}=\pm 1$; in the case of the $d_{xy}/d_{x^{2}-y^{2}}$ doublet, the circularly polarized state carries angular momentum ∓2, which are equivalent to ±1 due the 3-fold rotation symmetry; in the case of the $E_{g}$ doublet, it carries magnetic octupole moment. These complex orbital states play an important role in the topological properties of the orbital-active Dirac material and will be addressed in Sect. 6.

4.5 Response of the flat band to magnetic field

One interesting question regarding the flat band that appeared is its response to an external magnetic field. In a flat band, because the kinetic energy of the electrons is completely quenched, the usual semi-classical picture is no longer valid. Recent work [42] demonstrates that the response of flat bands to an external magnetic field is closely related to the quantum distance of the flat band. The quantum distance between two Bloch wavefunctions is defined as,

(26)

which ranges from 0 to 1. The flat band is singular if d is nonzero in the limit that $|k-k'|\rightarrow 0$. A singular point $k_{0}$ can be characterized by the maximal quantum distance $d_{\max}$ between the wavefunctions of k and $k'$ that are sufficiently close to $k_{0}$. In systems without orbital degrees of freedom, it is found that when $d_{\max}$ is nonzero, the flat band splits into Landau levels in an energy window, and the width of the energy window is determined by $d_{\max}$.

In our case, the flat band touches the dispersive Dirac band at the Γ point. The wavefunction of the flat band, expanded around the Γ point, is

(27)

where $\theta (\vec{k})$ is the azimuth angle of k⃗. Therefore, the wavefunction at θ and $\theta -\pi /2$ are orthogonal to each other, rendering the maximal quantum distance $d=1$. As a result, the flat band that appeared here is singular by definition.

We study the response of the flat band to an external magnetic field by including the magnetic field in the hopping parameter $t\rightarrow t\exp (i\int A(\vec{r})\,d\vec{r})$. For simplicity, the Landau gauge $A=B(0,x)$ is chosen and the strength of the magnetic field B is set by the flux through each hexagon $2\pi p/q$. In the presence of the magnetic field, the original four bands split into 4q sub-bands as shown in Fig. 5. While the middle two dispersive Dirac bands form the characteristic Landau levels, surprisingly, the two singular flat bands are completely inert to the magnetic field, in contrast with previous results on singular flat bands without orbital degrees of freedom.

Figure 5

(a) The flat band in the orbital active Dirac system is singular but does not respond to an external magnetic field due to the orbital nature of the singularity. (b) An example of the localized state in the presence of the magnetic field. Here the flux through each hexagon is $2\pi /3$. The localized state occupies three hexagons

The reason is due to the orbital nature of the singularity in Eq. (27). In the presence of the magnetic field, one can still construct localized states inside the flat band. Instead of occupying a hexagon, the localized states now occupy a magnetic unit cell. The wavefunction is nonzero only along the boundary of the magnetic unit cell, and the orbital configuration is parallel to the tangential direction. An example of the localized states is shown in Fig. 5(b) for the flux given by $\frac{2\pi}{3}$.

5 General symmetry consideration beyond strong anisotropic limit

In the last section, we demonstrate many remarkable features resulting from the tight-binding Hamiltonian Eq. (7), including orbital enriched Dirac cone, quadratic band touching and flat bands. Since the Hamiltonian only includes the nearest neighboring σ-bonding, a natural question is whether these features rely on the specific form of the Hamiltonian, or, are protected by symmetry.

In this section, we address this issue by general symmetry consideration. We study the band structure of the orbital active Dirac materials around the high symmetric points in the Brillouin zone using $k\cdot p$ theory. In general, the band flatness is not generic and can be bent by the π-bondings. However, the orbital configurations at the high symmetric points and k dependence of the dispersion around the Γ and K points are preserved as long as the symmetry of the system is respected. In the following, we consider the effects of the point group symmetry of the buckled honeycomb lattice $D_{3d}$.

5.1 Γ point

At the Γ point, the group of wavevector is the point group of the lattice, $D_{3d}$, which has an inversion symmetry. It has 6 irreps: $A_{1g}$, $A_{2g}$, $A_{1u}$, $A_{2u}$, $E_{g}$, and $E_{u}$. The Bloch wavefunction is composed of the orbital part and the plane wave part , which can be classified into the irreps of the group of wavevector separately. The orbital degrees of freedom form the two dimensional $E_{g}$ irrep, while the plane wave part and form the 1D $A_{1g}$ and $A_{2u}$ irreps. Therefore the composite Bloch wavefunction can be grouped into two two-dimensional irreps, $E_{g}$ and $E_{u}$, respectively.

This indicates that four energy levels can be grouped into two doubly-degenerate sets, where the degeneracy completely originates from orbitals. The Bloch wavefunctions of the $E_{g}$ irrep are

(28)

and the wavefunctions of the $E_{u}$ irrep are

(29)

This is consistent with Eq. (21) and Eq. (22). Furthermore, in both degeneracy sets, the orbital part of the wavefunctions can be regrouped into the complex orbital states . Therefore, spin-orbit coupling is able to gap out the degeneracy.

The generic dispersion around the Γ point can be obtained from the $k \cdot p$ theory. The $k \cdot p$ Hamiltonian is invariant under the $D_{3d}$ point group symmetry. In order to write down all the symmetry allowed terms, it is convenient to first classify the operators in the orbital space σ⃗, the operators in the sublattice space τ⃗ and the momentum k⃗ into irreps of the point group $D_{3d}$. In the sublattice space, the classification is

$$ \begin{aligned} \textstyle\begin{cases} \tau _{0}, \tau _{1}, & A_{1_{g}}, \\ \tau _{2}, \tau _{3}, & A_{2_{u}}. \end{cases}\displaystyle \end{aligned} $$

(30)

In the orbital space, the classification is

$$ \begin{aligned} \textstyle\begin{cases} \sigma _{0} , & A_{1g}, \\ \sigma _{2}, & A_{2g}, \\ (\sigma _{1}, \sigma _{3}), & E_{g}. \end{cases}\displaystyle \end{aligned} $$

(31)

For the momentum, the classification is

$$ \begin{aligned} \textstyle\begin{cases} (k_{x}, k_{y}), & E_{u}, \\ k_{x}^{2}+k_{y}^{2}, & A_{1g}, \\ (2k_{x} k_{y}, k_{x}^{2}-k_{y}^{2}), & E_{g}. \end{cases}\displaystyle \end{aligned} $$

(32)

Based on these classifications and the product table of the $D_{3d}$ point group, the most general Hamiltonian takes the following form,

$$ \begin{aligned}[b] H_{\Gamma }(\vec{k})={}&\bigl(h_{1}+h_{2} \bigl( k_{x}^{2}+ k_{y}^{2}\bigr)\bigr) \tau _{1} \otimes \sigma _{0} \\ &{} + k_{x} (h_{3} \tau _{2} \otimes \sigma _{1} + h_{4} \tau _{3} \otimes \sigma _{1} ) \\ &{}+ k_{y} (h_{3} \tau _{2} \otimes \sigma _{3} + h_{4} \tau _{3} \otimes \sigma _{3} ) \\ &{}+\bigl(k_{x}^{2}- k_{y}^{2}\bigr) (h_{5} \tau _{0} \otimes \sigma _{3} + h_{6} \tau _{1} \otimes \sigma _{3} ) \\ &{}+ 2 k_{x} k_{y} (h_{5} \tau _{0} \otimes \sigma _{1} +h_{6} \tau _{1} \otimes \sigma _{1} ), \end{aligned} $$

(33)

where $h_{1}\sim h_{6}$ are constants with the unit of energy. At the first order of k⃗, the degeneracy is still preserved, so we have to include second-order terms of k⃗. It recovers the tight-binding Hamiltonian around the Γ point presented in Eq. (19), when the h’s are set to

$$ \begin{aligned} &h_{1}=-4 h_{2}=-2h_{3} = -8 h_{6}=\frac{3}{2}t_{\parallel }, \\ &h_{4}=h_{5}=0. \end{aligned} $$

(34)

In the leading order, the dispersion is

$$ \begin{aligned} &E_{1,2}=-h_{1}- \biggl(2h_{2}+\frac{h_{3}^{2}+h_{4}^{2}}{h_{1}} \biggr) \vert k \vert ^{2}\pm (h_{5}-h_{6}) \vert k \vert ^{2}, \\ &E_{3,4}=h_{1}+ \biggl(2h_{2}+\frac{h_{3}^{2}+h_{4}^{2}}{h_{1}} \biggr) \vert k \vert ^{2}\pm (h_{5}+h_{6}) \vert k \vert ^{2}. \end{aligned} $$

(35)

At finite k, the degeneracy is lifted by

$$ \begin{aligned} & \bigl\vert E^{\Gamma}_{1} - E^{\Gamma}_{2} \bigr\vert =\frac{m_{-}}{2} \vert k \vert ^{2}, \\ & \bigl\vert E^{\Gamma}_{3} - E^{\Gamma}_{4} \bigr\vert =\frac{m_{+}}{2} \vert k \vert ^{2} , \end{aligned} $$

(36)

where the effect mass $m_{\pm}=4 |h_{5} \pm h_{6}|$. Therefore, the bands touch quadratically at both degenerate points. It is known that quadratic bound touching is unstable to interaction and can lead to exotic phases such quantum Hall effect and nematicity [43].

5.2 $K(K')$ point

At the K point, the group of wavevector is $D_{3}$, containing the three-fold rotations around the perpendicular axis and three 2-fold rotations around horizontal axis that interchanges the two sublattices.

The Bloch wavefunctions , containing both the plane wave part and the orbital part, can be organized into irreps of the group of wavevector. The plane wave part contains two sublattice components, forming the E-irrep, with the $A/B$ sublattice component carrying chirality ±1. The on-site orbital degrees of freedom $\gamma _{x}$ and $\gamma _{y}$ also transform as the E irrep, the complex combination $\gamma _{x}\pm i \gamma _{y}$ carrying the chirality ±1. Therefore, the four composite wavefunctions can be decomposed into three irreps as $2\otimes 2=1\oplus 1\oplus 2$. There are two trivial $A_{1}$ representations, where the chiralities of the orbital and planewave cancel each other and an E irrep where the chiralities of the orbital and planewave add up. In general, the two $A_{1}$ states do not have the same energy. In contrast, the two states in the E irrep are degenerate from symmetry and carry opposite chirality at the Dirac point. Explicitly, the two states are:

(37)

This is consistent with the wavefunctions of the tight-binding model at K in Eq. (25).

To obtain the generic dispersion around $K(K')$, we again employ the $k\cdot p$ theory. The Hamiltonian around K, a combination of the plane wave, orbital, and sublattice has to be invariant under the $C_{3v}$ point group. Following the same strategy, we first organize σ⃗, τ⃗ and the momentum $\Delta \vec{k} =\vec{k}-\vec{K}$ into irreps of the little group $d_{3}$. In the sublattice space, we have,

$$ \begin{aligned} \textstyle\begin{cases} \tau _{0}, &A_{1}, \\ \tau _{3}, &A_{2}, \\ (\tau _{1}, -\tau _{2}), &E. \end{cases}\displaystyle \end{aligned} $$

(38)

In orbital space, we have,

$$ \begin{aligned} \textstyle\begin{cases} \sigma _{0} , &A_{1}, \\ \sigma _{2}, &A_{2}, \\ (\sigma _{3}, -\sigma _{1}), &E. \end{cases}\displaystyle \end{aligned} $$

(39)

In addition, the momentum $(\Delta k_{x}, \Delta k_{y})$ belongs to the E irrep as well.

Therefore, based on the product table of $D_{3}$ point group, the most general Hamiltonian, apart from an overall constant, reads,

$$ \begin{aligned} H_{K}(\Delta \vec{k})={}&h_{1} \tau _{3} \otimes \sigma _{2} + h_{2} ( \tau _{1} \otimes \sigma _{3} + \tau _{2} \otimes \sigma _{1}) \\ &{}+\Delta k_{x} \bigl\{ + h_{3} \tau _{0} \otimes \sigma _{3}+h_{4} \tau _{1} \otimes \sigma _{0} \\ &{} + h_{5} \tau _{2} \otimes \sigma _{2} + h_{6} \tau _{3} \otimes \sigma _{1} \\ &{} +h_{7} ( \tau _{1} \otimes \sigma _{3} - \tau _{2} \otimes \sigma _{1}) \bigr\} \\ &+\Delta k_{y} \bigl\{ -h_{3}\tau _{0} \otimes \sigma _{1} - h_{4} \tau _{2} \otimes \sigma _{0} \\ &{} + h_{5} \tau _{1} \otimes \sigma _{2} + h_{6} \tau _{3} \otimes \sigma _{3} \\ &{} + h_{7} ( \tau _{2} \otimes \sigma _{3} + \tau _{1} \otimes \sigma _{1}) \bigr\} . \end{aligned} $$

(40)

The expansion of the σ-bonding Hamiltonian at K in Eq. (23) is a special case with

$$ \begin{aligned} &h_{2}=h_{4}=2 h_{7}=-\frac{3}{4} t_{\parallel }, \\ &h_{1}=h_{3}=h_{5}=h_{6}=0. \end{aligned} $$

(41)

In the general situation, at the leading order of Δk⃗, the dispersions of the four bands read,

$$ \begin{aligned} &E^{K}_{1,4}=-h_{1} \pm 2 h_{2}+ \mathcal{O}\bigl( \vert \Delta k \vert ^{2} \bigr), \\ &E^{K}_{2,3}=h1 \pm 2 h_{7} \vert \Delta k \vert +\mathcal{O}\bigl( \vert \Delta k \vert ^{2}\bigr), \end{aligned} $$

(42)

which is consistent with those given by the nearest-neighboring tight-binding model. The dispersion $E^{K}_{2,3}$ is Dirac-like as long as $h_{7} \neq 0$. The situation of the $K'$ point can be obtained by performing the reflection symmetry with respect to the y axis.

The above analysis solely relies on the non-Abelian nature of the point group and therefore is widely applicable to the orbital-active Dirac material, independent of the origin of the orbitals.

6 Gap opening mechanism

We have shown that the symmetry of the honeycomb lattice protects the degeneracy of the band structure at $K(K')$ point and Γ point. The degenerate states form the 2-dimensional irrep of the little group at the high symmetry points. The degeneracy can be lifted by including various symmetry-breaking terms in the Hamiltonian, which introduces gaps at the Dirac point or/and the quadratic band touching point. The interplay of different symmetry-breaking terms can give rise to various topological band structures, rendering the orbital active Dirac system a flexible platform for realizing topological insulators with different edge-state properties. In this section, we discuss the gap opening mechanisms for different orbital doublets of the E irrep, previously studied in different contexts [8, 15], in a unified manner.

Based on Eq. (21) and Eq. (25), the degenerate wavefunctions at Γ and K can be grouped into circular polarized orbital state $\gamma _{x} \pm i\gamma _{y}$ with opposite chirality. As a result, a $\sigma _{2}$ term in the orbital space, which measures the chirality, is able to lift the degeneracy at both Γ and K points. In addition, at $K(K')$ points, the two complex orbital states only occupy A and B sublattices, respectively. As a result, a $\tau _{3}$ term in the sublattice space can also gap out the Dirac points. In contrast, since the degenerate states at the Γ point have the same weight on both sublattices, they remain degenerate after the $\tau _{3}$ term is added to the Hamiltonian. One can also add other terms to the $k\cdot p$ Hamiltonian to open up a gap in the spectrum. But the two terms mentioned above, $\tau _{3}$ and $\sigma _{2}$, denoted as $H_{m}$ and $H_{\lambda}$ respectively, are among the simplest and have a clear physical origin. In the real space, they have the following form,

$$ \begin{aligned} &H_{m}=m \biggl\{ \sum _{\vec{r}\in A ,\sigma} \gamma ^{\dagger}_{ \sigma}(\vec{r}) \gamma _{\sigma}(\vec{r})-\sum_{\vec{r}\in B , \sigma} \gamma ^{\dagger}_{\sigma}(\vec{r}) \gamma _{\sigma}(\vec{r}) \biggr\} , \\ &H_{\lambda}=\lambda \biggl\{ \sum_{\vec{r} \in A, B} i \gamma _{x}^{\dagger }(\vec{r}) \gamma _{y} (\vec{r} ) + h.c. \biggr\} . \end{aligned} $$

(43)

The term $H_{m}$ represents the staggering mass resulting from the imbalance between the A and B sublattices, which for example, occurs in TMD materials. It only depends on the particle number on each sublattice and does not rely on the particular orbital state the electrons occupy. On the other hand, $H_{\lambda}$ measures the chirality of the orbital and originates from spin-orbit coupling $-\lambda _{0}\vec{L}\cdot \vec{S}$, where $\lambda _{0}$ is the atomic spin-orbit coupling strength, L⃗ is the physical angular momentum operator and S⃗ is the spin operator of electrons.

In free space, L⃗ acts on the Hilbert space labeled by the angular momentum s, p, d, etc. In the case of the planar and buckled honeycomb lattices, the spherical symmetry reduces to the site symmetry $C_{3v}$. As the result, the physical angular momentum L⃗ should be projected into the 2d irrep. The result depends on the particular orbital realizations of the irrep, even though they are equivalent under the $C_{3v}$ point group. In the following, we discuss the different orbital realizations case by case.

In the case of the $(p_{x}, p_{y})$ and $(d_{xz}, d_{yz})$ doublets, the circular polarized orbital state have angular momentum ±1. The $L_{z}$ operator, projecting into the two-dimensional space, becomes $\sigma _{2}$, while $L_{x}$ and $L_{y}$ are zero. The spin orbit coupling $-\lambda \vec{L}\cdot \vec{s}$ becomes $-\lambda \sigma _{y} s_{z}$. As a result, after the spin-orbit coupling is included in the Hamiltonian, which lifts the degeneracy at the Γ point and the K points, the complex orbital state with positive chirality, $p_{x} + i p_{y}$ or $d_{xz} + i d_{yz}$, has higher energy than its partner, as shown in Fig. 6(a). In the case of the $(d_{xy}, d_{x^{2}-y^{2}})$ doublet, the complex orbital states carry angular momentum ∓2. The $L_{z}$ operator in this space is $-2\sigma _{y}$ while the other components vanish. Therefore, the spin-orbit coupling term is $2\lambda _{0} \sigma _{y} s_{z}$. Note the factor of 2 and extra minus sign compared with the previous two cases. Therefore, the complex orbital state $d_{xy} + i d_{x^{2}-y^{2}}$ has lower energy than its partner at the Γ point and the K points when the degeneracy is lifted, as shown in Fig. 6(b).

Figure 6

(a) The gap opening pattern from the $H_{\lambda}$ term for the $(p_{x}, p_{y})$ and $(d_{xz},d_{yz})$ doublets. (b) The gap opening pattern from the $H_{\lambda}$ term for the $d_{xy}/d_{x^{2}-y^{2}}$ doublet. (c) The gap opening pattern from the stagger mass term for all cases. The complex orbitals state naturally occurs at the $K (K')$ point

The $E_{g}$ doublet is special. As discussed in Sect. 3, the complex orbital combinations do not carry angular momentum. The angular momentum operator L, projecting into this space, vanishes for all components. The $\sigma _{y}$ term measure the octupole momentum $\hat{f}_{xyz}$ instead, which is the lowest rank of non-vanishing multipole order for $E_{g}$ orbitals. On the level of single-particle physics, the $\sigma _{y}$ term cannot be obtained directly from the spin-orbit coupling in the $E_{g}$ space. It can appear as a result of second-order perturbation, taking into account the virtual excitation from the $E_{g}$ orbitals to the $t_{2g}$ orbitals. As discussed in Eq. (3), the $T_{2g}$ orbitals splits into one 1d irrep $A_{1}$ and one 2d irrep E of the site symmetry group $C_{3v}$, which in general have distinct onsite energies. We denote the energy difference from the two irreps derived from the $T_{2g}$ orbitals to the $E_{g}$ orbitals as $\Delta _{1}$ and $\Delta _{2}$, respectively. The second order spin-orbit coupling reads,

$$\begin{aligned} H_{\lambda}' =&-\lambda _{0}^{2}P_{E_{g}} \biggl\{ \frac{\vec{L}\otimes \vec{S} P_{A_{1g}}\vec{L}\otimes \vec{S}}{\Delta _{1}}+ \frac{\vec{L}\otimes \vec{S} P_{E}\vec{L}\otimes \vec{S}}{\Delta _{2}} \biggr\} P_{E_{g}} \\ =&-\biggl(\frac{\lambda _{0}^{2}}{2\Delta _{1}}+ \frac{\lambda _{0}^{2}}{\Delta _{2}}\biggr)\sigma _{0} \\ &{}- \frac{\lambda _{0}^{2}}{\Delta _{1}\Delta _{2}}(\Delta _{1}-\Delta _{2}) \sigma _{2}\otimes S_{(1,1,1)}, \end{aligned}$$

(44)

where $P_{E_{g}}$, $P_{E}$ and $P_{A_{1}}$ are the projection operators, and $S_{(1,1,1)}$ is the spin operator along the (1,1,1) direction. The first term is proportional to the identity operator and thus can be absorbed into the chemical potential. The second represents the effective spin-orbit coupling in the $E_{g}$ doublets with the spin-orbit coupling strength,

$$ \begin{aligned} \lambda = \frac{\lambda _{0}^{2}}{\Delta _{1}\Delta _{2}}(\Delta _{1}- \Delta _{2}). \end{aligned} $$

(45)

Recall that $\Delta _{1}-\Delta _{2}$ is the energy difference between the $A_{1}$ and E irreps derived from the $T_{2g}$ orbital in Eq. (3). Therefore, the energy splitting of the $T_{2g}$ triplet under $C_{3v}$ site symmetry is essential for nonzero λ.

The gaps introduced by the $\sigma _{y}$ term in the orbital space are topological. It is straightforward to show that the four bands in Fig. 6(a) and (b) acquire Chern numbers 1, 0, 0, −1 from the bottom to the top. As a result, edge states appear on the boundary of the material. We consider the Hamiltonian on a ribbon with finite width but infinite length, in which case $k_{x}$ remains a good quantum number. The spectrum is plotted in Fig. 7(a) as a function $k_{x}$, showing the edge states between the four bulk bands. The orbital wavefunctions of the edge states are in general complex. The expectation value of the $\sigma _{y}$ operator in the orbital space is indicated by the color bar. When the orbital degree of freedom is the $E_{g}$ doublet, the edge states carry the magnetic octupole moment instead of the dipole moment, sketched in Fig. 7(b).

Figure 7

(a) The spectrum of the Hamiltonian including $H_{\lambda}$ term on a ribbon geometry. λ is set to $0.2t_{\parallel}$. Edge states appear between the bulk bands. The orbital wavefunction of the edges is complex, the chirality indicated by the color bar. (b) The ribbon geometry. When the orbital realization is from the $E_{g}$ doublet, the edge states carry magnetic octupole momentum

The topological gaps are proportional to the coefficient λ of the $\sigma _{2}$ term in the Hamiltonian. Remarkably, when the orbital degrees freedom are realized by the $(p_{x}, p_{y})$, $(d_{xz}, d_{yz})$ or $(d_{xy}, d_{x^{2}-y^{2}})$, λ is directly related to the atomic spin-orbit coupling strength $\lambda _{0}$, which could be quite large for heavy atoms. This leads to a robust topological phase, such quantum spin hall effect, at high temperatures. In contrast, in $E_{g}$ systems, the $\sigma _{2}$ term comes from the second-order perturbation of the spin-orbit coupling. Therefore, in the $E_{g}$ Dirac materials, the band degeneracy is much more stable than other realizations. Even though the quadratic band touching can be gaped out from dynamic spin-orbit coupling generated by interaction, the Dirac points are stable against interaction, making the $E_{g}$ Dirac materials an ideal 2D Dirac semi-metal.

We also present the gap opening pattern from the staggering mass term $H_{m}$ in Fig. 6(c), which is the same for different orbital realizations. $H_{m}$ term leads to a trivial band insulator by itself However, including $H_{m}$ in the presence of $H_{\lambda}$ can lead to richer topological phases with various Chern bands and edge state configurations [15].

7 The interaction effects

The interplay between band structure and interactions can result in various interesting phases of matter, depending on the filling factors. Some of the most interesting phases are discussed below.

For simplicity, let us first consider the spinless fermions. In this case, each site are maximally occupied by two fermions because of the orbital degrees of freedom. At half-filling, i.e., one fermion per site, the Fermi energy is at the Dirac points, which are stable against weak interactions. Therefore, the system remains a Dirac semi-metal for weak interactions but becomes a Mott insulator for strong enough interactions. Consider the simplest on-site interaction

$$ \sum_{i} V n_{x,i} n_{y,i}, $$

(46)

where $n_{x,i}$ and $n_{y,i}$ are the number operators for $p_{x}$ and $p_{y}$ orbitals, respectively. In the large V limit, the system is expected to undergo a phase transition to a Mott insulating phase. The orbital super-exchange is described by a quantum 120^∘ model, which is a frustrated orbital-exchange model. The order-from-disorder analysis show the ground state possesses a $\sqrt {3}\times \sqrt {3}$ type orbital-ordering [44].

At quarter filling, the Fermi level is at the quadratic band touching point, which is unstable against interaction. Infinitesimal interaction opens a gap, leading to the anomalous quantum hall phase that breaks the time-reversal symmetry or a nematic phase that breaks the rotational symmetry [8–10, 21].

As the filling factor becomes smaller than 1/4, the Fermi level is within the flat band when the π-bonding is negelcted, and the system starts to develop different types of order orderings at commensurate fillings [6]. In particular, when the flat band is $1/3$-filled, the localized states close-pack the lattice. If only on-site interactions are considered, such a close-packing many-body state is the exact many-body ground state. This close-packing state breaks the original lattice translation symmetry with an enlarged $\sqrt{3}\times \sqrt{3}$ unit cell, sketched in Fig. 4(b). When long-range interactions are considered, the Wigner crystal appears at even lower fillings.

Furthermore, when the Dirac band is 3/4-filled, the Fermi surface is a regular hexagon by connecting the middle points of the first Brillouin zone edge. This causes Fermi surface nesting by three inequivalent momenta, which makes the system unstable against weak interactions. This can lead to the formation of exotic states of matter, such as orbital density waves or superconductivity [45, 46].

Including the spin degrees of systems can further enrich the aforementioned phases. In this case, the on-site interaction takes the following form:

$$ \begin{aligned} H_{\mathrm{int}} ={}& U \bigl(n_{x}^{\uparrow }n_{x}^{\downarrow }+ n_{y}^{\uparrow }n_{y}^{ \downarrow }\bigr) + \Delta \bigl(\gamma ^{\uparrow ,\dagger}_{x} \gamma ^{ \downarrow ,\dagger}_{x} \gamma ^{\downarrow}_{y} \gamma ^{\uparrow}_{y} + h.c. \bigr) \\ &{}- J \biggl(\vec{S}_{x} \cdot \vec{S}_{y} - \frac{1}{4} n_{x} n_{y}\biggr) + V n_{x}n_{y}, \end{aligned} $$

(47)

where U is the Hubbard interaction, J is the Hund’s coupling, V is the inter-orbital repulsion and Δ is the pairing hopping term. The site index i is omitted.

In the spinful case, at the half-filling, each site is occupied by two electrons. Because of the strong intra-orbital repulsion and Hund’s coupling, two electrons prefer to stay in two different orbitals and form a triplet. As a result, the low-energy effective theory of the Mott insulator is described by a spin-1 Heisenberg model on the honeycomb lattice where the orbital degrees of freedom are inert, which is in sharp contrast with the spinless model.

At the quarter-filling, the bottom two spinful bands are filled. The Fermi surface is right at the quadratic band-touching point. When the interaction is weak, it dynamically generates the spin-orbital coupling term, which gives rise to the quantum spin Hall effect. As the interaction strength grows, the system becomes a Mott insulator where each site is occupied by one electron with both orbital and spin degrees of freedom, and the system is expected to form both magnetic and orbital order.

At filling one-eighth, the Fermi surface is within the bottom two spinful bands. In the absence of π-bonding, the two bands become flat, which enhances the interaction effect. Due to the Coulomb interaction, the system favors a flat-band ferromagnetic state [47]. Therefore, effectively, one of the spinful flat bands is filled, and the Fermi surface is at the quadratic band touching point again. The resulting weak interaction phase exhibits the anomalous quantum Hall effect [21].

As the filling becomes even lower, the systems start to Wigner-crystallize. When spin-orbit coupling is included, the flat band becomes nearly flat and acquires Chern number ±1. In this case, the Chern fractional insulator [43] may become a ground state candidate and competes with the Wigner crystal phase.

8 Discussion and summary

We have studied the orbital-active Dirac materials in a unified manner. The various orbital realizations can be understood as the irreps of the point group symmetry $C_{3v}$. All belonging to the two-dimensional E irrep of $C_{3v}$, the $(p_{x}, p_{y})$ doublet, the $(d_{xy}, d_{x^{2}-y^{2}})$ doublet and the $E_{g}$ doublet can be mapped to each other, and the Dirac materials based on these two sets of different doublet have the same universal properties. Using $k\cdot p$ theory, we demonstrate that the symmetry leads to the orbital enriched Dirac cone at $K(K^{\prime})$ point and quadratic band touching at the Γ point. The symmetry also enforces the unique orbital configuration of the wavefunction at these high symmetry points. When only the σ-bonding is considered, the spectrum hosts two flat bands, which can lead to exotic phases such as the Wigner crystal.

Compared with other doublets, the $E_{g}$ doublet exhibits unique features. First, this doublet is naturally realized in a buckled honeycomb lattice instead of the planar one, leading to a distinct pattern of the Wigner crystal. Furthermore, in the $E_{g}$ doublet, the angular momentum is completely quenched, and the lowest order of the magnetic moment is the octupole moment. This leads to edge states carrying octupole moment once a topological gap is opened.

Orbital active Dirac materials are not only limited to the electronic systems but also include systems of phonons and polaritons [31–36], where their polarization modes realize the orbital degrees of freedom. The symmetry argument in Sect. 5 also enforces chiral valley phonons in materials with a honeycomb structure, such as boron nitride and transition metal dichalcogenides. Thus the interplay between the chirality of electrons’ wavefunction and the chirality of the phonons opens a new door for valleytronics.

Finally, we briefly discuss the band flatness of the Majorana fermions. One dimensional Majorana edge modes can appear with flat dispersion as protected by time-reversal symmetry [48]. The divergence of density of states leads to interesting interaction effects by lifting band flatness via spontaneous time-reversal symmetry breaking. Majorana modes with the cubic dispersion relation can also be realized as the surface state with high topological index superconductivity [49]. Its density of states diverges at $k=0$, which can be viewed as nearly flat.

Appendices

Appendix A: The $C_{3V}$ group and its double group $C_{3V}^{D}$

The $C_{3v}$ point group is the simplest non-abelian group, containing six elements generating by a three-fold rotation and an in-plane reflection. It has three irreps $A_{1}$, $A_{2}$ and E. The first two are one-dimensional while the last one is two-dimensional. The $A_{1}$ irrep is trivial and examples include s orbitals and $p_{z}$ orbitals; the $A_{2}$ irrep is odd under the reflection with realizations such as pseudovector $L_{z}$ and f orbital $y(3x^{2}-y^{2})$. In this work, we are mostly interested in the two-dimensional E irrep.

The $C_{3v}$ group includes 6 operations in 3 conjugacy classes: the identity I, the 3-fold rotations $\{C_{3}^{1}, C_{3}^{2}\}$ around the vertical axis, and the reflection operations with respect to three vertical planes $\{\sigma _{v_{i}}\}$ with $i=1\sim 3$. It possesses two one-dimensional representations $A_{1}$ and $A_{2}$, and one two-dimensional representation E. Their character table is presented in Table 1. The bases of the $A_{1,2}$ representations carry angular momentum quantum number $L_{z}=0$, and those of the E representation can be chosen with $L_{z}=\pm 1$.

Table 1 The character table of the $C_{3v}$ group, which has two one dimensional representations $A_{1,2}$ and one two-dimensional representation E. $A_{1,2}$ carry orbital angular momentum $L_{z}=0$, and E carries $L_{z}=\pm 1$

In the presence of spin-orbit coupling, $C_{3v}$ is augmented to its double group $C_{3v}^{D}=C_{3v} + \bar{C}_{3v}$. $\bar{C}_{3v}=\bar{I} C_{3v}$ is the coset by multiplying Ī to $C_{3v}$, where Ī is the rotation of 2π. The $C^{D}_{3v}$ group has six conjugacy classes, and hence six non-equivalent irreducible representations whose characteristic table is presented in Table 2. $A_{1,2}$ and E remain the representations of $C_{3v}^{D}$ of integer angular momentum, for which Ī is the same as the identity operation. In addition, $C^{D}_{3V}$ also possesses half-integer angular momentum representations, for which Ī is represented as the negative of the identity matrix. For example, a new two-dimensional representation $E_{\frac{1}{2}}$ appears corresponding to the angular momentum $J_{z}=\pm \frac{1}{2}$. The cases of $J_{z}=\pm \frac{3}{2}$ are often denoted as the $E_{\frac{3}{2}}$ representation. Actually, they are not an irreducible two-dimensional representation, but two non-equivalent one-dimensional representations. The two bases of $\psi _{J_{z}=\pm \frac{3}{2}}$ are equivalent under the 3-fold rotations since $\frac{3}{2}\equiv -\frac{3}{2} (\text{mod} 3)$, and neither of them are eigenstates of the reflections $\sigma _{v}$ and $\bar{\sigma}_{v}=\bar{I} \sigma _{v}$. Instead, their superpositions $\frac{1}{\sqrt {2}} (\psi _{\frac{3}{2}}\pm i\psi _{-\frac{3}{2}})$ carry the characters of ±i for $\sigma _{v}$ and ∓i for $\bar{\sigma}_{v}$, respectively.

Table 2 Spinor representations for the $C_{3v}^{D}$ group: The two-dimensional representation $E_{\frac{1}{2}}$ is of $J_{z}=\pm \frac{1}{2}$. $E_{\frac{3}{2}}$ splits into two non-equivalent one-dimensional representations with different characters under vertical reflections

Appendix B: Spherical tensor operators in the d-orbital space

In the Hilbert space of d orbitals, the angular momentum operators are defined in the standard way

$$\begin{array}{rl}& {\hat{L}}_x=\left(\begin{array}{ccccc}0& 1& 0& 0& 0\\ 1& 0& \sqrt{\frac{3}{2}}& 0& 0\\ 0& \sqrt{\frac{3}{2}}& 0& \sqrt{\frac{3}{2}}& 0\\ 0& 0& \sqrt{\frac{3}{2}}& 0& 1\\ 0& 0& 0& 1& 0\end{array}\right),\\ & {\hat{L}}_y=\left(\begin{array}{ccccc}0& -i& 0& 0& 0\\ i& 0& -i\sqrt{\frac{3}{2}}& 0& 0\\ 0& i\sqrt{\frac{3}{2}}& 0& -i\sqrt{\frac{3}{2}}& 0\\ 0& 0& i\sqrt{\frac{3}{2}}& 0& -i\\ 0& 0& 0& i& 0\end{array}\right),\\ & {\hat{L}}_z=\left(\begin{array}{ccccc}2& 0& 0& 0& 0\\ 0& 1& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& -1& 0\\ 0& 0& 0& 0& -2\end{array}\right).\end{array}$$

(B.1)

The total angular momentum operator $\hat{L}^{2} =\hat{L}_{x}^{2}+\hat{L}_{y}^{2}+\hat{L}_{z}^{2}$, and the ladder operators $\hat{L}_{\pm}=\hat{L}_{x} \pm \hat{L}_{y}$. The spherical tensors $\hat{Y}_{lm}$ satisfy the following commutation relation,

$$ [ L_{+}, Y_{l,m} ]=\sqrt{(l-m) (l+m+1)}Y_{l,m+1}. $$

(B.2)

Fixing l, the tensor operator with the lowest m can be easily expressed as powers of $\hat{L}_{-}$,

$$ \hat{Y}_{l,-l}=\frac{\sqrt{(2l) !}}{2^{l} l!}(\hat{L}_{-})^{l}. $$

(B.3)

Based on these relations, the general rank l spherical tensors can be constructed systematically from the angular momentum operators. All 25 linear independent operators acting on d orbitals can be organized into spherical tensor operators with rank $0\sim 4$. The rank 1 tensor operators are,

$$ \begin{aligned} \hat{Y}_{1,-1}=\frac{1}{\sqrt{2}}\hat{L}_{-} ,\qquad \hat{Y}_{1,0}=\hat{L}_{z},\qquad \hat{Y}_{1,1}=-\frac{1}{\sqrt{2}} \hat{L}_{+} .\end{aligned} $$

(B.4)

The rank 2 tensor operators are

$$ \begin{aligned} &\hat{Y}_{2,-2}=\sqrt{\frac{3}{8}} \hat{L}_{-}^{2} , \qquad \hat{Y}_{2,-1}= \sqrt{ \frac{3}{2}} \overline{\hat{L}_{-} \hat{L}_{z}}, \\ & \hat{Y}_{2,0}= \frac{1}{2} \bigl(2\hat{L}_{z}^{2}- \hat{L}_{x}^{2}-\hat{L}_{y}^{2}\bigr),\qquad \hat{Y}_{2,1}=-\sqrt{\frac{3}{2}} \overline{\hat{L}_{+} \hat{L}_{z}}, \\ &\hat{Y}_{2,2}=\sqrt{\frac{3}{8}} \hat{L}_{+}^{2}, \end{aligned} $$

(B.5)

where bars over the operators represent the average over all possible operators ordering. The rank three tensor operators are

$$ \begin{aligned} &\hat{Y}_{3,-3}=\frac{\sqrt {5}}{4} \hat{L}_{-}^{3}, \qquad \hat{Y}_{3,-2}= \sqrt{ \frac{15}{8}} \overline{\hat{L}_{-}^{2} \hat{L}_{z}}, \\ &\hat{Y}_{3,-1}=\frac{\sqrt{3}}{4} \overline{\hat{L}_{-}\bigl(4L_{z}^{2}-L_{x}^{2}-L_{y}^{2} \bigr)}, \\ & \hat{Y}_{3,0}= \frac{1}{2} \overline{L_{z} \bigl(2\hat{L}_{z}^{2}-3 L_{x}^{2}-3L_{y}^{2} \bigr)}, \\ &\hat{Y}_{3,1}=-\frac{\sqrt{3}}{4} \overline{\hat{L}_{+} \bigl(4L_{z}^{2}-L_{x}^{2}-L_{y}^{2} \bigr)}, \\ &\hat{Y}_{3,3}= \sqrt{\frac{15}{8}} \overline{\hat{L}_{+}^{2} \hat{L}_{z}},\qquad \hat{Y}_{3,-3}=- \frac{\sqrt {5}}{4} \hat{L}_{+}^{3}. \end{aligned} $$

(B.6)

Lastly, the rank 4 tensors are

$$\begin{aligned}& \hat{Y}_{4,-4}=\sqrt{\frac{35}{128}} \hat{L}_{-}^{4}, \qquad \hat{Y}_{4,-3}= \frac{\sqrt {35}}{4} \overline{\hat{L}_{-}^{3} \hat{L}_{z}}, \\& \hat{Y}_{4,-2}=\sqrt{\frac{5}{32}} \overline{\hat{L}_{-}^{2} \bigl(7 \hat{L}_{z}^{2}-\hat{L}^{2}\bigr)}, \\& \hat{Y}_{4,-1}= \frac{\sqrt {5}}{4} \overline{\hat{L}_{-}\hat{L}_{z} \bigl(7\hat{L}_{z}^{2}-3\hat{L}^{2} \bigr)}, \\& \hat{Y}_{4,0}=\frac{1}{8} \overline {\bigl(35 \hat{L}_{z}^{4}-30 \hat{L}_{z}^{2} \hat{L}^{2}+3 \hat{L}^{4}\bigr)}, \\& \hat{Y}_{4,1}=- \frac{\sqrt {5}}{4} \overline{\hat{L}_{+}\hat{L}_{z} \bigl(7 \hat{L}_{z}^{2}-3\hat{L}^{2}\bigr)}, \\& \hat{Y}_{4,2}=\sqrt{\frac{5}{32}} \overline{\hat{L}_{+}^{2} \bigl(7 \hat{L}_{z}^{2}-\hat{L}^{2}\bigr)}, \\& \hat{Y}_{4,3}=-\frac{\sqrt {35}}{4} \overline{\hat{L}_{+}^{3} \hat{L}_{z}}, \qquad \hat{Y}_{4,4}=\sqrt{ \frac{35}{128}} \hat{L}_{+}^{4}. \end{aligned}$$

(B.7)