Introduction

Twisted bilayer graphene (TBG) has drawn significant attention recently due to the observations of the correlated insulating phases, anomalous Hall effect, and unconventional superconductivity1,2,3,4,5,6,7,8,9,10. At small twist angles, the low-energy states of TBG are characterized by two low-energy bands for each valley and spin degrees of freedom11,12. Around the “magic angles”, the bandwidths of the low-energy bands become very small, and these nearly flat bands are believed to be responsible for most of the unconventional properties observed in TBG. Numerous theories have been proposed to understand the intriguing phenomena observed in TBG13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45.

Recently unconventional superconducting and correlated insulating behavior, have been observed in twisted double bilayer graphene46,47,48 and trilayer graphene with hexagonal boron nitride (hBN) substrate49. Quantum anomalous Hall effect with Chern number 2 has also been observed in trilayer graphene with hBN substrate50. These experimental works have stimulated extensive theoretical interests51,52,53,54,55,56. In particular, it has been proposed that topological flat bands generically exist in twisted double bilayer52,53,54 and twisted multilayer graphene systems53, and that the topological flat bands with non-vanishing valley Chern numbers are associated with large and valley-contrasting orbital magnetizatons, which may lead to an orbital ferromagnetic state once the valley symmetry is broken either spontaneously or due to external magnetic fields53. The orbital ferromagnetic states are also believed to exist in TBG aligned with hBN substrate4,9,35, in which (quantum) anomalous Hall effect has been observed at 3/4 filling of the flat bands around the magic angle4,10.

The orbital ferromagnetic state is a completely new state in condensed matter systems. It is then important to ask what are the most unique properties and how to probe this new state of matter. In typical magnetic materials, anomalous Hall effect results from the interplay between spin ferromagnetism and spin-orbit coupling (SOC)57: time-reversal (\({\mathcal{T}}\)) symmetry is first broken in the spin sector leading to spin magnetizations, then the \({\mathcal{T}}\) symmetry breaking is transmitted from the spin sector to the orbital sector via SOC. In orbital ferroamgnetic systems, however, \({\mathcal{T}}\) symmetry is directly broken in the orbital sector, and the anomalous Hall effect (AHE) does not require any microscopic SOC. One thus expects that the AHE in an orbital ferromagnetic system would be much more conspicuous than that in a spin ferromagnetic system, and that the AHE will be quantized as nonzero integers if the orbital magnet is insulating. Such argument also applies to magneto-optical effects. As light is directly coupled to the orbital degrees of freedom of electrons, the magneto-optical effects in an orbital ferromagnet should be much more pronounced than those in a spin ferromagnet. Therefore, we expect that there would be significant magneto-optical responses in both hBN-aligned TBG and twsited double bilayer graphene (TDBG). On the other hand, inversion symmetry is broken in both systems, which allows for nonlinear optical effects58,59. It is intriguing to ask whether the orbital magnetization and valley polarization in hBN-aligned TBG and TDBG can be probed by nonlinear optical responses. Even without orbital magnetism, the nonlinear optical properties of the twisted graphene systems from structural inversion-symmetry breaking is still an open question, which deserves a comprehensive study.

In this paper, we systematically study the AHE, magneto-optical properties, and nonlinear optical properties of both hBN-aligned TBG and TDBG. We find that in additional to AHE, there are also giant magneto-optical Kerr and Faraday rotations in both systems by virtue of the orbital magnetization induced by the valley polarization. Therefore, we propose that the Faraday and Kerr rotations can be a powerful tool to detect the presence of orbital magnetism in twisted graphene systems. Moreover, we also study the nonlinear optical responses such as the shift current and second harmonic generation (SHG) in hBN-aligned TBG and TDBG. We find that both systems exhibit colossal nonlinear optical responses by virtue of the small bandwidths and the small excitation gaps of the twisted graphene systems. To be specific, our calculations indicate that in either system, the shift-current conductivity is on the order of \(1{0}^{3}\ \mu \)A/V\({}^{2}\), and the SHG susceptibility is on the order of \(1{0}^{6}\) pm/V in the terahertz and infrared frequency regime. In TDBG with \(AB\)-\(BA\) stacking, we propose that the non-vanishing valley polarization (orbital magnetization) would generate a new component of the nonlinear photoconductivity, which will exhibit remarkable hysteresis behavior in response to out-of-plane magnetic fields. The nonlinear photoconductivities that are generated and/or enhanced by the orbital magnetizations may be considered as strong experimental evidence for the valley polarization and orbital ferromagnetism in the twisted graphene systems.

Results

The TBG system aligned with hexagonal BN substrate

We first study the AHE, magneto-optical properties, and nonlinear optical properties of the hBN-aligned TBG system. We consider the situation that TBG is placed on top of a hBN substrate, and the hBN substrate is aligned with the bottom graphene layer. This is actually the situation considered in refs. 4,10 in which an anomalous Hall conductivity as large as \(2.4{e}^{2}/h\) has been observed at 3/4 filling of the conduction flat band around the magic angle. The hBN substrate is believed to have two effects on the electronic structures of TBG. First, the alignment of the hBN substrate with the bottom graphene layer would impose a staggered sublattice potential on the bottom layer graphene and break the \({C}_{2z}\) symmetry, which opens a gap at the Dirac points of the flat bands of the magic-angle TBG. Actually the two flat bands for the \(K\) valley acquires nonzero Chern numbers \(\pm\!1\) (±1 for the \(K^{\prime} \) valley) once a gap is opened up at the Dirac points. Second, the hBN substrate would generate a new moiré pattern, which roughly has the same period as the one generated by the twist of the two graphene layers, but are orthogonal to each other60. However, the moiré potential generated by the hBN substrate is one order of magnitude weaker than that generated by the twist of the two graphene layers60,61. Therefore, as a leading-order approximation, it is legitimate to neglect the moiré potential generated by the hBN substrate25,37, and only consider the staggered sublattice potential on the bottom layer.

In order to study the effects of breaking the valley and spin symmetries, we artificially apply valley and spin energy splittings

$${H}^{\mu s}={H}_{0}^{\mu }+\mu {E}_{v}{\tau }_{z}+s{E}_{s}{s}_{z},$$
(1)

where \({E}_{v}\) and \({E}_{s}\) are positive real numbers denoting the valley and spin splittings, and \(\mu\!=\!\pm\! 1\) and \(s\!=\!\pm\!1\) represent the valley and spin indices respectively. \({\tau }_{z}\) and \({s}_{z}\) both denote the third Pauli matrix, and are defined in the valley and spin subspaces respectively. \({H}_{0}^{\mu }\) is the effective Hamiltonian for hBN-aligned TBG, which is given in Eq. (12) in Methods section. The bandstructures at the first magic angle \(\theta \ =\ 1.0{5}^{\circ }\) are shown in Fig. 1, where the solid blue lines, dashed blue lines, solid red lines, and dashed red lines denote the bandstructures of electrons with {\(\mu\!=\!-\!1\), \(s\!=\!-\!1\)}, {\(\mu\!=\!-\!1\), \(s\!=\!+\!1\)}, {\(\mu\!=\!+\!1\), \(s\!=\!-\!1\)}, and {\(\mu\!=\!+\!1\), \(s\!=\!+\!1\)} respectively. The thick dashed gray line denotes the chemical potential for some given valley and spin splittings, which is determined by the charge filling \(+3/4\), i.e., filling 7 out of the 8 flat bands including the valley and spin degrees of freedom. The high symmetry points \({\Gamma }_{s}\), \({M}_{s}\), \({K}_{s}\), \(K^{\prime}_s\) are marked in the moiré Brillouin zone (BZ) as shown in the inset of Fig. 1. The bandstructures are all plotted along the high-symmetry path marked by the solid gray lines in the moiré BZ. When \({E}_{v} = 0\), \({E}_{s} = 0\), the bandstructures are spin degenerate for each valley as shown in Fig. 1a. When \({E}_{v}=3\) meV and \({E}_{s}=0\), the bandstructures are shown in Fig. 1b. Clearly the two valleys have been splitted: the flat bands of the \(K\) valley are completely filled, while the conduction band of the \(K^{\prime} \) valley is half filled. Later we will show that the AHE, orbital magnetization, and magneto-optical effects would be maximal in such a situation. In Fig. 1c we show the bandstructures with \({E}_{v} = 0\) and \({E}_{s} = 3\) meV. We see that the spins have been splitted but the valley symmetry is still preserved. In this situation, both AHE and magneto-optical effects vanish since \({\mathcal{T}}\) symmetry is still preserved in the orbital sector. In Fig. 1d, we show the bandstructures with \({E}_{v}=3\) meV and \({E}_{s}=3\) meV at +3/4 filling. In this situation, both the spin and valley splittings are strong enough such that both the valley and spin polarizations \({\xi }_{v}\) and \({\xi }_{s}\) reach their maximal values with \({\xi }_{v}={\xi }_{s}=1/7\).Footnote 1 Then the system enters a quantum anomalous Hall (QAH) insulating phase with Chern number \(-1\), which would give rise to the QAH effect reported in ref. 10.

Fig. 1: The bandstructures of hBN-aligned TBG.
figure 1

a \({E}_{v}=0\), \({E}_{v}=0\), b \({E}_{v}=3\ \)meV and \({E}_{s}=0\), c \({E}_{v}=0\) and \({E}_{s}=3\ \)meV, d \({E}_{v}=3\ \)meV and \({E}_{v}=3\ \)meV. The moiré Brillouin zone is shown in the inset of a, where the locations of the high symmetry points \({\Gamma }_{s}\), \({M}_{s}\), \({K}_{s}\), and \(K^{\prime}_s\) are marked. The bandstructures are plotted along the high-symmetry paths marked by the solid gray lines.

Full size image

It is important to note that as a result of the staggered sublattice potential from the hBN substrate, a gap \( \sim \ 4\ \)meV has opened up at the Dirac points \({K}_{s}\) and \(K^{\prime}_s\) as shown in Fig. 1. The \({C}_{2z}\) symmetry breaking and the gap opening at the Dirac points are essential in achieving AHE, magneto-optical effect and nonlinear optic effects in the TBG system. If \({C}_{2z}\) symmetry is preserved, both \({C}_{2z}\) and time-reversal (\({\mathcal{T}}\)) would transform the Hamiltonian of one valley to that of the opposite valley, thus the combined symmetry operation \({C}_{2z}{\mathcal{T}}\) is a symmetry for the Hamiltonian of each valley, which would enforce the Berry curvature to be zero at every \({\bf{k}}\) point. As a result, both the AHE and magneto-optical effects would be forbidden. The \({C}_{2z}\) operation also connects the two valleys. If \({C}_{2z}\) symmetry is preserved, the nonlinear optical response is also prohibited as the contributions from the two valleys would exactly cancel each other.

We take the valley and spin splittings (\({E}_{v}\) and \({E}_{s}\) in Eq. (1)) as two free parameters which are varied from 0 to \(3\ \)meV. Then we study the dependence of AHE, orbital magnetization, and magneto-optical effects on the valley and spin splittings at +3/4 filling. In Fig. 2a we first show the dependence of the anomalous Hall conductivity (in units of \({e}^{2}/h\)) on \({E}_{v}\) and \({E}_{s}\). One may notice in Fig. 2a that there is a small region in the upper right corner in which \({\sigma }_{xy}\) is quantized as \(-{e}^{2}/h\). This is the region in which both the valley and spin polarizations reach their maximal values \({\xi }_{v}={\xi }_{s}=1/7\), and the system enters a QAH phase with quantized Hall plateau. We note that \({E}_{v} \approx {E}_{s} \approx 2.5\) meV would be strong enough to approach the QAH phase.

Fig. 2: Anomalous Hall conductivity and orbital magnetization of hBN-aligned TBG.
figure 2

a Anomalous Hall conductivity \({\sigma }_{xy}\) (in units of \({e}^{2}/h\)) of 3/4-filled TBG aligned with hBN substrate at the magic angle. b The orbital magnetization of 3/4-filled hBN-aligned TBG at the magic angle, in units of \({\mu }_{B}\) per moire unit cell. The vertical and horizontal axes denote the valley and spin splittings respectively, in units of meV.

Full size image

It is also interesting to note that when \({E}_{v}\sim 3\) meV and \({E}_{s}\sim 0\) meV, i.e., in the upper left corner of Fig. 2a, the anomalous Hall conductivity (AHC) is maximal with \({\sigma }_{xy}\! \sim \! -1.9 {e}^{2}/h\), and the system is metallic as shown by the bandstructures in Fig. 1b. Such a large AHC results from the inhomogeneous distribution of the Berry curvature in Brillouin zone. At 3/4 filling with zero spin splitting and a positive valley splitting, the states of the \(K\) valley is fully occupied and does not contribute to AHE; while the \(K^{\prime} \) valley is half filled, and the chemical potential is just slightly above the (spin degenerate) conduction band minimum due to the giant density of states, as shown in Fig. 1b. On the other hand, the Berry curvature of the conduction band is mostly distributed near \({\Gamma }_{s}\) point, which implies that the conduction band of the \(K^{\prime} \) valley barely contributes to the total AHC, which is still around \(-2{e}^{2}/h\) due to the spin degenerate valence band (of the \(K^{\prime} \) valley) with Chern number \(-1\). Fixing \({E}_{v}\sim 3\) meV, \({\sigma }_{xy}\) would decrease from \(-1.9\ {e}^{2}/h\) to \(-{e}^{2}/h\) as \({E}_{s}\) increases from 0 to \(3\ \)meV, and the system would go through a transition from a metal to a QAH insulator. We expect that such a phase transition may be assisted by applying in plane magnetic field which only couples to the spin magnetization.

In Fig. 2b we show the orbital magnetization (in units of \({\mu }_{B}\) per moiré primitive cell) in the parameter space spanned by \({E}_{v}\) and \({E}_{s}\). The orbital magnetization can be as large as \( \sim -1\ {\mu }_{B}\) when \({E}_{v}\sim 3\) meV and \({E}_{s} \sim 0\) meV, and gradually decreases to \(\sim\! 0.1\,{\mu }_{B}\) as \({E}_{s}\) increases to \(\sim 3\)meV. On the other hand, in the QAH phase with \({E}_{v} \approx {E}_{s}\sim 3\) meV, the spin polarization is fully saturated, and the spin magnetization is as large as \(1\ {\mu }_{B}\) per moiré cell, indicating that the magnetization in the QAH phase is dominated by the spin component. However, the orbital magnetic order is extremely anisotropic, which breaks a discrete \({{\mathbb{Z}}}_{2}\) symmetry; while the spin magnetic order is isotropic due to the absence of atomic SOC, which breaks continuous spin rotational symmetry. Therefore, despite being small in magnitude, the orbital magnetization is expected to be much more robust to thermal fluctuations according to Mermin-Wagner theorem.

In Fig. 3a, b, we present the Faraday and Kerr rotations (denoted by \({\theta }_{F}\) and \({\theta }_{K}\)) as the valley and spin splittings \({E}_{v}\) and \({E}_{s}\) increases from 0 to \(3\ \)meV. We consider the case that the incident light is normal to the two dimensional (2D) plane, and we first fix the incident photon energy \(\hslash \omega = 0.05\) eV. As shown in the figures, both \({\theta }_{F}\) and \({\theta }_{K}\) vanish when \({E}_{v}=0\), and their magnitudes increase with the increase of the valley splittings. When \({E}_{v} \sim 3\) meV, \({\theta }_{F} \sim -0.{4}^{\circ }\) and \({\theta }_{K}\) is as large as \({9}^{\circ }\). In the QAH phase (upper right corner), \({\theta }_{F} \approx -0.{2}^{\circ }\) and \({\theta }_{K}\approx 5.{6}^{\circ }\) with the incident light frequency \(\hslash \omega =0.05\) eV.

Fig. 3: The Kerr and Faraday angles for hBN-aligned TBG.
figure 3

a The Faraday angle \({\theta }_{F}\), and b the Kerr angle \({\theta }_{K}\), for hBN-aligned TBG at the magic angle at 3/4 filling. The vertical and horizontal axes are the valley and spin splittings respectively. For the same system, the frequency dependence of the Faraday angle (c), and the Kerr angle (d). The blue circles and red diamonds represent the situations with \(\{{E}_{v} = 3\, {\rm{meV}},{E}_{s} = 0\}\), and \(\{{E}_{v} = 3\, {\rm{meV}},{E}_{s} = 3\, {\rm{meV}}\}\), respectively.

Full size image

We note that the calculated Kerr rotation in hBN-aligned TBG is at least an order of magnitude greater than those observed in typical spin ferromagnetic materials62,63,64. In the latter, the magneto-optical phenomena result from the interplay between spin ferromagnetism and SOC: the SOC transimits the \({\mathcal{T}}\) symmetry breaking from spin sector to orbital sector, and generates the orbital magnetization. However, in most magnetic materials, the effects of SOC are perturbative compared with the bandwidths. Therefore, the observed \({\theta }_{K}\) are typically small (\( \sim 0.{1}^{\circ }\)) in ferromagnetic transition-metal compounds. On the other hand, in twisted grapehen systems, the Faraday and Kerr rotations directly result from the orbital ferromagnetism as indicated by the significant orbital magnetization shown in Fig. 2b. The spin and orbital magnetization coexist in the hBN-aligned TBG system and are interwined with each other, although there is no microscopic SOC at the single-particle level. Thus it is expected that the Kerr and Faraday rotations in hBN-aligned TBG (with valley symmetry breaking) would exhibit similar behavior as those in Landau levels65,66,67,68 and in quantum anomalous Hall (QAH) insulators69,70. The difference is that here we do not need external magnetic fields nor any SOC to generate the quantized (anomalous) Hall conductivity; instead, the time-reversal symmetry is expected to be broken spontaneously due to Coulomb interactions.

In Fig. 3c, d we plot the frequency dependence of \({\theta }_{F}\) and \({\theta }_{K}\) for {\({E}_{v}=3\ \)meV, \({E}_{s}=0\)} (blue circles) and {\({E}_{v}=3\ \)meV, \({E}_{s}=3\) meV} (red diamonds). Both \({\theta }_{F}\) and \({\theta }_{K}\) increase dramatically as \(\omega \) decreases. In particular, in the QAH phase (\({E}_{v}= 3\) meV, \({E}_{s} = 3\) meV) \({\theta }_{F}=0.4{4}^{\circ }\) and \({\theta }_{K} = -8.{2}^{\circ }\) at \(\hslash \omega = 0.01\) eV; while when \({E}_{v} = 3\) meV and \({E}_{s} = 0\), \({\theta }_{F} = 0.8{7}^{\circ }\), and \({\theta }_{K} = -40.0{4}^{\circ }\) for \(\hslash \omega = 0.01\) eV. Actually for an QAH insulator with Chern number \(C\), in the limit \(\omega \to 0\), \({\theta }_{F}\) should be quantized as integer multiples of the fine-structure constant \(C\alpha \approx C/137\ \)rad\(\approx C\times 0.4{2}^{\circ }\) 69,71, which is consistent with the results shown in Fig. 3c. In the QAH phae, the Kerr angle is predicted to be quantized as \(\pm \pi /2\) in the low-frequency limit69. It worthwhile to note that both \({\theta }_{F}\) and \({\theta }_{K}\) would change signs at \(\hslash \omega \sim 0.035\ \)eV and \(\hslash \omega \sim 0.075\ \)eV. This is because the real part of the optical anomalous Hall conductivity \({\rm{Re}}[\ {\sigma }_{yx}(\omega )\ ]\) changes sign at \(\hslash \omega \approx 0.035\ \)eV and \(0.075\ \)eV, leading to the sign change in the Faraday and Kerr rotations.

We continue to study the nonlinear optical properties of hBN-aligned TBG. In general, the photocurrent \({j}^{c}({\omega }_{3})\) is related to the time-dependent electric fields of light via the second-order photoconductivity: \({j}^{c}({\omega }_{3})={\sum }_{ab}{\sigma }_{ab}^{c}({\omega }_{3})\ {E}_{a}({\omega }_{1})\ {E}_{b}({\omega }_{2})\), where \(a,b,c=x,y,z\) denotes the spatial directions in Cartesian coordinates, and \({\omega }_{1}\) and \({\omega }_{2}\) are the frequencies of the two incident photons, with \({\omega }_{1}+{\omega }_{2}={\omega }_{3}\)59. For monochromatic light, the frequency of the incident photons is fixed as \(\pm\!\omega\), thus there could be two distinct second-order optical processes with \({\omega }_{3}=0\) or \({\omega }_{3}=2\omega \), corresponding to the generation of the shift current and the second harmonic generation respectively59. Regardless of the microscopic mechanism, the nonlinear optical conductivity tensor \({\sigma }_{ab}^{c}(0)\) and \({\sigma }_{ab}^{c}(2\omega )\) have the same properties under symmetry operations, thus they have the same symmetry-allowed expressions. In particular, one may expand \({\sigma }_{ab}^{c}\) to the leading order of the orbital magnetization \({M}_{z}\),

$${\sigma }_{ab}^{c}={\sigma }_{ab,0}^{c}+{\sigma }_{ab,z}^{c}{M}_{z}\,$$
(2)

where \({\sigma }_{ab,0}^{c}\) denotes the component of the nonlinear photoconductivity that is independent of the orbital magnetization, while \({\sigma }_{ab,z}^{c}{M}_{z}\) is the component which is linear in the orbital magnetization \({M}_{z}\), with the coefficient \({\sigma }_{ab,z}^{c}\). The only symmetry the hBN-aligned TBG system has is \({C}_{3z}\), which restricts \({\sigma }_{ab,0}^{c}\) and \({\sigma }_{ab,z}^{c}\) to the following form

$$\begin{array}{l}{\sigma }_{xx,0}^{x}=-{\sigma }_{yy,0}^{x}=-{\sigma }_{xy,0}^{y}=-{\sigma }_{yx,0}^{y}\\{\sigma }_{xx,0}^{y}={\sigma }_{xy,0}^{x}={\sigma }_{yx,0}^{x}=-{\sigma }_{yy,0}^{y}\\{\sigma }_{xx,z}^{x}=-{\sigma }_{yy,z}^{x}=-{\sigma }_{xy,z}^{y}=-{\sigma }_{yx,z}^{y}\\{\sigma }_{xx,z}^{y}={\sigma }_{xy,z}^{x}={\sigma }_{yx,z}^{x}=-{\sigma }_{yy,z}^{y},\end{array}$$
(3)

where we assume the incident light is along the \(z\) direction such that the polarizations of the light are confined to the \(x-y\) plane. It turns out that there are only two independent photoconductivities \({\sigma }_{xx}^{x}\) and \({\sigma }_{xx}^{y}\). Each of them include two components: one is independent of the orbital magnetization \({M}_{z}\), and the other is linear in \({M}_{z}\). The component that is linear in \({M}_{z}\) may vary with perpendicular magnetic field, and show hysteresis behavior due to the hysteresis loop of the orbital magnetization. The nonlinear photocurrents induced by spin magnetization have already been investigated in antiferromagnetic CrI\({}_{3}\) bilayer72,73. Here it is interesting to investigate the nonlinear optical responses induced by pure orbital magnetization.

Microscopically, the shift-current photoconductivity \({\sigma }_{ab}^{c}(0)\) and the second-harmonic photoconductivity \({\sigma }_{ab}^{c}(2\omega )\) can be derived using second-order perturbation theory, which are expressed as74

$$\begin{array}{l}{\sigma }_{ab}^{c}(0)=\frac{{e}^{3}}{{\omega }^{2}}\mathop{\sum }\limits_{\Omega = \pm \omega }\mathop{\sum }\limits_{lmn}\int\frac{d{\bf{k}}}{{(2\pi )}^{d}}\ {\rm{Re}}\ \left[{\phi }_{ab}\ ({f}_{l}-{f}_{n})\ \frac{{v}_{nl}^{a}\ {v}_{lm}^{b}\ {v}_{mn}^{c}}{({E}_{n{\bf{k}}}\,-\,{E}_{m{\bf{k}}}\,-\,i\delta )\ ({E}_{n{\bf{k}}}\,-\,{E}_{l{\bf{k}}}\,+\,\hslash \Omega \,-\,i\delta )}\right]\\{\sigma }_{ab}^{c}(2\omega )=\frac{{e}^{3}}{{\omega }^{2}}\mathop{\sum }\limits_{\Omega = \pm \omega }\mathop{\sum }\limits_{lmn}\int\frac{d{\bf{k}}}{{(2\pi )}^{d}}\ {\phi }_{ab}\ ({f}_{l}-{f}_{n})\ \frac{{v}_{nl}^{a}\ {v}_{lm}^{b}\ {v}_{mn}^{c}}{({E}_{n{\bf{k}}}\,-\,{E}_{m{\bf{k}}}\,-\,2\hslash \Omega \,-\,i\delta )\ ({E}_{n{\bf{k}}}\,-\,{E}_{l{\bf{k}}}\,+\,\hslash \Omega \,-\,i\delta )},\end{array}$$
(4)

where \(l,n,m\) are the band indices, \({\bf{k}}\) is the wavevector in the moiré Brillouin zone, \({E}_{n{\bf{k}}}\) is the eigenenergy of the \(n\)th band at \({\bf{k}}\), \({f}_{n}\) is the Fermi-Dirac distribution with respect to \({E}_{n{\bf{k}}}\), and \({v}_{nl}^{a}\ =\ \left\langle {u}_{n{\bf{k}}}\right|{\partial }_{{k}_{a}}{H}_{{\bf{k}}}\left|{u}_{l{\bf{k}}}\right\rangle /\hslash \) is the matrix element of the velocity operator in the basis of the (periodic part of) Bloch eigenfunctions \(\{\left|{u}_{n{\bf{k}}}\right\rangle \}\). The susceptibility of SHG \({\chi }_{ab}^{c}(2\omega ) = i{\sigma }_{ab}^{c}(2\omega )/(2{\epsilon }_{0}\ \omega )\).

In Fig. 4a we plot the frequency dependence of the shift-current photoconductivities at the +3/4 filling of hBN-aligned TBG, where the blue and red markers denote the situations with \({E}_{v} = {E}_{s} = 0\) and \({E}_{v} = {E}_{s} = 3\) meV respectively, and the circles and diamonds represent \({\sigma }_{xx}^{x}(0)\) and \({\sigma }_{xx}^{y}(0)\) respectively. We note that the photoconductivities \( \sim \pm 4000\ \mu \)A/V\({}^{2}\) at \(\hslash \omega \, \lessapprox \,0.07\) eV, which is unprecedentedly large. As the frequency increases, the photoconductivities can change sign and decrease to \(\sim\!10^{2}\ \mu\)A/V\({}^{2}\) at relatively high frequencies \(\hslash \omega \sim 0.2-0.3\) eV. In Fig. 4b we show the imaginary part of the SHG susceptibilities, where the blue and red markers denote situations with \({E}_{v} = {E}_{s} = 0\) and \({E}_{v} ={E}_{s} = 3\) meV (the QAH phase), and the circles and diamonds represent \({\chi }_{xx}^{x}\) and \({\chi }_{xx}^{y}\) respectively. At low frequencies \(\hslash \omega \, \lessapprox\, 0.1\) the SHG susceptibilities are extremely large, on the order of \(1{0}^{6}\ \)pm/V, and they gradually decrease to \(\sim 1{0}^{3}-1{0}^{4}\) pm/V when \(\hslash \omega \sim 0.2-0.3\,{\rm{eV}}\). Such colossal SHG susceptibilities in the terahertz and infrared frequency regime are orders of magnitudes larger than those observed in other 2D materials with broken inversion symmetry such as monolayer MoS\({}_{2}\)75,76, monolayer WSe\({}_{2}\)77, WS\({}_{2}\)78, and antiferromagnetic bilayer CrI\({}_{3}\)73. In these 2D materials, the reported SHG susceptibilities are typically on the order of \(1{0}^{3}-1{0}^{5}\) pm/V, and the maximal responses typically occur in the visible-light frequency regime. On the other hand, the typical size of TBG sample is a few microns. When the photon wavelength is comparable to the TBG sample size, the photon energy \( \sim 0.5-1\ \)eV. The calculated SHG susceptibility in this frequency regime \( \sim 1{0}^{2}-1{0}^{3}\ \)pm/V, and the shift-current photoconductivities \( \sim 1-10\ \mu \)A/V\({}^{2}\).

Fig. 4: Nonlinear optical properties for hBN-aligned TBG at 3/4 filling.
figure 4

a Shift-current photoconductivities \({\sigma }_{ab}^{c}(0)\), and b the imaginary parts of the SHG susceptibilities \({\rm{Im}}{\chi }_{ab}^{c}(2\omega )\) for the 3/4-filled hBN-aligned TBG at the magic angle. The blue and red markers represent the cases of \({E}_{v} = {E}_{s} =0\) and \({E}_{v} = {E}_{s} =3\) meV respectively. The circles and diamonds denote \({\sigma }_{xx}^{x}\) (or Im\({\chi }_{xx}^{x}\)) and \({\sigma }_{xx}^{y}\) (or Im\({\chi }_{xx}^{y}\)) respectively.

Full size image

The colossal shift-current and SHG responses shown in Fig. 4 can be interpreted as follows. First, it is straightforward to see from Eq. (4) that the SHG and shift-current responses would be significantly enhanced when the one-photon or two-photon energy (\(\omega \) or \(2\omega \)) is in resonance with some excited electronic states at some \({\bf{k}}\) points. Such a resonant enhancement would be much more pronounced if the energy bands are flat, as the flatness of the bands implies that the all electronic states at different \({\bf{k}}\) points would have the same resonant frequency. One could imagine having some sets of perfectly flat bands, e.g., Landau levels, with broken inversion symmetry. Then if the frequency is in resonance with the Landau-level spacing, one would get enormous shift-current and SHG responses. Such giant nonlinear optical effect has not been observed because there is always inversion symmetry in Landau levels of free 2D electrons’ gas or free Dirac fermions in graphene. In hBN-aligned TBG, however, the low-energy states can be interpreted as pseudo Landau levels24 with broken inversion symmetry due to the presence of the hBN substrate. These bands are roughly flat around the magic angle as shown in Fig. 1, which is expected to contribute to giant nonlinear optical responses once the incident photon frequency is somewhere in resonance with the electronic excitations. The electronic excitation gaps in hBN-aligned TBG turn out to be small (see Fig. 1), and are associated with small resonant frequencies, which would further amplify the giant nonlinear optical responses due to the \(1/{\omega }^{2}\) dependence in Eq. (4).

Twisted double bilayer graphene

We continue to study the TDBG system. Again, in order to study the effects of breaking the valley and spin symmetries, we apply artificial valley and spin splittings to the original continuum Hamiltonian of TDBG:

$${H}_{\lambda ,\lambda ;}^{\mu s}={H}_{\lambda ,\lambda ^{\prime} }^{\mu }+\mu {E}_{v}{\tau }_{z}+s{E}_{s}{s}_{z},$$
(5)

where \({H}_{\lambda ,\lambda ^{\prime} }^{\mu s}\) is the continuum model Hamiltonian for TDBG with different stacking configurations, with \(\lambda ,\lambda ^{\prime}=\pm\) denoting the \(AB/BA\) stacked bilayers. The specific expression is given by Eq. (14) in Methods section. \(\mu {E}_{v}{\tau }_{z}\) and \(s{E}_{s}{s}_{z}\) are valley-splitting and spin-splitting terms defined below Eq. (1).

Using the Hamiltonian of Eq. (5), we have calculated the AHC and magneto-optical properties of TDBG with both \(AB\)-\(BA\) and \(AB\)-\(AB\) stackings at different integer fillings. For \(AB\)-\(BA\) stacked TDBG, the low-energy bands possess non-vanishing valley Chern numbers53,54, which would lead to orbital ferromagnetism once the valley symmetry is broken either spontaneously or by external magnetic fields. Such orbital ferromagnetic states are associated with giant AHE and remarkable magneto-optical effects in the terahertz frequency regime. On the other hand, in \(AB\)-\(AB\) stacked TDBG, the orbital magnetization for each valley vanishes as a result of \({C}_{2x}\) symmetry53, which kills both AHE and magneto-optical effects. However, the \({C}_{2x}\) symmetry for each valley can be broken by a vertical displacement fields, leading to isolated topological flat bands52,53,54, which may rise to AHE and Kerr/Faraday rotations once the valley symmetry is broken. We refer the readers to Supplementary Information for more detailed results. In what follows, we will discuss the nonlinear optical properties of TDBG in detail.

We continue to study the nonlinear optical properties of TDBG. Again, before going into the details, we first make symmetry analysis on the nonlinear photo-conductivity tensor. \(AB\)-\(BA\) stacked TDBG has both \({C}_{2y}\) and \({C}_{3z}\) symmetries. The \(K\) and \(K^{\prime} \) valleys are invariant under \({C}_{3z}\) operation, but are interchanged with respect to each other under \({C}_{2y}\) operation. Therefore, for each valley of \(AB\)-\(BA\) stacked TDBG, there is only \({C}_{3z}\) symmetry, and the symmetry-allowed form of the second-order photo-conductivity tensor for each valley is already given by Eq. (3). However, in \(AB\)-\(BA\) TDBG, the additional \({C}_{2y}\) symmetry would impose the following relationship between the photoconductivities of the \(K\) and \(K^{\prime} \) valleys

$$\begin{array}{l}{\sigma }_{xx,0}^{x}(K)=-{\sigma }_{xx,0}^{x}(K^{\prime} )\\{\sigma }_{xx,z}^{x}(K)={\sigma }_{xx,z}^{x}(K^{\prime} )\\{\sigma }_{xx,0}^{y}(K)={\sigma }_{xx,0}^{y}(K^{\prime} )\\{\sigma }_{xx,z}^{y}(K)=-{\sigma }_{xx,z}^{y}(K^{\prime} ),\end{array}$$
(6)

where \({\sigma }_{xx,0}^{x}\), \({\sigma }_{xx,z}^{x}\), \({\sigma }_{xx,0}^{y}\), \({\sigma }_{xx,z}^{y}\) are defined in Eq. (2). It follows that if the two valleys remain degenerate, \({\sigma }_{xx}^{x}={\sigma }_{xx,0}^{x}+{\sigma }_{xx,z}^{x}{M}_{z}\) must vanish because the two valleys have opposite \({M}_{z}\). If the valley symmetry is broken, the net orbital magnetization \(\delta {M}_{z}\) would be non-vanishing, leading to nonzero \({\sigma }_{xx}^{x} \sim {\sigma }_{xx,z}^{x}\delta {M}_{z}\). Thus the \({\sigma }_{xx}^{x}\) component can be used as a probe to detect the valley symmetry breaking and the associated orbital magnetization in \(AB\)-\(BA\) stacked TDBG. Once the orbital \({\mathcal{T}}\) symmetry (valley symmetry) is spontaneously broken, the \({\sigma }_{xx}^{x}\) component would be linearly proportional to the orbital magnetization of the system, which is expected to exhibit hysteresis behavior in response to the out-of-plane magnetic field.

In Fig. 5a we plot the calculated shift-current photoconductivity \({\sigma }_{xx}^{x}(0)\) at +1/2 filling of \(AB\)-\(BA\) stacked TDBG. The blue circles, red diamonds, and magenta plus signs represent the situations with the valley polarization \({\xi }_{v}=0\), \(+0.1\), and \(-0.1\) respectively. With \(\pm\!10 \%\) of valley polarizations, \({\sigma }_{xx}^{x}\) can be as large as \(\pm\! 10^{3}\ \mu\)AV\({}^{-2}\) at relatively low frequencies \(\hslash \omega \lesssim 0.1\) eV. Moreover, \({\sigma }_{xx}^{x}\) are opposite for opposite valley polarizations as \({\sigma }_{xx}^{x} \sim {\sigma }_{xx,z}^{x}\delta {M}_{z}\). In Fig. 5b we show the shift-current photoconductivity \({\sigma }_{xx}^{y}\) at 1/2 filling with the valley polarizations \({\xi }_{v}=0\) (blue circles), \(+0.1\) (red diamonds), and \(-0.1\) (magenta plus signs). Contrary to the case of \({\sigma }_{xx}^{x}\), \({\sigma }_{xx}^{y}\) is not significantly changed by the valley polarization, and are identical for the opposite valley polarizations \({\xi }_{v}=\pm 0.1\), which is expected according to Eq. (6). In Fig. 5c, d we plot the imaginary part of the SHG susceptibilities \({\chi }_{xx}^{x}(2\omega )\) and \({\chi }_{xx}^{y}(2\omega )\) at 1/2 filling of \(AB\)-\(BA\) stacked TDBG. We see that, in the low-frequency regime \(\hslash \omega \lesssim 0.1\) eV, \({\rm{Im}}{\chi }_{xx}^{x}(2\omega )\) is as large as \(\sim \pm\! 10^{6}\) pm/V when the valley polarization \({\xi }_{v}=\pm 0.1\), and vanishes when \({\xi }_{v}=0\). On the other hand, \({\rm{Im}}{\chi }_{xx}^{y}(2\omega )\) in Fig. 5d seems to be not sensitive to the valley polarizations.

Fig. 5: Nonlinear optical properties of \(AB\)-\(BA\) stacked twisted double bilayer graphene.
figure 5

a, b The shift-current photoconductivities of \(AB\)-\(BA\) stacked TDBG at 1/2 filling: a \({\sigma }_{xx}^{x}(0)\), and b \({\sigma }_{xx}^{y}(0)\). c, d The imaginary parts of the SHG susceptibilities of \(AB\)-\(BA\) stacked TDBG at 1/2 filling: c Im\({\chi }_{xx}^{x}(2\omega )\), and d Im\({\chi }_{xx}^{y}(2\omega )\). The blue circles, red diamonds, and magenta plus signs denote the cases of 0%, +10% and -10% valley polarizations respectively.

Full size image

In \(AB\)-\(AB\) stacked TDBG, there are \({C}_{3z}\) and \({C}_{2x}\) symmetries for each valley. As a result, the only non-vanishing photoconductivity components for each valley are \({\sigma }_{xx,0}^{x}\) and \({\sigma }_{xx,z}^{y}\). However, the \({C}_{2x}\) symmetry further enforces that \({M}_{z} = 0\) for each valley, which implies \({\sigma }_{xx}^{y}={\sigma }_{xx,z}^{y}\ {M}_{z}\) must vanish.

The \({C}_{2x}\) symmetry can be broken by vertical electric fields, which would allow for valley-contrasting orbital magnetizations. If the valley symmetry is further broken either spontaneously by Coulomb interactions or by external magnetic fields, then the orbital magnetization would contribute to \({\sigma }_{xx}^{y}\) in such a way that \({\sigma }_{xx}^{y}\) would exhibit hysteresis behavior under out-of-plane magnetic fields. To be specific, in the presence of vertical electric fields, for the valley \({K}^{\mu }\), \({\sigma }_{xx}^{y}({K}^{\mu })\) is expressed as

$${\sigma }_{xx}^{y}({K}^{\mu })={\sigma }_{xx,0}^{y}({K}^{\mu })+{\sigma }_{xx,z}^{y}({K}^{\mu })\ {M}_{z}({K}^{\mu }),$$
(7)

Note that the coefficients \({\sigma }_{xx,0}^{y}({K}^{\mu })\) and \({\sigma }_{xx,z}^{y}({K}^{\mu })\) do not change signs under time-reversal operation, and they are dependent on the valley indices only if the bandstructures and/or chemical potentials of the two valleys are different. Then we consider the situation that the system has nonzero valley splittings \(\pm {E}_{v}\) (see Eq. (5)). When the valley splitting is \(+{E}_{v}\), the orbital magnetization of the \(K\) and \(K^{\prime} \) valleys are denoted as \({M}_{z}^{-}\) and \({M}_{z}^{+}\) respectively; if the valley splitting is reversed to \(-{E}_{v}\), then the orbital magnetizations of the \(K\) and \(K^{\prime} \) valleys would become \(-{M}_{z}^{+}\) and \(-{M}_{z}^{-}\). Plugging these relationships into Eq. (7), one obtains

$$\begin{array}{l}{\sigma }_{xx}^{y}(+{E}_{v})=\mathop{\sum}\limits _{\mu =\pm 1}{\sigma }_{xx,0}^{y}(\mu {E}_{v})+\mathop{\sum}\limits _{\mu =\pm 1}{\sigma }_{xx,z}^{y}(\mu {E}_{v})\ {M}_{z}^{\mu }\\{\sigma }_{xx}^{y}(-{E}_{v})=\mathop{\sum}\limits _{\mu =\pm 1}{\sigma }_{xx,0}^{y}(-\mu {E}_{v})-\mathop{\sum}\limits _{\mu =\pm 1}{\sigma }_{xx,z}^{y}(-\mu {E}_{v})\ {M}_{z}^{-\mu },\end{array}$$
(8)

where \({\sigma }_{xx}^{y}(\!\pm {E}_{v})\) stands for the total \({\sigma }_{xx}^{y}\) of with \(\pm {E}_{v}\) valley splittings, summing over the contributions from the two valleys. Subtracting \({\sigma }_{xx}^{y}(+Ev)\) by \({\sigma }_{xx}^{y}(-{E}_{v})\) would eliminate the \({\sigma }_{xx,0}^{y}\) term, leaving the term that is proportional to the total orbital magnetization of the system, i.e.,

$$\begin{array}{ll}&\left(\ {\sigma }_{xx}^{y}(+{E}_{v})-{\sigma }_{xx}^{y}(-{E}_{v})\ \right)/2\\ =&{\sigma }_{xx}^{y}(-{E}_{v}){M}_{z}^{-}+{\sigma }_{xx}^{y}(+{E}_{v}){M}_{z}^{+}\\ \approx &{\sigma }_{xx}^{y}(0)\ ({M}_{z}^{-}+{M}_{z}^{+}),\end{array}$$
(9)

where \({M}_{z}^{+}+{M}_{z}^{-}\) is the total orbital magnetization of the system with \(+{E}_{v}\) valley splitting. In the last line of Eq. (9) we have assumed \({\sigma }_{xx}^{y}(-{E}_{v})\approx {\sigma }_{xx}^{y}(0)\approx {\sigma }_{xx}^{y}({E}_{v})\) for small valley splittings. Similar argument also applies to \({\sigma }_{xx}^{x}\). When \({C}_{2x}\) symmetry is broken by vertical electric fields, the \({\sigma }_{xx,z}^{x}\) parameter would be non-vanishing for each valley, such that the orbital magnetization would also contribute to \({\sigma }_{xx}^{x}\). One can also extract the term proportional to \({M}_{z}\) by subtracting \({\sigma }_{xx}^{x}(-{E}_{v})\) from \({\sigma }_{xx}^{x}(+{E}_{v})\).

In Fig. 6 we show the shift-current response of \(AB\)-\(AB\) stacked TDBG with vertical electrostatic potential drop \({U}_{d}=0.045\ \)eV across the four layers. In Fig. 6a we present the shift-current photoconductivity \({\sigma }_{xx}^{y}(0)\) at 3/4 filling of the isolated conduction flat band at \(\theta =1.0{5}^{\circ }\). The blue, red and magenta circles represent the cases with valley splitting \({E}_{v}=0\), \(+1\) meV and \(-1\ \)meV respectively. The black diamonds denote \({\sigma }_{xx}^{y}(+{E}_{v})/2-{\sigma }_{xx}^{y}(-{E}_{v})/2\) with \({E}_{v}=1\ \)meV, which extracts the orbital-magnetization contribution to \({\sigma }_{xx}^{y}\) as explained in Eqs. (7)–(9). We see that \({\sigma }_{xx}^{y}\) is actually dominated by the \({\sigma }_{xx,z}^{y}{M}_{z}\) term, which is expected to show remarkable hysteresis loops under out-of-plane magnetic fields. In Fig. 6b we plot the dependence of \({\sigma }_{xx}^{x}(0)\) on the light frequency \(\omega \) at the same filling and the same twist angle. Clearly \({\sigma }_{xx}^{x}(0)\) is not changed too much by the valley splitting \({E}_{v}\), indicating that the orbital-magnetization contribution (\({\sigma }_{xx,z}^{x}{M}_{z}\)) plays a minor role in \({\sigma }_{xx}^{x}(0)\).

Fig. 6: Nonlinear optical properties of \(AB\)-\(AB\) stacked twisted double bilayer graphene.
figure 6

a The shift-current photoconductivity \({\sigma }_{xx}^{y}\), and (b) \({\sigma }_{xx}^{x}\), for 3/4 filled \(AB\)-\(AB\) stacked TDBG with vertical electric fields at the twist angle \(\theta \ =\ 1.0{5}^{\circ }\). The blue, red, and magenta circles represent the situations with the valley splitting \({E}_{v}=0\), \(+1\ \)meV, and \(-1\ \)meV respectively. The black diamonds represent \({\sigma }_{xx}^{y}(+{E}_{v})/2-{\sigma }_{xx}^{y}(-{E}_{v})/2\) in (a), and \({\sigma }_{xx}^{x}(+{E}_{v})/2-{\sigma }_{xx}^{x}(-{E}_{v})/2\) in b, with \({E}_{v} = 1\) meV.

Full size image

Discussion

We have systematically studied the anomalous Hall effect, magneto-optical properties, and nonlinear optical properties of hBN-aligned TBG and TDBG. We have studied the dependence of AHC on the valley and spin splittings in hBN-aligned TBG, and found that the AHE can be engineered using in-plane magnetic fields. In additional to AHE, we also show that there exists giant magneto-optical effect by virtue of the valley-symmetry breaking and orbital magnetizations. We propose that the Faraday and Kerr rotations may be a powerful tool to detect the presence of orbital magnetism in twisted graphene systems. Moreover, we have also studied the nonlinear optical responses, i.e., the shift current and second harmonic generation in both hBN-aligned TBG and TDBG. Our calculations indicate that both systems exhibit colossal nonlinear optical responses. To be specific, the shift-current photoconductivity \({\sigma }_{ab}^{c}(0)\) is on the order of \(1{0}^{3}\ \mu \)A/V\({}^{2}\), and the SHG susceptibility \({\chi }_{ab}^{c}(2\omega )\) is on the order of \(1{0}^{6}\) pm/V in the terahertz frequency regime. Such gigantic nonlinear optic responses are by virtue of the inversion symmetry breaking, the presence of the low-energy flat bands, and the small excitation gaps in the twisted graphene systems. In TDBG with \(AB\)-\(BA\) stacking, we propose that the non-vanishing valley polarization and orbital magnetization (\({M}_{z}\)) are associated with \({C}_{2y}\) crystalline symmetry breaking, which would generate a new component of the nonlinear photoconductivity \({\sigma }_{xx}^{x} \sim {M}_{z}\); while in \(AB\)-\(AB\) stacked TDBG with vertical electric fields, the valley polarization and orbital magnetization would make significant contributions to \({\sigma }_{xx}^{y}\). These new components of photoconductivities generated by the orbital magnetizations would exhibit notable hysteresis behavior in response to out-of-plane magnetic fields, and may be considered as strong and robust experimental evidence for the valley polarized state and orbital magnetism in the TDBG system. Our work is a significant step forward in understanding the optical properties of the twisted graphene systems, and may provide useful guidelines for future experimental works.

Methods

Continuum Hamiltonians for twisted bilayer and twisted double bilayer graphene systems

As discussed above, in hBN-aligned TBG we have neglected the effects of the moiré potential generated by the hBN substrate. With such an approximation, the continuum model of TBG aligned with hBN substrate can be simplified as

$${H}_{0}^{\mu }={H}_{TBG}^{\mu }+{H}_{mass}.$$
(10)

\({H}_{mass}\) is the “Dirac mass” term at the bottom layer graphene generated by the hBN substrate, which is expressed as

$${H_{mass}} = \left( {\begin{array}{*{20}{l}}{\Delta {\sigma _z}}&0\\0&0\end{array}} \right),$$
(11)

where \(\Delta \) is the staggered sublattice potential exerted on the bottom graphene layer, which is fixed as \(17\ \)meV throughout this paper. \({H}_{TBG}^{\mu }\) is the continuum model of TBG for the \(\mu \)th valley, and \({V}_{hBN}^{\mu }\) is the moiré potential generated by the hBN substrate for the \(\mu \)th valley, with \(\mu =-1\) corresponding to the \(K\) valley, and \(\mu=+1\) corresponding to the \(K^{\prime} \) valley, i.e., \({\bf{K}}\equiv {{\bf{K}}}^{-}\), and \({\bf{K}}^{\prime} \equiv {{\bf{K}}}^{+}\). More specifically,

$${H}_{TBG}^{\mu }({\bf{k}})=\left(\begin{array}{*{20}{l}}-\hslash {v}_{F}({\bf{k}}-{{\bf{K}}}_{1})\cdot {{\boldsymbol{\sigma }}}^{\mu }&{U}_{\mu }^{\dagger }({\bf{r}})\\ {U}_{\mu }&-\hslash {v}_{F}({\bf{k}}-{{\bf{K}}}_{2})\cdot {{\boldsymbol{\sigma }}}^{\mu }\end{array}\right)$$
(12)

where \({{\boldsymbol{\sigma }}}^{\mu }=[\mu {\sigma }_{x},{\sigma }_{y}]\) represent the Pauli matrices, and the interlayer coupling \({U}_{\mu }\) is expressed as

$$\begin{array}{*{20}{l}}{U}_{\mu }=\left(\begin{array}{*{20}{l}}{u}_{0}&u^{\prime} \ \\ u^{\prime} &{u}_{0}\end{array}\right)+\left(\begin{array}{*{20}{l}}{u}_{0}&u^{\prime} {e}^{-i\mu \frac{2\pi }{3}}\ \\ u^{\prime} {e}^{i\mu \frac{2\pi }{3}}&{u}_{0}\end{array}\right){e}^{i\mu {{\bf{G}}}_{1}\cdot {\bf{r}}}\\+\, \left(\begin{array}{*{20}{l}}{u}_{0}&u^{\prime} {e}^{i\mu \frac{2\pi }{3}}\ \\ u^{\prime} {e}^{-i\mu \frac{2\pi }{3}}&{u}_{0}\end{array}\right){e}^{i\mu ({{\bf{G}}}_{1}+{{\bf{G}}}_{2})\cdot {\bf{r}}}\end{array}$$
(13)

with \({{\bf{K}}}_{1}={\bf{K}}^{\prime} =( 2\pi /\sqrt{3}{L}_{s},-2\pi /3{L}_{s} )\), and \({{\bf{K}}}_{2}={{\bf{K}}}_{1}=( 2\pi /\sqrt{3}{L}_{s},2\pi /3{L}_{s})\), and the moiré lattice constant \({L}_{s} = a/[2\sin (\theta /2)]\), \(a=2.46\ \)ang. is the graphene lattice constant. The Fermi velocity \(\hslash {v}_{F}=5.253\ {\rm{eV}}\)ang, and the interlayer coupling parameters \({u}_{0}=0.0797\ \)eV, \(u^{\prime} =0.0975\ \)eV. \({{\bf{G}}}_{1}\) and \({{\bf{G}}}_{2}\) are the reciprocal lattice vectors of the TBG moiré supercell, which are chosen as \({{\bf{G}}}_{1}=(-2\pi /\sqrt{3}{L}_{s},-2\pi /{L}_{s})\), and \({{\bf{G}}}_{2}=(4\pi /\sqrt{3}{L}_{s},0)\).

The effective Hamiltonian for TDBG can be generalized from Eq. (12). There are two different kinds of staking configurations in TDBG: (a) both the bottom and the top bilayers are \(AB\) (or \(BA\)) stacked; (b) the bottom bilayer is \(AB\) stacked, but the top bilayer is \(BA\) stacked, or vice versa. We introduce \(\lambda ,\lambda ^{\prime} =\pm \) to denote the stacking chiralities of the bottom and top bilayers, with \(\lambda ,\lambda ^{\prime} =+\) (\(-\)) for \(AB\) (\(BA\)) stacked bilayers respectively.

$${H}_{\lambda ,\lambda ^{\prime} }^{\mu }=\left(\begin{array}{*{20}{l}}{H}_{\lambda }^{\mu }({K}_{1})&{{\mathbb{U}}}_{\mu }^{\dagger }({\bf{r}})\\ {{\mathbb{U}}}_{\mu }({\bf{r}})&{H}_{\lambda ^{\prime} }^{\mu }({K}_{2})\end{array}\right)$$
(14)

where \(\mu =\pm \) is the valley index, and \({H}_{\lambda }^{\mu }\) is the Hamiltonian for \(AB\) or \(BA\) stacked bilayer graphene

$${H}_{\lambda }^{\mu }(K)=\left(\begin{array}{*{20}{l}}-\hslash {v}_{F}({\bf{k}}-{{\bf{K}}}^{\mu })\cdot {{\boldsymbol{\sigma }}}^{\mu }&{h}_{\lambda }\\ {h}_{\lambda }^{\dagger }&-\hslash {v}_{F}({\bf{k}}-{{\bf{K}}}^{\mu })\cdot {{\boldsymbol{\sigma }}}^{\mu }\end{array}\right)$$
(15)

where \({h}_{\lambda }\) is the interlayer hopping for Bernal stacked bilayer graphene. In particular,

$${h}_{+}=\left(\begin{array}{*{20}{l}}{t}_{2}f({\bf{k}})&{t}_{2}{f}^{* }({\bf{k}})\\ {t}_{\perp }-3{t}_{3}&{t}_{2}f({\bf{k}})\end{array}\right),$$
(16)

where \({t}_{2}\,=\,0.21\) eV, \({t}_{3}\approx 0.05\ \)eV, and \({t}_{\perp }\,=\,0.48\ \)eV. The phase factor \(f({\bf{k}})=\left({e}^{-i\sqrt{3}a{k}_{y}/3}+{e}^{i\left({k}_{x}a/2+\sqrt{3}a{k}_{y}/6\right.}\right)+\left.{e}^{i\left(-{k}_{x}a/2+\sqrt{3}a{k}_{y}/6\right.}\right)\). The interlayer hopping with \(-\) stacking chirality \({h}_{-}\,=\,{h}_{+}^{\dagger }\).

Kerr and Faraday angles of quasi-2D systems

We consider the limit that the thickness of the twisted graphene systems \({d}_{z}\) is much smaller than the optical wavelength \(\lambda \), such that they can be treated as ideal 2D systems with zero thickness. We consider the case of normal incident light with linear polarization, and calculate the reflected and transmitted light following the idea of ref. 79. The incident electric field is denoted as \({{\bf{E}}}_{0}{e}^{i(qz-\omega t)}\), and the transmitted and reflected electric fields are denoted as \({{\bf{E}}}_{t}{e}^{i(qz-\omega t)}\) and \({{\bf{E}}}_{r}{e}^{i(-qz-\omega t)}\) respectively, where \(q=2\pi /\lambda \) is the wavevector of the light. The twist graphene layer can be considered as the interface of two semi-infinite vacuum, and the boundary conditions of the electromagnetic fields at the interface \(z=0\) are expressed as

$$\begin{array}{l}\hat{{\bf{z}}}\,\times ({{\bf{E}}}_{t}-{{\bf{E}}}_{0}-{{\bf{E}}}_{r})=0\\\hat{{\bf{z}}}\,\times ({{\bf{H}}}_{t}-{{\bf{H}}}_{0}-{{\bf{H}}}_{r})={{\bf{j}}}_{s}\ ,\end{array}$$
(17)

where \({{\bf{H}}}_{0}\), \({{\bf{H}}}_{t}\), \({{\bf{H}}}_{r}\) represent the incident, transmitted and reflected magnetic fields. it follows from the Maxwell equation, \(\nabla \times {\bf{E}}=-{\mu }_{0}\partial {\bf{H}}/\partial t\), that

$$\begin{array}{l}{{\bf{H}}}_{0}=\sqrt{\displaystyle\frac{{\epsilon }_{0}}{{\mu }_{0}}}{[-{E}_{0,y},{E}_{0,x}]}^{T},\\{{\bf{H}}}_{t}=\sqrt{\displaystyle\frac{{\epsilon }_{0}}{{\mu }_{0}}}{[-{E}_{t,y},{E}_{t,x}]}^{T},\\{{\bf{H}}}_{r}=\sqrt{\displaystyle\frac{{\epsilon }_{0}}{{\mu }_{0}}}{[{E}_{r,y},-{E}_{r,x}]}^{T}.\\ \end{array}$$
(18)

\({{\bf{j}}}_{s}\) denotes the current density at the interface with

$${j}_{s,a}=\sum _{b=x,y}{\sigma }_{ab}(\omega ){E}_{t,b},$$
(19)

where \({\sigma }_{ab}(\omega )\) is the optical conductivity tensor of the twist graphene layer, given by

$$\begin{array}{ll}\displaystyle{\sigma }_{ab}(\omega )=i\frac{{e}^{2}}{\hslash }\int\frac{d{\bf{k}}}{{(2\pi )}^{3}}{\sum \limits_{m\ne n}}{\hslash }^{2}\frac{{v}_{a,mn}({\bf{k}}){v}_{b,nm}({\bf{k}})}{{\epsilon }_{n{\bf{k}}}-{\epsilon }_{m,{\bf{k}}}-\nu -i\delta }\,\times\, \frac{f({\epsilon }_{n{\bf{k}}})-f({\epsilon }_{m{\bf{k}}})}{{\epsilon }_{n{\bf{k}}}-{\epsilon }_{m,{\bf{k}}}}.\end{array}$$
(20)

Combining Eqs. (17)–(19), assuming the incident light polarization is along the \(x\) direction, one obtains

$$\begin{array}{l}\displaystyle\frac{{E}_{r,x}}{{E}_{0}}=\frac{{\sigma }_{xy}{\sigma }_{yx}\,-\,{\sigma }_{xx}(2\sqrt{{\epsilon }_{0}/{\mu }_{0}}\,+\,{\sigma }_{yy})}{-\,{\sigma }_{xy}{\sigma }_{yx}\,+\,(2\sqrt{{\epsilon }_{0}/{\mu }_{0}}\,+\,{\sigma }_{xx})(2\sqrt{{\epsilon }_{0}/{\mu }_{0}}\,+\,{\sigma }_{yy})},\\ \displaystyle\frac{{E}_{r,y}}{{E}_{0}}=\frac{2\sqrt{{\epsilon }_{0}/{\mu }_{0}}{\sigma }_{yx}}{-\,{\sigma }_{xy}{\sigma }_{yx}\,+\,(2\sqrt{{\epsilon }_{0}/{\mu }_{0}}\,+\,{\sigma }_{xx})(2\sqrt{{\epsilon }_{0}/{\mu }_{0}}\,+\,{\sigma }_{yy})},\\ \displaystyle\frac{{E}_{t,x}}{{E}_{0}}=\frac{2\sqrt{{\epsilon }_{0}/{\mu }_{0}}(2\sqrt{{\epsilon }_{0}/{\mu }_{0}}\,+\,{\sigma }_{yy})}{-\,{\sigma }_{xy}{\sigma }_{yx}\,+\,(2\sqrt{{\epsilon }_{0}/{\mu }_{0}}\,+\,{\sigma }_{xx})(2\sqrt{{\epsilon }_{0}/{\mu }_{0}}\,+\,{\sigma }_{yy})},\\ \displaystyle\frac{{E}_{t,y}}{{E}_{0}}=\frac{2\sqrt{{\epsilon }_{0}/{\mu }_{0}}{\sigma }_{yx}}{-\,{\sigma }_{xy}{\sigma }_{yx}\,+\,(2\sqrt{{\epsilon }_{0}/{\mu }_{0}}\,+\,{\sigma }_{xx})(2\sqrt{{\epsilon }_{0}/{\mu }_{0}}\,+\,{\sigma }_{yy})}.\\ \end{array}$$
(21)

The Faraday and Kerr rotations are given by80

$$\begin{array}{l}\displaystyle{\theta }_{F}={\rm{Re}}\left[\arctan \left(\frac{{E}_{t,y}}{{E}_{t,x}}\right)\right]\\ \displaystyle{\theta }_{K}={\rm{Re}}\left[\arctan \left(\frac{{E}_{r,y}}{{E}_{r,x}}\right)\right]\end{array}$$
(22)

where \({\rm{Re}}[z]\) represents the real part of a complex number \(z\), and \(\arctan (z)=(i/2)({\log}\,(i+z)-{\log}\,(i-z))\) is the inverse tangent function of a complex number \(z\). Combining Eqs. (21) and (22), and assuming \(2\sqrt{{\epsilon }_{0}/{\mu }_{0}}\gg | {\sigma }_{ab}| \) for \(a,b=x,y\) (which is an excellent approximation given that \(2\sqrt{{\epsilon }_{0}/{\mu }_{0}}=(1/\alpha ){e}^{2}/h\approx 137{e}^{2}/h\)), \({\theta }_{F}\) and \({\theta }_{K}\) are simplified as

$$\begin{array}{l}{\theta }_{F}\approx {\rm{Re}}\left[\arctan \left(\frac{{\sigma }_{yx}}{2\sqrt{{\epsilon }_{0}/{\mu }_{0}}}\right)\right]\\{\theta }_{K}\approx {\rm{Re}}\left[\arctan \left(\frac{-{\sigma }_{yx}}{{\sigma }_{xx}}\right)\right].\end{array}$$
(23)

It follows that if the real part \({\sigma }_{yx}\) (and/or \({\sigma }_{xy}\)) changes sign, both \({\theta }_{F}\) and \({\theta }_{K}\) are expected to change sign as well. This is why \({\theta }_{F}\) and \({\theta }_{K}\) change their signs as the light frequency increases, as shown in Fig. 3c, d.