1 Introduction

In analogy to the unconventional superconductors of cuprates and iron pnictides, interface-enhanced high-transition temperature (high-TC) superconductivity in single-layer FeSe films on SrTiO3 has boomed an upsurge in a great deal of attractable interest in the condensed-matter community [1,2,3,4,5], including the unusual interfacial effect, the underlying pairing mechanism, the magnetic ground states and other competing orders. Till now, some degree of consensus has been reached on the origin of TC enhancement, ascribing to a combination of two major contributing factors—the interface-induced channel (tensile strain [6,7,8,9] and electron-phonon coupling [3, 10,11,12]) and electron doping effect [13,14,15,16]. Meanwhile, there emerge extensive explorations concerning the electron phase diagram [16,17,18], electron pairing symmetries [19,20,21,22], electron-phonon coupling [23,24,25], pre-formed pairing of Cooper pairs and pseudogap behavior [26,27,28], etc.

Among these debates, searching for other FeSe-derived compounds, associated with their relationship to single-layer FeSe/SrTiO3 films, may be a plausible route to understanding the mystery of unconventional mechanisms and interfacial enhancement in superconductivity. For instance, both transport and ARPES experiments argued that the Fe-vacancy ordered Fe4Se5 [29, 30] (or undoped FeSe/SrTiO3 films [31]) should be the insulating parent phase of FeSe (or FeSe/SrTiO3). In particular, the Fe4Se5 phase is manifested as a Mott insulator possessing a block-checkerboard antiferromagnetic (AFM) order, resembling the A2Fe4Se5 compounds (A = K, Tl, Rb, or Cs) [32, 33]. Such similarity between A1–xFe2–ySe2 and FeSe/SrTiO3 films is also consistent with the dramatic TC enhancement of FeSe superconductor by heavy electron doping [15, 34, 35]. Also, there report a large amount of FeSe-based superconductors by intercalation of organic molecules or inorganic elements, such as Lix(NH2)y(NH3)1yFe2Se2 [36], (Li0.8Fe0.2)OHFeSe [37], (CTA)0.3FeSe [38]. Different from the wet-chemical synthesis of liquid-ammonia and electrochemical methods, molecular beam epitaxy (MBE) is a liquid-free and controllable growth to gain different constituents of FeSe with high quality of atomically flat thin films, as well as investigating their complex phase transitions. Previously, we have also actualized the MBE-controlled synthesis on the multiple phases of the sister Fe–Te compound with three structural phases: α-FeTe, β-FeTe, and FeTe2 of different intersections [39]. Other FeSex phases are sporadically reported [40, 41]; however, there is still a lack of a concordant growth scenario for the FeSe-derived thin films.

In this study, we report a compositive route to synthesize distinct FexSey phases based on the single-layer FeSe/SrTiO3 films via MBE growth. The key essentials lie in the elaborate control of supplying excess Se flux and low-temperature annealing treatment. Various stoichiometries of FeSe-derived compounds are synthesized with atomically smooth surfaces and largely flat terraces, including α-FeSe, β-FeSe, Fe3Se4, Fe4Se5, Fe5Se6, Fe6Se7, and Fe5Se8. While Fe4Se5, Fe5Se6, and Fe6Se7 are distinguished by different densities of ordered Fe vacancies, Fe3Se4 and Fe5Se8 are constructed by selective intercalations of Fe atoms. Scanning tunneling microscopy/spectroscopy (STM/STS) measurements are used to reveal the structural characteristics, electronic properties, and phase transitions among these FexSey phases. Specifically, Fe4Se5 exhibits an in-plane AFM order with unidirectional stripes as real-space spin contrasts, which is sustained by the magnetic response of AFM domains. We also discuss the electronic doping effect in insulating Fe4Se5 by surface K dosing, which seems inadequate to drive Fe4Se5 into the superconducting regime. Our MBE implication of fabricating different stoichiometries of FexSey phases sheds light on constructing and transforming FeSe-related families, ranging from 3D to 2D systems, which is expected to be an appealing platform to create and tune potentially exotic states in the extensive library of quantum materials.

2 Experiments and methods

All FexSey thin samples were prepared by the MBE method in a customized ultrahigh vacuum chamber at a base pressure of 5 × 10−10 mbar [42]. Before growth, the atomically flat Nb-doped SrTiO3(001) substrate (0.7 wt %) was obtained by (1) degassing at 873 K for 2 h, (2) thermally heated to ∼ 1223 K with a Se flux for half an hour, (3) flashed to 1320 K for an hour. High-purity Fe (powder, 99.99%; Alfa Aesar) and Se sources (shot, 99.9999%; Alfa Aesar) were used for co-evaporating from standard K cells. During the growth process, the Fe flux was fixed at 1370 K, giving a growth rate of ∼ 0.2 monolayer/min. The Se/Fe flux ratio was adjusted by precisely varying the temperature of the Se source. The substrate temperature was controlled by direct current heating, which was monitored by an infrared pyrometer.

As a starting point, epitaxial β-FeSe/SrTiO3 films were first fabricated, with the substrate holding at 673 K [2]. It can become superconducting by annealing to 773 K for 2 h [14]. The hexagonal α-FeSe phase was obtained by annealing β-FeSe at 473 K for 1 h. Variant Fe-vacancy phases were achieved by gently annealing the as-grown sample at 373–423 K with a Se overflux for 30 min to 2 h. The K deposition was performed in situ at a temperature of 20–50 K by using a potassium dispenser (SAES Getters). The coverage of K adatoms is accumulated by repeated depositions onto the FexSey/SrTiO3 surface, with 10–30 s in each cycle [43].

The low-temperature STM/STS measurements were conducted on a commercial STM system (Unisoku-1500) at a base temperature of 2 K [44]. W tip was electrochemically etched and then cleaned by e-beam at ~ 1900 K to remove its apex oxides, which were calibrated on a standard Ag(111) sample prior to STM measurements. All topographic images were taken in a constant-current mode, and the tunneling spectra and conductance mappings were acquired by a standard lock-in technique with a modulation voltage (Vmod) at 983 Hz [45]. The spin-polarized STM (SP-STM) measurements were carried out by using a spin-polarized Fe-coated W tip, which was fabricated by depositing ~ 20 nm Fe clusters onto the apex of the calibrated W tip, followed by annealing at ~ 500 K for 10 min. The spin-polarized properties of the Fe-coated tip were verified by mapping the zigzag patterns on an AFM CrTe2 surface [46]. An external magnetic field was applied perpendicular to the surface of FeSe/SrTiO3 films.

3 Results and discussion

The formation of different FexSey phases is achieved by the precise combination of (i) controlling the substrate temperature, (ii) adjusting the Se/Fe ratio, and (iii) post-annealing under Se flux, as illustrated in Fig. 1a. Firstly, we start from the easily obtained β-FeSe films by the mature process as reported [2, 14]. On the one hand, it can convert to α-FeSe thin films by annealing at a low temperature of ~ 473 K, with the typical STM morphology shown in Fig. 1b. Obviously, the islands are triangle-shaped with large terraces of atomically smooth Se-terminated surface, indicating a phase transition from tetragonal β-FeSe (layered) into hexagonal α-FeSe (nonlayered). Their high-resolution STM images associated with their crystalline structural models are shown in Fig. 2a, b. On the other hand, by gradually increasing the Se flux, we can obtain various ordered phases from β-FeSe. As reported in our previous study [47], the distinct superstructures of √5 × √10 (1/7 Fe vacancies or Fe6Se7), 2 × √10 (1/6 Fe vacancies or Fe5Se6), and √5 × √5 (1/5 Fe vacancies or Fe4Se5) mainly depend on the amount of Fe vacancies. Figs. 1d and 2c show the STM image of the √5 × √5 phase, with the orientation of the unit cell rotated 26.6° relative to the topmost Se lattice. It is noteworthy that all the ordered phases of Fe vacancies can only be observed with a thickness ≥ 2 layers, but is always absent in the monolayer case, suggesting that the site of Fe vacancies should be occupied in the van der Waals gap of FeSe–FeSe interlayer instead of the FeSe-SrTiO3 interface. Also, these superstructures with varied Fe vacancies show no difference among different thicknesses, they should be bulk structures instead of only surface reconstructions.

Fig. 1
figure 1

a Schematic illustration of the different growth conditions for different FexSey phases by MBE. b Large-scale STM morphology for the α-FeSe surface (scanning size: 120 × 120 nm2, bias voltage Vbias = + 2.0 V, tunneling current It = 20 pA). c-f Atomically resolved STM images for the α-FeSe (c, 10 × 10 nm2, Vbias = + 0.05 V, It = 100 pA), Fe4Se5 (d, 20 × 20 nm2, Vbias = + 1.0 V, It = 100 pA), Fe3Se4 (e, 25 × 25 nm2, Vbias = + 0.5 V, It = 100 pA), and Fe5Se8 (f, 20 × 20 nm2, Vbias = + 1.5 V, It = 100 pA) surfaces, respectively

Fig. 2
figure 2

Atomically resolved STM images for different FexSey phases grown on STO substrates under different growth conditions, respectively, for α-FeSe (a), β-FeSe (b), Fe4Se5 (c), Fe3Se4 (d), Fe5Se8 (e) surfaces, associated with their side views of crystalline structures. Their top views of toy models are also superposed onto the STM image. The red shapes stand for the unit cells of √5 × √5, √3 × 1, and 2 × 2 for Fe4Se5, Fe3Se4, and Fe5Se8, respectively. All are acquired by 2 × 2 nm2, Vbias = + 0.1 V, It = 100 pA

Upon increasing the Se overflux at low temperature, more Fe vacancies are formed and the √5 × √5 phase further transforms into Fe3Se4 and Fe5Se8, respectively. Strategically speaking, additional interstitial Fe atoms are expected to effectively intercalate into the β-FeSe layers by such a low-temperature Se annealing process [48], and we can regard the Fe3Se4 and Fe5Se8 phase as the different levels of self-intercalation of Fe atoms within the interlayer space of the 1T-FeSe2 compound. For Fe3Se4, while the Fe atoms along the b axis are intact, Fe vacancies are only formed along the a-axis for every two Fe rows. This leads to a √3×1 reconstruction of the intercalated Fe atoms and can be reflected by the striped patterns of the topmost Se surface, as seen in Fig. 2d. For the Fe5Se8 phase, the intercalated Fe atoms are alternatively arranged along both the a and b axes of the Fe layer, resulting in a 2 × 2 pattern of kagome-like morphology (Fig. 2e). Such a kagome lattice is comprised of ordered hexagons that are interconnected by interlaced triangles, combining into arrays of corner-sharing triangles. In Table 1, we compare the crystalline properties of different FexSey phases, including stoichiometry, Fe/Se atomic ratio, STM morphology, and in-plane lattice constant.

Table 1 Comparison of the stoichiometry, Fe/Se atomic ratio, STM morphology, and in-plane lattice constant of different FexSey phases

Overall, the growth phase strongly depends on the detailed annealing treatments with Se flux at low temperatures. Low substrate temperature mainly drives the triangular structure of α-FeSe, and the supplied Se flux provides the formation of Fe-vacancies (or intercalated Fe). These transitions are 100% with pure α-FeSe and Fe4Se5 phases. With more Fe-vacancies, the compounds Fe3Se4 and Fe5Se8 are formed, where the intercalation concentration strongly relies on the annealing time. Statistically, a nearly pure Fe3Se4 phase (~ 100%) can be obtained with different striped domains, as displayed in Fig. 1e. Unfortunately, with further annealing, the sample is maximumly covered by 70% Fe5Se8 phase, whereas the other 30% regions are Fe3Se4. This means the efficiency of intercalating Fe atoms is limited at this stage. In the future, we hope to construct the expected FeSe2 phase with a layered structure [49] via such an intercalation strategy.

Having identified the detailed crystalline structure and STM morphologies of different FexSey phases, we then systematically study their electronic structure by performing low-temperature STS measurements, where the dI/dV spectrum is proportional to its local density of states. Figure 3a summarizes the tunneling conductance for all the observed FexSey phases. While β-FeSe is metallic (superconductive), α-FeSe becomes semiconducting with a band gap of ~ 0.2 eV. With increasing the concentration of Fe vacancies, it shows an insulating behavior with a typical gap size of 0.5–1.0 eV. When coming into the intercalation regime, it turns to be metallic again, with an overall “V” shape of STS spectra. The conductance minimum located around the Fermi level may be possibly related to the preformed kagome band structure of Dirac cones [48]. Such distinct electronic properties can also be apparently compared by taking STS line cuts across the domain boundaries among different phases. As seen in Fig. 3b–e, an insulator-to-metal transition is clearly observed when the √5 × √5 Fe4Se5 is partially converted into the kagome Fe5Se8 phase. However, the increase of Fe intercalation from Fe3Se4 to Fe5Se8 only scarcely affects their electronic structure, namely, the metallic behavior, the overall landscape, the intensities of conductance, and the location of conductance minimum are slightly changed. This similarity of electronic structure is qualitatively in accordance with their surface consistency, where their topmost Se atoms are connected continuously, with only difference in the intercalated Fe densities of the buried Fe layers (50% for Fe3Se4 and 25% for Fe5Se8) [48].

Fig. 3
figure 3

a STS spectra for different FexSey phases, respectively. All spectra are shifted vertically for clarity. The dashed horizontal lines with the same colors mark the zero differential conductance for each curve. b STM image of a sharp boundary between Fe4Se5 and Fe5Se8 surfaces (10 × 10 nm2, Vbias = + 0.1 V, It = 100 pA). c 2D plot for a series of tunneling spectra measured along the cyan line in b. The dashed vertical line represents the location of the phase boundary. d STM image of a sharp boundary between Fe4Se5 and Fe3Se4 surfaces (10 × 10 nm2, Vbias = + 0.1 V, It = 100 pA). e 2D plot for a series of tunneling spectra measured along the cyan line in d. The dashed vertical line represents the location of the phase boundary

For the insulating Fe4Se5, we have previously demonstrated its pair-checkerboard AFM ground state with in-plane magnetization by SP-STM measurements [47], where every √5 × √5 lattice is centered by a four-Fe block with two up and two down spins that are collinear AFM aligned, as illustrated in Fig. 4a, b. Its real-space spin contrast is featured by unidirectional stripes in the STM images, which recover as a 4-fold symmetry morphology by applying a perpendicular magnetic field larger than 1 T. Here, the tunneling current (\({I}_{t}\)) is the total sum of the spin-averaged tunneling current (\({I}_{0}\)) and spin-polarized tunneling current (\({I}_{\mathrm{SP}})\) expressed by \({I}_{t}= {I}_{0}+ {I}_{\mathrm{SP}}\), where \({I}_{\mathrm{SP}}\) is sensitive to the relative spin angle (\(\varphi\)) between the magnetization of the tip (\({P}_{\mathrm{tip}}\)) and the sample (\({P}_{\mathrm{sample}}\)), describing as \({I}_{SP}\propto {P}_{\mathrm{tip}}\bullet {P}_{\mathrm{sample}}\bullet cos\varphi\) [50]. In this way, the acquired SP-STM image strongly depends on the relative orientation of local spin for the sample and tip. In our Fe-coated W tip, the ferromagnetic (FM) tip apex is generally magnetized along the normal direction of the tip axis. Thus, it is sensitive to the in-plane component of sample magnetization (φ = 0 or π) but behaves as a nonmagnetic (NM) W tip if the spin structure orientates in the out-of-plane direction (φ = π/2) at the surface. The orientation of \({P}_{\mathrm{tip}}\) can easily follow the direction of an external magnetic field (B) of ~ 1 T, tilting \(\varphi\) between in-plane and out-of-plane of the surface, while \({P}_{\mathrm{sample}}\) remains undisturbed [50]. Therefore, we can image the surface in a constant-current mode by varying an applied magnetic field: tip- and energy-dependent measurements help to differentiate the magnetic information from topography, and the local spin polarization of the sample can be determined at the atomic scale.

Fig. 4
figure 4

a Experimental schematic of SP-STM measurements on the Fe4Se5 surface with pair-checkerboard AFM spin configuration of Fe atoms. Magnetic Fe clusters are attached at the W tip apex with in-plane spin polarization. b Crystal structure and magnetic order of the √5 × √5 phase, whose unit cell is labeled as a magenta square. The red (blue) arrows schematically represent the ordered magnetic moments of spin up (down) on the Fe sites. c The FM tip suddenly loses its magnetization into an NM tip during scanning, exhibiting 4-fold symmetric patterns without stripe modulations. d The FM tip suddenly reverses its magnetization in the opposite direction, revealing a π-phase slip of the unidirectional stripes. Yellow arrows indicate the orientation of magnetic stripes. Both are acquired by 12 × 12 nm2, Vbias = + 1.5 V, It = 100 pA, B = 0 T. e, f SP-STM images obtained by a Fe-coated W tip when ramping the magnetic field from 0 to 3 T, then backward to 0 T. The inset schematics depict the magnetization directions of the sample and tip, respectively. White dashed lines donate the domain boundaries, magenta squares stand for the √5 × √5 unit cell, and the yellow circle is a defect marker for the atomic registry. All are obtained by 10 × 10 nm2, Vbias = + 1.5 V, It = 100 pA

Occasionally, due to the instability of the Fe clusters attached to the cusp of the STM tip, the FM property of the Fe-coated tip may suddenly change its spin-polarized state. Fig. 4c, d are two examples of SP-STM images by a Fe-coated W tip experiencing condition change, that is into an NM order, or reversing its magnetization in the opposite direction. As shown in Fig. 4c, the SP-STM image loses the magnetic contrast without striped features (lower half of the image). Fig. 4d is a direction comparison of opposite tip magnetization, exhibiting a π-phase slip of the unidirectional stripes. We note that the detailed structure of stripes varies slightly between Fig. 4c, d, mainly involving the shapes of Se atoms and their connection. This may be explained by the different shapes of Fe tips that affect the spin contrast of the topmost Se atom. However, there persists the key characteristic of filed-dependent STM stripes and tip magnetization flipping under historical ramping experiences for different Fe tips. In this regard, the striped features should stem from the magnetic properties of the sample itself, but not from the specificity of Fe tips.

The AFM domains also provide an alternative view to demonstrate its pair-checkerboard AFM configuration. Fig. 4e, f illustrate the SP-STM images comprised of two AFM domains with 0° boundary. It experiences a magnetic field ramp from 0 to 3 T, then backward to 0 T. A 1/2-unit-cell shift (π-phase slip) of the unidirectional stripes is obviously visible across the domain boundaries in Fig. 4d. Keep the defect marker (yellow circle) for atomic registry in mind, the relative position of the √5 × √5 unit cell (magenta squares) exhibits a phase shift of 1/2-unit-cell along both a’ and b’ directions after the magnetic field cycle, while the π-phase slip can still be observed. This can be well understood by the relative orientations of tip and sample magnetizations, as vividly manifested by the inserted cartoons.

We also check the charge doping effect of alkali metals dosing on the AFM Fe4Se5 surface, which has been demonstrated as an impactful strategy to tune the superconductivity of FeSe films towards the heavily doped region [43]. As displayed in Fig. 5a–f, extra potassium (K) adatoms are introduced with different coverages and, thus different electronic doping levels. In Fig. 5g, even though the individual K atoms are randomly adsorbed, the dI/dV spectra show the homogenously distributed insulating gap against disordered K adatoms, indicating a uniform charge doping effect. Moreover, the band gap gradually shrinks as the K coverage increases, as seen in Fig. 5h. We summarize the doping dependence of insulating gap size on the Fe4Se5 phase in Fig. 5i. Obviously, through a wide range of electron doping to a coverage of 0.3 ML K atoms, Fe4Se5 remains insulating with a decreased band gap by heavily doping. It had been theoretically proposed that the √5-pair-checkerboard AFM state will turn into the √5-blocked-checkerboard AFM order (reminiscent of K2Fe4Se5) upon electron doping, meanwhile, both states keep insulating [51].

Fig. 5
figure 5

af STM images (20 × 20 nm2, Vbias = + 2.0 V, It = 10 pA) for different surface K adsorption at various coverages: 4% (a), 6% (b), 8% (c), 10% (d), 14% (e), 30% (f), respectively. g Series of dI/dV spectra recorded along the white line in b. h Typical dI/dV curves taken on the K-dosing Fe4Se5 films with various doping levels as indicated. All show insulating band gaps. The horizontal broken lines mark the zero conductance for each curve. i Phase diagram showing the band gap sizes of Fe4Se5 films (blue balls) under different charge doping. The error bars are statistics of 5–10 spectra at different locations. Other FeSex phases with different Fe vacancies are also displayed (diamond symbols, acquired from Ref. [47]) for comparison

Annealing the Fe4Se5 film can annihilate Fe vacancies, which are considered as hole dopants. Thus, the FexSey phases with reduced Fe vacancy concentrations are equivalent to electron doping to the Fe4Se5 films, as displayed by the decreasing band gap of other FeSex phases with lower Fe vacancies in Fig. 5i. Such ascription has been reported in [29], and is also collaborated from our experiments of K-doping to the Fe4Se5 film. Unfortunately, we failed to observe the induced superconductivity with more K doping, different from what had been done in multiple-layer FeSe/SrTiO3 films [15,16,17, 43]. Here, we speculate a plausible scenario as follows: Ideally, every isolated K adatom can donate a free electron to the Fe vacancies with positive charges buried underneath the topmost Se surface. However, when the K coverage grows, they tend to form clusters or islands. As such, it becomes difficult for the inner K atoms to dope electrons to the Fe vacancy layer. That is to say, it becomes difficult to achieve effective doping with increasing K for high coverage. This can be seen from the rapid decrease of gap size at low K coverage but gradually becomes saturated by adding more K adatoms.

Finally, we also carry out the STS mappings on the kagome Fe5Se8 phase at different energies. As shown in Fig. 6a, the tunneling conductance shows a continuous evolution with increasing the bias voltage, which also delivers analogous modulations to the topographic images at each individual energy [48], suggesting a close correlation between the morphology and local density of states. These energy-dependent dI/dV mappings are further analyzed by the 2D FFT in Fig. 6b–d. By plotting the FFT intensities along the ΓM and ΓK directions, respectively, we find that the signals show nondispersive behavior of the kagome geometry in the whole energy range, well excluding the possible origin of quasiparticle interference. Moreover, the segmented FFT signals in selective energy windows, such as [− 0.5 V, − 0.2 V], [0.05 V, 0.25 V], and [0.6 V, + 1 V], suggest that the kagome effect of Fe5Se8 phase should not be a pure structural geometry but is complicated by electronic properties.

Fig. 6
figure 6

a A series of STS mappings acquired the Fe5Se8 surface at different bias voltages (6 × 6 nm2, It = 100 pA). The individual energies are labeled. b Corresponding 2D FFT of the conductance map at 1.0 V from a. A 6-fold symmetrization is implemented. c,d 2D plot of the FFT intensities along the ΓM and ΓK directions in b, respectively, showing the nondispersive properties of the kagome geometry in the energy range of [− 1 V, + 1 V]

4 Conclusions

In summary, by taking full advantage of a well-controlled MBE growth process, we successfully fabricate various kinds of FexSey phases. Their synthetic paths are summarized by the strategy of lowering substrate temperature combined with overflux Se annealing. The structural transitions and electronic properties are also systematically characterized by STM/STS measurements. While the Fe5Se8 surface presents a kagome geometry with an electronic effect, the Fe4Se5 phase is insulating with an AFM ground state. The spin-polarized signals are validated by the striped features, as well as its magnetic response and phase shift of adjacent domains. The surface electronic doping by K dosing is also used to balance the hole doping of Fe vacancies in Fe4Se5, which however is insufficient to induce superconductivity. Our work encourages to expansion of accessible atomic construction for FeSe-derived thin films, as well as exploring the close connections between magnetic stripes and charge-ordering phases crossing over the FeSe phase diagram.