In analogy to the unconventional superconductors of cuprates and iron pnictides, interface-enhanced high-transition temperature (high-T_C) superconductivity in single-layer FeSe films on SrTiO₃ has boomed an upsurge in a great deal of attractable interest in the condensed-matter community [1,2,3,4,5], including the unusual interfacial effect, the underlying pairing mechanism, the magnetic ground states and other competing orders. Till now, some degree of consensus has been reached on the origin of T_C enhancement, ascribing to a combination of two major contributing factors—the interface-induced channel (tensile strain [6,7,8,9] and electron-phonon coupling [3, 10,11,12]) and electron doping effect [13,14,15,16]. Meanwhile, there emerge extensive explorations concerning the electron phase diagram [16,17,18], electron pairing symmetries [19,20,21,22], electron-phonon coupling [23,24,25], pre-formed pairing of Cooper pairs and pseudogap behavior [26,27,28], etc.

Among these debates, searching for other FeSe-derived compounds, associated with their relationship to single-layer FeSe/SrTiO₃ films, may be a plausible route to understanding the mystery of unconventional mechanisms and interfacial enhancement in superconductivity. For instance, both transport and ARPES experiments argued that the Fe-vacancy ordered Fe₄Se₅ [29, 30] (or undoped FeSe/SrTiO₃ films [31]) should be the insulating parent phase of FeSe (or FeSe/SrTiO₃). In particular, the Fe₄Se₅ phase is manifested as a Mott insulator possessing a block-checkerboard antiferromagnetic (AFM) order, resembling the A₂Fe₄Se₅ compounds (A = K, Tl, Rb, or Cs) [32, 33]. Such similarity between A_1–xFe_2–ySe₂ and FeSe/SrTiO₃ films is also consistent with the dramatic T_C enhancement of FeSe superconductor by heavy electron doping [15, 34, 35]. Also, there report a large amount of FeSe-based superconductors by intercalation of organic molecules or inorganic elements, such as Li_x(NH₂)_y(NH₃)₁–_yFe₂Se₂ [36], (Li_0.8Fe_0.2)OHFeSe [37], (CTA)_0.3FeSe [38]. Different from the wet-chemical synthesis of liquid-ammonia and electrochemical methods, molecular beam epitaxy (MBE) is a liquid-free and controllable growth to gain different constituents of FeSe with high quality of atomically flat thin films, as well as investigating their complex phase transitions. Previously, we have also actualized the MBE-controlled synthesis on the multiple phases of the sister Fe–Te compound with three structural phases: α-FeTe, β-FeTe, and FeTe₂ of different intersections [39]. Other FeSe_x phases are sporadically reported [40, 41]; however, there is still a lack of a concordant growth scenario for the FeSe-derived thin films.

In this study, we report a compositive route to synthesize distinct Fe_xSe_y phases based on the single-layer FeSe/SrTiO₃ films via MBE growth. The key essentials lie in the elaborate control of supplying excess Se flux and low-temperature annealing treatment. Various stoichiometries of FeSe-derived compounds are synthesized with atomically smooth surfaces and largely flat terraces, including α-FeSe, β-FeSe, Fe₃Se₄, Fe₄Se₅, Fe₅Se₆, Fe₆Se₇, and Fe₅Se₈. While Fe₄Se₅, Fe₅Se₆, and Fe₆Se₇ are distinguished by different densities of ordered Fe vacancies, Fe₃Se₄ and Fe₅Se₈ are constructed by selective intercalations of Fe atoms. Scanning tunneling microscopy/spectroscopy (STM/STS) measurements are used to reveal the structural characteristics, electronic properties, and phase transitions among these Fe_xSe_y phases. Specifically, Fe₄Se₅ exhibits an in-plane AFM order with unidirectional stripes as real-space spin contrasts, which is sustained by the magnetic response of AFM domains. We also discuss the electronic doping effect in insulating Fe₄Se₅ by surface K dosing, which seems inadequate to drive Fe₄Se₅ into the superconducting regime. Our MBE implication of fabricating different stoichiometries of Fe_xSe_y phases sheds light on constructing and transforming FeSe-related families, ranging from 3D to 2D systems, which is expected to be an appealing platform to create and tune potentially exotic states in the extensive library of quantum materials.

All Fe_xSe_y thin samples were prepared by the MBE method in a customized ultrahigh vacuum chamber at a base pressure of 5 × 10⁻¹⁰ mbar [42]. Before growth, the atomically flat Nb-doped SrTiO₃(001) substrate (0.7 wt %) was obtained by (1) degassing at 873 K for 2 h, (2) thermally heated to ∼ 1223 K with a Se flux for half an hour, (3) flashed to 1320 K for an hour. High-purity Fe (powder, 99.99%; Alfa Aesar) and Se sources (shot, 99.9999%; Alfa Aesar) were used for co-evaporating from standard K cells. During the growth process, the Fe flux was fixed at 1370 K, giving a growth rate of ∼ 0.2 monolayer/min. The Se/Fe flux ratio was adjusted by precisely varying the temperature of the Se source. The substrate temperature was controlled by direct current heating, which was monitored by an infrared pyrometer.

As a starting point, epitaxial β-FeSe/SrTiO₃ films were first fabricated, with the substrate holding at 673 K [2]. It can become superconducting by annealing to 773 K for 2 h [14]. The hexagonal α-FeSe phase was obtained by annealing β-FeSe at 473 K for 1 h. Variant Fe-vacancy phases were achieved by gently annealing the as-grown sample at 373–423 K with a Se overflux for 30 min to 2 h. The K deposition was performed in situ at a temperature of 20–50 K by using a potassium dispenser (SAES Getters). The coverage of K adatoms is accumulated by repeated depositions onto the Fe_xSe_y/SrTiO₃ surface, with 10–30 s in each cycle [43].

The low-temperature STM/STS measurements were conducted on a commercial STM system (Unisoku-1500) at a base temperature of 2 K [44]. W tip was electrochemically etched and then cleaned by e-beam at ~ 1900 K to remove its apex oxides, which were calibrated on a standard Ag(111) sample prior to STM measurements. All topographic images were taken in a constant-current mode, and the tunneling spectra and conductance mappings were acquired by a standard lock-in technique with a modulation voltage (V_mod) at 983 Hz [45]. The spin-polarized STM (SP-STM) measurements were carried out by using a spin-polarized Fe-coated W tip, which was fabricated by depositing ~ 20 nm Fe clusters onto the apex of the calibrated W tip, followed by annealing at ~ 500 K for 10 min. The spin-polarized properties of the Fe-coated tip were verified by mapping the zigzag patterns on an AFM CrTe₂ surface [46]. An external magnetic field was applied perpendicular to the surface of FeSe/SrTiO₃ films.

The formation of different Fe_xSe_y phases is achieved by the precise combination of (i) controlling the substrate temperature, (ii) adjusting the Se/Fe ratio, and (iii) post-annealing under Se flux, as illustrated in Fig. 1a. Firstly, we start from the easily obtained β-FeSe films by the mature process as reported [2, 14]. On the one hand, it can convert to α-FeSe thin films by annealing at a low temperature of ~ 473 K, with the typical STM morphology shown in Fig. 1b. Obviously, the islands are triangle-shaped with large terraces of atomically smooth Se-terminated surface, indicating a phase transition from tetragonal β-FeSe (layered) into hexagonal α-FeSe (nonlayered). Their high-resolution STM images associated with their crystalline structural models are shown in Fig. 2a, b. On the other hand, by gradually increasing the Se flux, we can obtain various ordered phases from β-FeSe. As reported in our previous study [47], the distinct superstructures of √5 × √10 (1/7 Fe vacancies or Fe₆Se₇), 2 × √10 (1/6 Fe vacancies or Fe₅Se₆), and √5 × √5 (1/5 Fe vacancies or Fe₄Se₅) mainly depend on the amount of Fe vacancies. Figs. 1d and 2c show the STM image of the √5 × √5 phase, with the orientation of the unit cell rotated 26.6° relative to the topmost Se lattice. It is noteworthy that all the ordered phases of Fe vacancies can only be observed with a thickness ≥ 2 layers, but is always absent in the monolayer case, suggesting that the site of Fe vacancies should be occupied in the van der Waals gap of FeSe–FeSe interlayer instead of the FeSe-SrTiO₃ interface. Also, these superstructures with varied Fe vacancies show no difference among different thicknesses, they should be bulk structures instead of only surface reconstructions.

Upon increasing the Se overflux at low temperature, more Fe vacancies are formed and the √5 × √5 phase further transforms into Fe₃Se₄ and Fe₅Se₈, respectively. Strategically speaking, additional interstitial Fe atoms are expected to effectively intercalate into the β-FeSe layers by such a low-temperature Se annealing process [48], and we can regard the Fe₃Se₄ and Fe₅Se₈ phase as the different levels of self-intercalation of Fe atoms within the interlayer space of the 1T-FeSe₂ compound. For Fe₃Se₄, while the Fe atoms along the b axis are intact, Fe vacancies are only formed along the a-axis for every two Fe rows. This leads to a √3×1 reconstruction of the intercalated Fe atoms and can be reflected by the striped patterns of the topmost Se surface, as seen in Fig. 2d. For the Fe₅Se₈ phase, the intercalated Fe atoms are alternatively arranged along both the a and b axes of the Fe layer, resulting in a 2 × 2 pattern of kagome-like morphology (Fig. 2e). Such a kagome lattice is comprised of ordered hexagons that are interconnected by interlaced triangles, combining into arrays of corner-sharing triangles. In Table 1, we compare the crystalline properties of different Fe_xSe_y phases, including stoichiometry, Fe/Se atomic ratio, STM morphology, and in-plane lattice constant.

Overall, the growth phase strongly depends on the detailed annealing treatments with Se flux at low temperatures. Low substrate temperature mainly drives the triangular structure of α-FeSe, and the supplied Se flux provides the formation of Fe-vacancies (or intercalated Fe). These transitions are 100% with pure α-FeSe and Fe₄Se₅ phases. With more Fe-vacancies, the compounds Fe₃Se₄ and Fe₅Se₈ are formed, where the intercalation concentration strongly relies on the annealing time. Statistically, a nearly pure Fe₃Se₄ phase (~ 100%) can be obtained with different striped domains, as displayed in Fig. 1e. Unfortunately, with further annealing, the sample is maximumly covered by 70% Fe₅Se₈ phase, whereas the other 30% regions are Fe₃Se₄. This means the efficiency of intercalating Fe atoms is limited at this stage. In the future, we hope to construct the expected FeSe₂ phase with a layered structure [49] via such an intercalation strategy.

Having identified the detailed crystalline structure and STM morphologies of different Fe_xSe_y phases, we then systematically study their electronic structure by performing low-temperature STS measurements, where the dI/dV spectrum is proportional to its local density of states. Figure 3a summarizes the tunneling conductance for all the observed Fe_xSe_y phases. While β-FeSe is metallic (superconductive), α-FeSe becomes semiconducting with a band gap of ~ 0.2 eV. With increasing the concentration of Fe vacancies, it shows an insulating behavior with a typical gap size of 0.5–1.0 eV. When coming into the intercalation regime, it turns to be metallic again, with an overall “V” shape of STS spectra. The conductance minimum located around the Fermi level may be possibly related to the preformed kagome band structure of Dirac cones [48]. Such distinct electronic properties can also be apparently compared by taking STS line cuts across the domain boundaries among different phases. As seen in Fig. 3b–e, an insulator-to-metal transition is clearly observed when the √5 × √5 Fe₄Se₅ is partially converted into the kagome Fe₅Se₈ phase. However, the increase of Fe intercalation from Fe₃Se₄ to Fe₅Se₈ only scarcely affects their electronic structure, namely, the metallic behavior, the overall landscape, the intensities of conductance, and the location of conductance minimum are slightly changed. This similarity of electronic structure is qualitatively in accordance with their surface consistency, where their topmost Se atoms are connected continuously, with only difference in the intercalated Fe densities of the buried Fe layers (50% for Fe₃Se₄ and 25% for Fe₅Se₈) [48].

For the insulating Fe₄Se₅, we have previously demonstrated its pair-checkerboard AFM ground state with in-plane magnetization by SP-STM measurements [47], where every √5 × √5 lattice is centered by a four-Fe block with two up and two down spins that are collinear AFM aligned, as illustrated in Fig. 4a, b. Its real-space spin contrast is featured by unidirectional stripes in the STM images, which recover as a 4-fold symmetry morphology by applying a perpendicular magnetic field larger than 1 T. Here, the tunneling current (\({I}_{t}\)) is the total sum of the spin-averaged tunneling current (\({I}_{0}\)) and spin-polarized tunneling current (\({I}_{\mathrm{SP}})\) expressed by \({I}_{t}= {I}_{0}+ {I}_{\mathrm{SP}}\), where \({I}_{\mathrm{SP}}\) is sensitive to the relative spin angle (\(\varphi\)) between the magnetization of the tip (\({P}_{\mathrm{tip}}\)) and the sample (\({P}_{\mathrm{sample}}\)), describing as \({I}_{SP}\propto {P}_{\mathrm{tip}}\bullet {P}_{\mathrm{sample}}\bullet cos\varphi\) [50]. In this way, the acquired SP-STM image strongly depends on the relative orientation of local spin for the sample and tip. In our Fe-coated W tip, the ferromagnetic (FM) tip apex is generally magnetized along the normal direction of the tip axis. Thus, it is sensitive to the in-plane component of sample magnetization (φ = 0 or π) but behaves as a nonmagnetic (NM) W tip if the spin structure orientates in the out-of-plane direction (φ = π/2) at the surface. The orientation of \({P}_{\mathrm{tip}}\) can easily follow the direction of an external magnetic field (B) of ~ 1 T, tilting \(\varphi\) between in-plane and out-of-plane of the surface, while \({P}_{\mathrm{sample}}\) remains undisturbed [50]. Therefore, we can image the surface in a constant-current mode by varying an applied magnetic field: tip- and energy-dependent measurements help to differentiate the magnetic information from topography, and the local spin polarization of the sample can be determined at the atomic scale.

Occasionally, due to the instability of the Fe clusters attached to the cusp of the STM tip, the FM property of the Fe-coated tip may suddenly change its spin-polarized state. Fig. 4c, d are two examples of SP-STM images by a Fe-coated W tip experiencing condition change, that is into an NM order, or reversing its magnetization in the opposite direction. As shown in Fig. 4c, the SP-STM image loses the magnetic contrast without striped features (lower half of the image). Fig. 4d is a direction comparison of opposite tip magnetization, exhibiting a π-phase slip of the unidirectional stripes. We note that the detailed structure of stripes varies slightly between Fig. 4c, d, mainly involving the shapes of Se atoms and their connection. This may be explained by the different shapes of Fe tips that affect the spin contrast of the topmost Se atom. However, there persists the key characteristic of filed-dependent STM stripes and tip magnetization flipping under historical ramping experiences for different Fe tips. In this regard, the striped features should stem from the magnetic properties of the sample itself, but not from the specificity of Fe tips.

The AFM domains also provide an alternative view to demonstrate its pair-checkerboard AFM configuration. Fig. 4e, f illustrate the SP-STM images comprised of two AFM domains with 0° boundary. It experiences a magnetic field ramp from 0 to 3 T, then backward to 0 T. A 1/2-unit-cell shift (π-phase slip) of the unidirectional stripes is obviously visible across the domain boundaries in Fig. 4d. Keep the defect marker (yellow circle) for atomic registry in mind, the relative position of the √5 × √5 unit cell (magenta squares) exhibits a phase shift of 1/2-unit-cell along both a’ and b’ directions after the magnetic field cycle, while the π-phase slip can still be observed. This can be well understood by the relative orientations of tip and sample magnetizations, as vividly manifested by the inserted cartoons.

We also check the charge doping effect of alkali metals dosing on the AFM Fe₄Se₅ surface, which has been demonstrated as an impactful strategy to tune the superconductivity of FeSe films towards the heavily doped region [43]. As displayed in Fig. 5a–f, extra potassium (K) adatoms are introduced with different coverages and, thus different electronic doping levels. In Fig. 5g, even though the individual K atoms are randomly adsorbed, the dI/dV spectra show the homogenously distributed insulating gap against disordered K adatoms, indicating a uniform charge doping effect. Moreover, the band gap gradually shrinks as the K coverage increases, as seen in Fig. 5h. We summarize the doping dependence of insulating gap size on the Fe₄Se₅ phase in Fig. 5i. Obviously, through a wide range of electron doping to a coverage of 0.3 ML K atoms, Fe₄Se₅ remains insulating with a decreased band gap by heavily doping. It had been theoretically proposed that the √5-pair-checkerboard AFM state will turn into the √5-blocked-checkerboard AFM order (reminiscent of K₂Fe₄Se₅) upon electron doping, meanwhile, both states keep insulating [51].

Annealing the Fe₄Se₅ film can annihilate Fe vacancies, which are considered as hole dopants. Thus, the Fe_xSe_y phases with reduced Fe vacancy concentrations are equivalent to electron doping to the Fe₄Se₅ films, as displayed by the decreasing band gap of other FeSe_x phases with lower Fe vacancies in Fig. 5i. Such ascription has been reported in [29], and is also collaborated from our experiments of K-doping to the Fe₄Se₅ film. Unfortunately, we failed to observe the induced superconductivity with more K doping, different from what had been done in multiple-layer FeSe/SrTiO₃ films [15,16,17, 43]. Here, we speculate a plausible scenario as follows: Ideally, every isolated K adatom can donate a free electron to the Fe vacancies with positive charges buried underneath the topmost Se surface. However, when the K coverage grows, they tend to form clusters or islands. As such, it becomes difficult for the inner K atoms to dope electrons to the Fe vacancy layer. That is to say, it becomes difficult to achieve effective doping with increasing K for high coverage. This can be seen from the rapid decrease of gap size at low K coverage but gradually becomes saturated by adding more K adatoms.

Finally, we also carry out the STS mappings on the kagome Fe₅Se₈ phase at different energies. As shown in Fig. 6a, the tunneling conductance shows a continuous evolution with increasing the bias voltage, which also delivers analogous modulations to the topographic images at each individual energy [48], suggesting a close correlation between the morphology and local density of states. These energy-dependent dI/dV mappings are further analyzed by the 2D FFT in Fig. 6b–d. By plotting the FFT intensities along the ΓM and ΓK directions, respectively, we find that the signals show nondispersive behavior of the kagome geometry in the whole energy range, well excluding the possible origin of quasiparticle interference. Moreover, the segmented FFT signals in selective energy windows, such as [− 0.5 V, − 0.2 V], [0.05 V, 0.25 V], and [0.6 V, + 1 V], suggest that the kagome effect of Fe₅Se₈ phase should not be a pure structural geometry but is complicated by electronic properties.

In summary, by taking full advantage of a well-controlled MBE growth process, we successfully fabricate various kinds of Fe_xSe_y phases. Their synthetic paths are summarized by the strategy of lowering substrate temperature combined with overflux Se annealing. The structural transitions and electronic properties are also systematically characterized by STM/STS measurements. While the Fe₅Se₈ surface presents a kagome geometry with an electronic effect, the Fe₄Se₅ phase is insulating with an AFM ground state. The spin-polarized signals are validated by the striped features, as well as its magnetic response and phase shift of adjacent domains. The surface electronic doping by K dosing is also used to balance the hole doping of Fe vacancies in Fe₄Se₅, which however is insufficient to induce superconductivity. Our work encourages to expansion of accessible atomic construction for FeSe-derived thin films, as well as exploring the close connections between magnetic stripes and charge-ordering phases crossing over the FeSe phase diagram.