Atomic resolution imaging the nematic to bicollinear antiferromagnetic order transition in epitaxial bilayer FeSe_1-xTe_x

Iron chalcogenides (FeSe_1-xTe_x) is a model system for studying the interplay between nematicity, antiferromagnetism, and superconductivity. In this work, we grew bilayer FeSe_1-xTe_x films on the SrTiO₃(001) substrate by molecular beam epitaxy. Using in situ low-temperature scanning tunneling microscopy (STM), we show that the STM image of the Se atom is anisotropic, which is elongated along the direction of underlying Fe–Fe lattice. This breaks the rotational symmetry of the Se lattice from C₄ down to C₂, indicating electronic nematicity. As Te composition (x) increases, the nematic order becomes more localized. At x=0.46, the nematic order exists only locally. At x=0.76, the elongated Se feature disappears completely accompanying by the appearance of short-range 2x1 order. For x> 0.95, long-range (2x1) order is observed, consistent with the bicollinear antiferromagnetic phase. The phase transition is also reflected in the evolution of the differential conductance spectra. In the nematic state, the dI/dV spectra are characterized by a gap near the Fermi level and a pronounced peak at -0.2 eV, which shifts away from the Fermi level with Te incorporation. Across the phase transition, the line shape of dI/dV spectra become V-shaped at Fermi level at x = 0.76. Further increase the Te concentration leads to a dip at Fermi level with finite density of states at x> 0.95.