Anti-site disorder and competing magnetic ground states of Mn_1-<i>x</i>Sb_{2+0.67<i>x</i>}Te₄

Intrinsic magnetic topological insulators (TIs) provide a fertile playground to pursuit exotic quantum states of matter at ambient conditions, such as the quantum anomalous Hall effect. MnBi₂Te₄, with the stacking of septuple-layers, has been recently predicted as the first instance of an antiferromagnetic TI, which has triggered intensive studies on related materials. The as-grown MnBi₂Te₄ is heavily electron-doped and substituting Bi with Sb is expected to fine-tune the Fermi level with little disturbance on the magnetic structure. Interestingly, we have found that the end compound MnSb₂Te₄, isostructural to MnBi₂Te₄ at room temperature, can host two dissimilar magnetic ground states with different magnetic wavevectors but similar ordering temperatures from magnetic measurements and single-crystal neutron diffraction (SCND) experiments. Energy-dispersive X-ray spectroscopy and SCND show Mn vacancies and Mn-Sb anti-site disorder in both cases. To explain the results, we consider a model where the sign of the effective inter-septuple-layer coupling becomes sensitive to the Mn-defect concentration on the Sb site.