Nematic excitonic insulator near topological phase transitions in transition metal dichalcogenide moire superlattices

It was recently demonstrated that AB-stacked MoTe2/WSe2 bilayer superlattices exhibit quantum anomalous Hall effect (QAHE) at the filling of one hole per moire unit cell. The bilayer undergoes a topological phase transition as the displacement field is tuned. However, the gap closing feature typical of such transition is not evident in the experiment. Here, by considering the effect of electron-electron interaction, we identify an excitonic insulator phase that can preempt the topological phase transition. The insulating phase breaks C3 rotational symmetry and is disfavored by the interlayer tunneling term with a non-trivial layer-pseudospin structure. We discuss the experimental implications of the close competition between the two phases and its relevance in similar moire superlattices with non-trivial topology.