Spin nematic, dimensional reduction, and chiral spin liquid in the S=1 Kitaev-Heisenberg model with biquadratic interactions

The Kitaev model on the honeycomb lattice is an elegant realization of a quantum spin liquid, showing fractionalized excitations and topological order [1]. While it has been discussed mainly for materials with an effective spin-orbital entangled moment S=1/2 [2], theoretical studies suggest that the model could also be realized for S=1 or even larger S [3]. S=1 spin moments are special since they allow not only for dipolar but also quadrupolar fluctuations on a single site, which can lead to unconventional states, as seen, e.g., in spin nematic phases [4]. In this work, we show that the S=1 Kitaev model with bilinear-biquadratic interactions hosts many unconventional ordered and disordered phases. By using a global energy optimization scheme in combination with classical Monte Carlo simulations in the spin space U(3) [5], we obtain a comprehensive phase diagram offering quadrupolar, chiral, and dimensionally-reduced ordered phases, in addition to the known dipolar phases, i.e., the ferro, antiferro, zigzag, and stripy phases. Intriguingly, we find that the competition between the antiferromagnetic Kitaev and positive biquadratic interactions also promotes a noncoplanar chiral spin liquid at finite temperature.

[1] A. Kitaev, Ann. Phys. 321, 2 (2006).

[2] G. Jackeli and G. Khaliullin, PRL 102, 017205 (2009).

[3] P. P. Stavropoulos et al., PRL 123, 037203 (2019).

[4] H. Tsunetsugu and M. Arikawa, JPSJ 75, 083701 (2006).

[5] K. Remund et al., PRR 4, 033106 (2022).