Three-dimensional nematic spin liquid in a stacked triangular lattice 6H-B structure

Recently, a number of experiments indicate the possible presence of spin-liquid phases in quantum magnets with spin-12 and spin-1 moments sitting on triangular-lattice-based structures in Ba₃CuSb₂O ₉ and Ba₃NiSb₂O₉, respectively. In relation to these experiments, several theoretical proposals have been made for spin-liquid phases and spin-liquid-like behaviors on the stacked triangular lattice. While the crystal structures of these materials are currently under debate, it is nonetheless interesting to understand possible spin-liquid phases on such frustrated lattices. In this work, we apply Schwinger boson mean-field theory and projective symmetry group analysis to investigate spin-liquid phases on the fully three-dimensional 6H-B structure, in contrast to previous works that considered two-dimensional systems. We find that a nematic Z₂ spin-liquid phase, where the lattice-rotational symmetry is spontaneously broken, is the most promising spin-liquid phase that is consistent with spiral magnetic ordering in the classical limit. We discuss the implications of our results to future theoretical and experimental works.