Theory of the spin-nematic to spin-Peierls quantum phase transition in ultracold spin-1 atoms in optical lattices

We present a theory of the anisotropy-tuned quantum phase transition between spin-nematic and spin-Peierls phases in S=1 systems with significant biquadratic exchange interactions. Based on quantum Monte Carlo studies on finite-size systems, it has been proposed that this phase transition is second order with new deconfined fractional excitations that are absent in either of the two phases. The possibility of a weak first-order transition, however, cannot be ruled out. To elucidate the nature of the transition, we construct a large- N SO (3N) model for this phase transition and find in the N→∞ limit that the transition is generically of first-order. Furthermore, we find a critical point in the one-dimensional (1D) limit, where two transition lines, separating spin-nematic, ferromagnetic, and spin-Peierls phases, meet. Our study indicates that the spin-nematic phase is absent in 1D, while its correlation length diverges at the critical point. Predictions for N 23 a atoms trapped in an optical lattice, where the nematic to spin-Peierls quantum phase transition naturally arises are discussed.