Smecticlike phase for modulated XY spins in two dimensions

The row model for frustrated XY spins on a triangular lattice in two dimensions (2D) is used to study incommensurate (IC) spiral and commensurate (C) antiferromagnetic phases, in the regime where a C-IC transition occurs. Using fluctuating boundary conditions and specific histogram techniques, a detailed Monte Carlo (MC) study reveals more structure in the phase diagram than found in previous MC simulations of the full parameter space. On the C side, equilibrium configurations consist of alternating stripes of spiral phases of opposite chirality separated by walls of the C phase. For this same parameter regime, thermodynamic quantities are computed analytically using the NSCHA, a generalization of the self-consistent harmonic approximation appropriate for chiral systems. On the commensurate side of the C-IC boundary, NSCHA predicts an instability of the C phase. This suggests that the state is spatially inhomogeneous, consistent with the present MC result: it resembles the smectic-A phase of liquid crystals, and its existence implies that the Lifshitz point is at T=0 for modulated XY spins in 2D. The connection between frustrated XY systems and the vortex state of strong type-II superconductors suggests that the smectic phase may correspond to a vortex liquid phase of superconducting layers.