Competing structures in two dimensions: square-to-hexagonal transition

We study a system of particles in two dimensions interacting via a dipolar long-range potential $D/r^3$ and subject to a square-lattice substrate potential $V({\bf r})$ with amplitude $V$ and lattice constant $b$. The isotropic interaction favors a hexagonal arrangement of the particles with lattice constant $a$, which competes against the square symmetry of the substrate lattice. We determine the minimal-energy states at fixed external pressure $p$ generating the commensurate density $n = 1/b^2 = (4/3)^{1/2}/a^2$ in the absence of thermal and quantum fluctuations, using both analytical and numerical techniques. At large substrate amplitude $V > 0.2\, e_D$, with $e_D = D/b^3$ the dipolar energy scale, the particles reside in the substrate minima and hence arrange in a square lattice. Upon decreasing $V$, the square lattice turns unstable with respect to a zone-boundary shear-mode and deforms into a period-doubled zig-zag lattice. Analytic and numerical results show that this period-doubled phase in turn becomes unstable at $V \approx 0.074\, e_D$ towards a non-uniform phase developing an array of domain walls or solitons; as the density of solitons increases, the particle arrangement approaches that of a rhombic (or isosceles triangular) lattice. At a yet smaller substrate value estimated as $V \approx 0.046\, e_D$, a further solitonic transition establishes a second non-uniform phase which smoothly approaches the hexagonal (or equilateral triangular) lattice phase with vanishing amplitude $V$. At small but finite amplitude $V$, the hexagonal phase is distorted and hexatically locked at an angle of $\varphi \approx 3.8^\circ$ with respect to the substrate lattice. The square-to-hexagonal transformation in this two-dimensional commensurate-incommensurate system thus involves a complex pathway with various non-trivial lattice- and modulated phases.