Ground state of indefinite coupled nonlinear Schr\"odinger systems

In this paper, we study the ground state solutions of the following coupled nonlinear Schr\"odinger system (P) $-\Delta u_1-\tau_1 u_1 =\mu_1u_1^3+\beta u_1u_2^2$, $-\Delta u_2-\tau_2 u_2 =\mu_2u_2^3+\beta u_1^2u_2$ in $\Omega$, $u_1=u_2=0$ on $\partial\Omega$, where $\mu_1, \mu_2>0$, $\beta>0$ and $\Omega\subset \mathbb{R}^N (N\le3)$ is a bounded domain with smooth boundary. We are concerned with the indefinite case, i.e., $\tau_1, \tau_2$ are greater than or equal to the principal eigenvalue of $-\Delta$ with the Dirichlet boundary datum. By delicate variational arguments, we obtain the existence of ground state solution to $(P)$, and also provide information on critical energy levels for coupling parameter $\beta$ in some ranges.