Positive Least Energy Solutions and Phase Separation for Coupled Schrodinger Equations with Critical Exponent: Higher Dimensional Case

We study the following nonlinear Schr\"{o}dinger system which is related to Bose-Einstein condensate: {displaymath} {cases}-\Delta u +\la_1 u = \mu_1 u^{2^\ast-1}+\beta u^{\frac{2^\ast}{2}-1}v^{\frac{2^\ast}{2}}, \quad x\in \Omega, -\Delta v +\la_2 v =\mu_2 v^{2^\ast-1}+\beta v^{\frac{2^\ast}{2}-1} u^{\frac{2^\ast}{2}}, \quad x\in \om, u\ge 0, v\ge 0 \,\,\hbox{in $\om$},\quad u=v=0 \,\,\hbox{on $\partial\om$}.{cases}{displaymath} Here $\om\subset \R^N$ is a smooth bounded domain, $2^\ast:=\frac{2N}{N-2}$ is the Sobolev critical exponent, $-\la_1(\om)<\la_1,\la_2<0$, $\mu_1,\mu_2>0$ and $\beta\neq 0$, where $\lambda_1(\om)$ is the first eigenvalue of $-\Delta$ with the Dirichlet boundary condition. When $\bb=0$, this is just the well-known Brezis-Nirenberg problem. The special case N=4 was studied by the authors in (Arch. Ration. Mech. Anal. 205: 515-551, 2012). In this paper we consider {\it the higher dimensional case $N\ge 5$}. It is interesting that we can prove the existence of a positive least energy solution $(u_\bb, v_\bb)$ {\it for any $\beta\neq 0$} (which can not hold in the special case N=4). We also study the limit behavior of $(u_\bb, v_\bb)$ as $\beta\to -\infty$ and phase separation is expected. In particular, $u_\bb-v_\bb$ will converge to {\it sign-changing solutions} of the Brezis-Nirenberg problem, provided $N\ge 6$. In case $\la_1=\la_2$, the classification of the least energy solutions is also studied. It turns out that some quite different phenomena appear comparing to the special case N=4.