Existence of ground state solutions of Nehari-Pankov type to Schr\"odinger systems

This paper is dedicated to studying the following elliptic system of Hamiltonian type: $$\left\{ \begin{array}{ll} -\varepsilon^2\triangle u+u+V(x)v=Q(x)F_{v}(u, v), \ \ \ \ x\in \mathbb{R}^N,\\ -\varepsilon^2\triangle v+v+V(x)u=Q(x)F_{u}(u, v), \ \ \ \ x\in \mathbb{R}^N,\\ |u(x)|+|v(x)| \rightarrow 0, \ \ \mbox{as} \ |x|\rightarrow \infty, \end{array}\right.$$ where $N\ge 3$, $V, Q\in \mathcal{C}(\mathbb{R}^N, \mathbb{R})$, $V(x)$ is allowed to be sign-changing and $\inf Q > 0$, and $F\in \mathcal{C}^1(\mathbb{R}^2, \mathbb{R})$ is superquadratic at both $0$ and infinity but subcritical. Instead of the reduction approach used in [Calc Var PDE, 2014, 51: 725-760], we develop a more direct approach -- non-Nehari manifold approach to obtain stronger conclusions but under weaker assumptions than these in [Calc Var PDE, 2014, 51: 725-760]. We can find an $\varepsilon_0>0$ which is determined by terms of $N, V, Q$ and $F$, then we prove the existence of a ground state solution of Nehari-Pankov type to the coupled system for all $\varepsilon\in (0, \varepsilon_0]$.