Anti-self-dual instantons with Lagrangian boundary conditions I : Elliptic theory

We study a nonlocal boundary value problem for anti-self-dual instantons on 4-manifolds with a space-time splitting of the boundary. The model case is $\R \times Y$, where $Y$ is a compact oriented 3-manifold with boundary $\Sigma$. The restriction of the instanton to each time slice ${t}\times\Sigma$ is required to lie in a fixed (singular) Lagrangian submanifold of the moduli space of flat connections over $\Sigma$. We establish the basic regularity and compactness properties (assuming $L^p$-bounds on the curvature) as well as the Fredholm theory in a compact model case. The motivation for studying this boundary value problem lies in the construction of instanton Floer homology for 3-manifolds with boundary. The present paper is part of a program proposed by Salamon for the proof of the Atiyah-Floer conjecture for homology-3-spheres.