On second variation of Wang-Yau quasi-local energy

We study a functional on the boundary of a compact Riemannian 3-manifold of nonnegative scalar curvature. The functional arises as the second variation of the Wang-Yau quasi-local energy in general relativity. We prove that the functional is positive definite on large coordinate spheres, and more general on nearly round surfaces including large constant mean curvature spheres in asymptotically flat 3-manifolds with positive mass; it is also positive definite on small geodesics spheres, whose centers do not have vanishing curvature, in Riemannian 3-manifolds of nonnegative scalar curvature. We also give examples of functions H, which can be made arbitrarily close to the constant 2, on the standard sphere such that the boundary data consisting of the standard spherical metric and H has positive Brown-York mass while the associated functional is negative somewhere.