Maximum principles for a fully nonlinear fractional order equation and symmetry of solutions

In this paper, we consider equations involving fully nonlinear nonlocal operators $$F_{\alpha}(u(x)) \equiv C_{n,\alpha} PV \int_{\mathbb{R}^n} \frac{G(u(x)-u(z))}{|x-z|^{n+\alpha}} dz= f(x,u).$$ We prove a maximum principle and obtain key ingredients for carrying on the method of moving planes, such as narrow region principle and decay at infinity. Then we establish radial symmetry and monotonicity for positive solutions to Dirichlet problems associated to such fully nonlinear fractional order equations in a unit ball and in the whole space, as well as non-existence of solutions on a half space. We believe that the methods develop here can be applied to a variety of problems involving fully nonlinear nonlocal operators. We also investigate the limit of this operator as $\alpha \rightarrow 2$ and show that $$F_{\alpha}(u(x)) \rightarrow a(-\Delta u(x)) + b |\bigtriangledown u(x)|^2 .$$