Fractional-Order Operators on Nonsmooth Domains

The fractional Laplacian $(-\Delta )^a$, $a\in(0,1)$, and its generalizations to variable-coefficient $2a$-order pseudodifferential operators $P$, are studied in $L_q$-Sobolev spaces of Bessel-potential type $H^s_q$. For a bounded open set $\Omega \subset \mathbb R^n$, consider the homogeneous Dirichlet problem: $Pu =f$ in $\Omega$, $u=0$ in $\mathbb R^n\setminus\Omega$. We find the regularity of solutions and determine the exact Dirichlet domain $D_{a,s,q}$ (the space of solutions $u$ with $f\in H_q^s(\overline\Omega )$) in cases where $\Omega$ has limited smoothness $C^{1+\tau }$, for $2a<\tau <\infty$, $0\le s<\tau -2a$. Earlier, the regularity and Dirichlet domains were determined for smooth $\Omega$ by the second author, and the regularity was found in low-order H\"older spaces for $\tau =1$ by Ros-Oton and Serra. The $H_q^s$-results obtained now when $\tau <\infty$ are new, even for $(-\Delta )^a$. In detail, the spaces $D_{a,s,q}$ are identified as $a$-transmission spaces $H_q^{a(s+2a)}(\overline\Omega )$, exhibiting estimates in terms of $\operatorname{dist}(x,\partial\Omega )^a$ near the boundary. The result has required a new development of methods to handle nonsmooth coordinate changes for pseudodifferential operators, which have not been available before; this constitutes another main contribution of the paper.