Abnormal boundary decay for stable operators

Assume $\alpha\in (0, 2)$ and $d\ge 2$. Let $\mathcal L^\alpha$ be the generator of a symmetric, but not necessarily isotropic, $\alpha$-stable process $X$ in $\mathbb R^d$ whose L\'evy density is comparable with that of an isotropic $\alpha$-stable process. In this paper, we show that the $C^{1, \rm Dini}$ regularity assumption on an open set $D\subset \mathbb R^d$ is optimal for the standard boundary decay property for nonnegative $\mathcal L^\alpha$-harmonic functions in $D$, and for the standard boundary decay property of the heat kernel $p^D(t,x,y)$ of the part process $X^D$ of $X$ on $D$ by proving the following: (i) If $D$ is a $C^{1, \rm Dini}$ open set and $h$ is a nonnegative function which is $\mathcal L^\alpha$-harmonic in $D$ and vanishes near a portion of $\partial D$, then the rate at which $h(x)$ decays to 0 near that portion of $\partial D$ is ${\rm dist} (x, D^c)^{\alpha/2}$. (ii) If $D$ is a $C^{1, \rm Dini}$ open set, then, as $x\to \partial D$, the rate at which $p^D(t,x,y)$ tends to 0 is ${\rm dist} (x, D^c)^{\alpha/2}$. (iii) For any non-Dini modulus of continuity $\ell$, there exist non-$C^{1, \rm Dini}$ open sets $D$, with $\partial D$ locally being the graph of a $C^{1, \ell}$ function, such that the standard boundary decay properties above do not hold for $D$.