On potential theory of Markov processes with jump kernels decaying at the boundary

Motivated by some recent potential theoretic results on subordinate killed L\'evy processes in open subsets of the Euclidean space, we study processes in an open set $D\subset {\mathbb R}^d$ defined via Dirichlet forms with jump kernels of the form $J^D(x,y)=j(|x-y|)\mathcal{B}(x,y)$ and critical killing functions. Here $j(|x-y|)$ is the L\'evy density of an isotropic stable process (or more generally, a pure jump isotropic unimodal L\'evy process) in $\mathbb{R}^d$. The main novelty is that the term $\mathcal{B}(x,y)$ tends to 0 when $x$ or $y$ approach the boundary of $D$. Under some general assumptions on $\mathcal{B}(x,y)$, we construct the corresponding process and prove that non-negative harmonic functions of the process satisfy the Harnack inequality and Carleson's estimate. We give several examples of boundary terms satisfying those assumptions. The examples depend on four parameters, $\beta_1, \beta_2, \beta_3$, $\beta_4$, roughly governing the decay of the boundary term near the boundary of $D$. In the second part of this paper, we specialise to the case of the half-space $D=\mathbb{R}_+^d=\{x=(\widetilde{x},x_d):\, x_d>0\}$, the $\alpha$-stable kernel $j(|x-y|)=|x-y|^{-d-\alpha}$ and the killing function$\kappa(x)=c x_d^{-\alpha}$, $\alpha\in (0,2)$, where $c$ is a positive constant. Our main result in this part is a boundary Harnack principle which says that, for any $p>(\alpha-1)_+$, there are values of the parameters $\beta_1, \beta_2, \beta_3$, $\beta_4$, and the constant $c$ such that non-negative harmonic functions of the process must decay at the rate $x_d^p$ if they vanish near a portion of the boundary. We further show that there are values of the parameters $\beta_1, \beta_2, \beta_3$, $\beta_4$, for which the boundary Harnack principle fails despite the fact that Carleson's estimate is valid.