Restricted Stirling and Lah number matrices and their inverses

Given $R \subseteq \mathbb{N}$ let ${n \brace k}_R$, ${n \brack k}_R$, and $L(n,k)_R$ be the number of ways of partitioning the set $[n]$ into $k$ non-empty subsets, cycles and lists, respectively, with each block having cardinality in $R$. We refer to these as the $R$-restricted Stirling numbers of the second and first kind and the $R$-restricted Lah numbers, respectively. Note that the classical Stirling numbers of the second kind and first kind, and Lah numbers are ${n \brace k} = {n \brace k}_{\mathbb{N}}$, ${n \brack k} = {n \brack k}_{\mathbb{N}}$ and $L(n,k) = L(n,k)_{\mathbb{N}}$, respectively. The matrices $[{n \brace k}]_{n,k \geq 1}$, $[{n \brack k}]_{n,k \geq 1}$ and $[L(n,k)]_{n,k \geq 1}$ have inverses $[(-1)^{n-k}{n \brack k}]_{n,k \geq 1}$, $[(-1)^{n-k} {n \brace k}]_{n,k \geq 1}$ and $[(-1)^{n-k} L(n,k)]_{n,k \geq 1}$ respectively. The inverse matrices $[{n \brace k}_R]^{-1}_{n,k \geq 1}$, $[{n \brack k}_R]^{-1}_{n,k \geq 1}$ and $[L(n,k)_R]^{-1}_{n,k \geq 1}$ exist if and only if $1 \in R$. We express each entry of each of these matrices as the difference between the cardinalities of two explicitly defined families of labeled forests. In particular the entries of $[{n \brace k}_{[r]}]^{-1}_{n,k \geq 1}$ have combinatorial interpretations, affirmatively answering a question of Choi, Long, Ng and Smith from 2006. If $1,2 \in R$ and if for all $n \in R$ with $n$ odd and $n \geq 3$, we have $n \pm 1 \in R$, we additionally show that each entry of $[{n \brace k}_R]^{-1}_{n,k \geq 1}$, $[{n \brack k}_R]^{-1}_{n,k \geq 1}$ and $[L(n,k)_R]^{-1}_{n,k \geq 1}$ is up to an explicit sign the cardinality of a single explicitly defined family of labeled forests. Our results also provide combinatorial interpretations of the $k$th Whitney numbers of the first and second kinds of $\Pi_n^{1,d}$, the poset of partitions of $[n]$ that have each part size congruent to $1$ mod $d$.