Hecke triangle groups, transfer operators and Hausdorff dimension

We consider the family of Hecke triangle groups $\Gamma_{w} = \langle S, T_w\rangle$ generated by the M\"obius transformations $S : z\mapsto -1/z$ and $T_{w} : z \mapsto z+w$ with $w > 2.$ In this case the corresponding hyperbolic quotient $\Gamma_{w}\backslash\mathbb{H}^2$ is an infinite-area orbifold. Moreover, the limit set of $\Gamma_w$ is a Cantor-like fractal whose Hausdorff dimension we denote by $\delta(w).$ The first result of this paper asserts that the twisted Selberg zeta function $Z_{\Gamma_{ w}}(s, \rho)$, where $\rho : \Gamma_{w} \to \mathrm{U}(V)$ is an arbitrary finite-dimensional unitary representation, can be realized as the Fredholm determinant of a Mayer-type transfer operator. This result has a number of applications. We study the distribution of the zeros in the half-plane $\mathrm{Re}(s) > \frac{1}{2}$ of the Selberg zeta function of a special family of subgroups $( \Gamma_w^n )_{n\in \mathbb{N}}$ of $\Gamma_w$. These zeros correspond to the eigenvalues of the Laplacian on the associated hyperbolic surfaces $X_w^n = \Gamma_w^n \backslash \mathbb{H}^2$. We show that the classical Selberg zeta function $Z_{\Gamma_w}(s)$ can be approximated by determinants of finite matrices whose entries are explicitly given in terms of the Riemann zeta function. Moreover, we prove an asymptotic expansion for the Hausdorff dimension $\delta(w)$ as $w\to \infty$.