A relative trace formula and counting geodesic arcs in the hyperbolic plane

We study a modification of the hyperbolic circle problem: instead of all elements of a Fuchsian group $\Gamma$, we consider the double cosets by two hyperbolic subgroups. This has a geometric interpretation in terms of the number of common perpendiculars between two closed geodesics for $\Gamma \backslash\mathbb{H}$. We prove an explicit relative trace formula, which is flexible for the counting problem. Using a large sieve inequality developed by the first author and Voskou, we prove a new bound in mean square for the error term of order $O(X^{1/2}\log X)$. We conjecture that this is the correct order of growth. Along the way we provide a new proof of the pointwise error bound $O(X^{2/3})$, originally proved by Good.