Isolated Points on $X_1(\ell^n)$ with rational $j$-invariant}

Let $\ell$ be a prime and let $n\geq 1$. In this note we show that if there is a non-cuspidal, non-CM isolated point $x$ with a rational $j$-invariant on the modular curve $X_1(\ell^n)$, then $\ell=37$ and the $j$-invariant of $x$ is either $7\cdot11^3$ or $-7.137^3\cdot2083^3$. The reverse implication holds for the first j-invariant but it is currently unknown whether or not it holds for the second.