Odd degree isolated points on $X_1(N)$ with rational $j$-invariant

Let $C$ be a curve defined over a number field $k$. We say a closed point $x\in C$ of degree $d$ is isolated if it does not belong to an infinite family of degree $d$ points parametrized by the projective line or a positive rank abelian subvariety of the curve's Jacobian. Building on work of Bourdon, Ejder, Liu, Odumodu, and Viray, we characterize elliptic curves with rational $j$-invariant which give rise to an isolated point of odd degree on $X_1(N)/\mathbb{Q}$ for some positive integer $N$.