Critical edge behavior in the modified Jacobi ensemble and Painlev\'e equations

We study the Jacobi unitary ensemble perturbed by an algebraic singularity at $t>1$. For fixed $t$, this is the modified Jacobi ensemble studied by Kuijlaars {\it{et al.}} The main focus here, however, is the case when the algebraic singularity approaches the hard edge, namely $t\to 1^+$. In the double scaling limit case when $t- 1$ is of the order of magnitude of $1/n^2$, $n$ being the size of the matrix, the eigenvalue correlation kernel is shown to have a new limiting kernel at the hard edge $1$, described by the $\psi$-functions for a certain second-order nonlinear equation. The equation is related to the Painlev\'e III equation by a M\"obius transformation. It also furnishes a generalization of the Painlev\'e V equation, and can be reduced to a particular Painlev\'e V equation via the B\"acklund transformations in special cases. The transitions of the limiting kernel to Bessel kernels are also investigated, with $n^2(t-1)$ being large or small. In the present paper, the approach is based on the Deift-Zhou nonlinear steepest descent analysis for Riemann-Hilbert problems.