High order three-term recursions, Riemann-Hilbert minors and Nikishin systems on star-like sets

We study monic polynomials $Q_n(x)$ generated by a high order three-term recursion $xQ_n(x)=Q_{n+1}(x)+a_{n-p} Q_{n-p}(x)$ with arbitrary $p\geq 1$ and $a_n>0$ for all $n$. The recursion is encoded by a two-diagonal Hessenberg operator $H$. One of our main results is that, for periodic coefficients $a_n$ and under certain conditions, the $Q_n$ are multiple orthogonal polynomials with respect to a Nikishin system of orthogonality measures supported on star-like sets in the complex plane. This improves a recent result of Aptekarev-Kalyagin-Saff where a formal connection with Nikishin systems was obtained in the case when $\sum_{n=0}^{\infty}|a_n-a|<\infty$ for some $a>0$. An important tool in this paper is the study of "Riemann-Hilbert minors", or equivalently, the "generalized eigenvalues" of the Hessenberg matrix $H$. We prove interlacing relations for the generalized eigenvalues by using totally positive matrices. In the case of asymptotically periodic coefficients $a_n$, we find weak and ratio asymptotics for the Riemann-Hilbert minors and we obtain a connection with a vector equilibrium problem. We anticipate that in the future, the study of Riemann-Hilbert minors may prove useful for more general classes of multiple orthogonal polynomials.