A Riemannian optimization method to compute the nearest singular pencil

Given a square pencil $A+ \lambda B$, where $A$ and $B$ are $n\times n$ complex (resp. real) matrices, we consider the problem of finding the singular complex (resp. real) pencil nearest to it in the Frobenius distance. This problem is known to be very difficult, and the few algorithms available in the literature can only deal efficiently with pencils of very small size. We show that the problem is equivalent to minimizing a certain objective function $f$ over the Riemannian manifold $SU(n) \times SU(n)$ (resp. $SO(n) \times SO(n)$ if the nearest real singular pencil is sought), where $SU(n)$ denotes the special unitary group (resp. $SO(n)$ denotes the special orthogonal group). This novel perspective is based on the generalized Schur form of pencils, and yields competitive numerical methods, by pairing it with { algorithms} capable of doing optimization on { Riemannian manifolds. We propose one algorithm that directly minimizes the (almost everywhere, but not everywhere, differentiable) function $f$, as well as a smoothed alternative and a third algorithm that is smooth and can also solve the problem} of finding a nearest singular pencil with a specified minimal index. We provide numerical experiments that show that the resulting methods allow us to deal with pencils of much larger size than alternative techniques, yielding candidate minimizers of comparable or better quality. In the course of our analysis, we also obtain a number of new theoretical results related to the generalized Schur form of a (regular or singular) square pencil and to the minimal index of a singular square pencil whose nullity is $1$.