On Siegel's problem and Dwork's conjecture for $G$-functions

We answer in the negative Siegel's problem for $G$-functions, as formulated by Fischler and Rivoal. Roughly, we prove that there are $G$-functions that cannot be written as polynomial expressions in algebraic pullbacks of hypergeometric functions; our examples satisfy differential equations of order two, which is the smallest possible. In fact, we construct infinitely many non-equivalent rank-two local systems of geometric origin which are not algebraic pullbacks of hypergeometric local systems, thereby providing further counterexamples to Dwork's conjecture and answering a question by Krammer. The main ingredients of the proof are a Lie algebra version of Goursat's lemma, the monodromy computations of hypergeometric local systems due to Beukers and Heckman, as well as results on invariant trace fields of Fuchsian groups.