We present a detailed study of the destruction of SU(N) magnetic order in square lattice bilayer anti-ferromagnets using unbiased quantum Monte Carlo numerical simulations and field theoretic techniques. We study phase transitions from an SU(N) N\'eel state into two distinct quantum disordered "valence-bond" phases: a valence-bond liquid (VBL) with no broken symmetries and a lattice-symmetry breaking valence-bond solid (VBS) state. For finite inter-layer coupling, the cancellation of Berry phases between the layers has dramatic consequences on the two phase transitions: the N\'eel-VBS transition is first order for all N≥5 accesible in our model, whereas the N\'eel-VBL transition is continuous for N=2 and first order for N>= 4; for N=3 the N\'eel-VBL transition show no signs of first-order behavior.