Mixing efficiency of trans-model Markov chain Monte Carlo algorithms with applications in Bayesian phylogenetics

Trans-model Markov chain Monte Carlo (MCMC) algorithms are widely used in Bayesian inference, and are particularly important in Bayesian phylogenetics where phylogenetic trees represent different statistical models. While the algorithm allows great flexibility, its mixing efficiency can vary hugely, and is poorly understood. Here we use mathematical analysis and simulation to explore the mixing efficiency of trans-model MCMC proposals, including the model-proposal probabilities and the proposal kernel for model parameters. Our analysis confirms the intuition that one should preferentially propose models with high posterior probabilities, and propose parameter values from the posterior as much as possible. Our results provide guidelines for constructing efficient trans-model MCMC algorithms. The principles are applied to MCMC algorithms in phylogenetic reconstruction using two real datasets for primates and mammals.