A generative model of function growth explains hidden self-similarities across biological and social systems

From genomes and ecosystems to bureaucracies and cities, the growth of complex systems occurs by adding new types of functions and expanding existing ones. We present a simple generative model that generalizes the Yule-Simon process by including: (i) a size-dependent probability of introducing new functions, and (ii) a generalized preferential attachment mechanism for expanding existing ones. We uncover a shared underlying structure that helps explain how function diversity evolves in empirical observations, such as prokaryotic proteomes, U.S. federal agencies, and urban economies. We show that real systems are often best represented as having non-Zipfian rank-frequency distributions, driven by sublinear preferential attachment, whilst still maintaining power-law scaling in their abundance distributions. Furthermore, our analytics explain five distinct phases of the organization of functional elements across complex systems. The model integrates empirical findings regarding the logarithmic growth of diversity in cities and the self-similarity of their rank-frequency distributions. Self-similarity previously observed in the rank-frequency distributions of cities is not observed in cells and federal agencies -- however, under a rescaling relative to the total diversity, all systems admit self-similar structures predicted by our theory.