Random self-similar trees and a hierarchical branching process

We study self-similarity in random binary rooted trees. In a well-understood case of Galton-Watson trees, a distribution on a space of trees is said to be self-similar if it is invariant with respect to the operation of pruning, which cuts the tree leaves. This only happens for the critical Galton-Watson tree (a constant process progeny), which also exhibits other special symmetries. We extend the prune-invariance setup to arbitrary binary trees with edge lengths. In this general case the class of self-similar processes becomes much richer and covers a variety of practically important situations. The main result is construction of the hierarchical branching processes that satisfy various self-similarity definitions (including mean self-similarity and self-similarity in edge-lengths) depending on the process parameters. Taking the limit of averaged stochastic dynamics, as the number of trajectories increases, we obtain a deterministic system of differential equations that describes the process evolution. This system is used to establish a phase transition that separates fading and explosive behavior of the average process progeny. We describe a class of critical Tokunaga processes that happen at the phase transition boundary. They enjoy multiple additional symmetries and include the celebrated critical binary Galton-Watson tree with independent exponential edge length as a special case. Finally, we discuss a duality between trees and continuous functions, and introduce a class of extreme-invariant processes, constructed as the Harris paths of a self-similar hierarchical branching process, whose local minima has the same (linearly scaled) distribution as the original process.