Finite-size and composition-driven topological phase transition in (Bi$_{1-x}$In$_x$)$_2$Se$_3$ thin films

In a topological insulator (TI), if its spin-orbit coupling (SOC) strength is gradually reduced, the TI eventually transforms into a trivial insulator beyond a critical point of SOC, at which point the bulk gap closes: this is the standard description of the topological phase transition (TPT). However, this description of TPT, driven solely by the SOC (or something equivalent) and followed by closing and reopening of the bulk band gap, is valid only for infinite-size samples, and little is known how TPT occurs for finite-size samples. Here, using both systematic transport measurements on interface-engineered(Bi$_{1-x}$In$_x$)$_2$Se$_3$ thin films and theoretical simulations (with animations in Supporting Information) we show that description of TPT in finite-size samples needs to be substantially modified from the conventional picture of TPT due to surface-state hybridization and bulk confinement effects. We also show that the finite-size TPT is composed of two separate transitions, topological-normal transition (TNT) and metal-insulator transition (MIT) by providing a detailed phase diagram in the two-dimensional phase space of sample size and SOC strength.