Cancellative hypergraphs and Steiner triple systems

A triple system is cancellative if it does not contain three distinct sets $A,B,C$ such that the symmetric difference of $A$ and $B$ is contained in $C$. We show that every cancellative triple system $\mathcal{H}$ that satisfies certain inequality between the sizes of $\mathcal{H}$ and its shadow must be structurally close to the balanced blowup of some Steiner triple system. Our result contains a stability theorem for cancellative triple systems due to Keevash and Mubayi as a special case. It also implies that the boundary of the feasible region of cancellative triple systems has infinitely many local maxima, thus giving the first example showing this phenomenon.