Spectral Tur\'an Type Problems on Cancellative Hypergraphs

Let $G$ be a cancellative $3$-uniform hypergraph in which the symmetric difference of any two edges is not contained in a third one. Equivalently, a $3$-uniform hypergraph $G$ is cancellative if and only if $G$ is $\{F_4, F_5\}$-free, where $F_4 = \{abc, abd, bcd\}$ and $F_5 = \{abc, abd, cde\}$. A classical result in extremal combinatorics stated that the maximum size of a cancellative hypergraph is achieved by the balanced complete tripartite $3$-uniform hypergraph, which was firstly proved by Bollob\'as and later by Keevash and Mubayi. In this paper, we consider spectral extremal problems for cancellative hypergraphs. More precisely, we determine the maximum $p$-spectral radius of cancellative $3$-uniform hypergraphs, and characterize the extremal hypergraph. As a by-product, we give an alternative proof of Bollob\'as' result from spectral viewpoint.