Membership problems for positive one-relator groups and one-relation monoids

Motivated by approaches to the word problem for one-relation monoids arising from work of Adian and Oganesian (1987), Guba (1997), and Ivanov, Margolis and Meakin (2001), we study the submonoid and rational subset membership problems in one-relation monoids and in positive one-relator groups. We give the first known examples of positive one-relator groups with undecidable submonoid membership problem, and apply this to give the first known examples of one-relation monoids with undecidable submonoid membership problem. We construct several infinite families of one-relation monoids with undecidable submonoid membership problem, including examples that are defined by relations of the form $w=1$ but which are not groups, and examples defined by relations of the form $u=v$ where both of $u$ and $v$ are non-empty. As a consequence we obtain a classification of the right-angled Artin groups that can arise as subgroups of one-relation monoids. We also give examples of monoids with a single defining relation of the form $aUb = a$, and examples of the form $aUb=aVa$, with undecidable rational subset membership problem. We give a one-relator group defined by a freely reduced word of the form $uv^{-1}$ with $u, v$ positive words, in which the prefix membership problem is undecidable. Finally, we prove the existence of a special two-relator inverse monoid with undecidable word problem, and in which both the relators are positive words. As a corollary, we also find a positive two-relator group with undecidable prefix membership problem. In proving these results, we introduce new methods for proving undecidability of the rational subset membership problem in monoids and groups, including by finding suitable embeddings of certain trace monoids.