Quotient sets in nonabelian groups

We show that for a finite, nonempty subset $A$ of a group, the quotient set $A^{-1}A:=\{a_1^{-1}a_2\colon a_1,a_2\in A\}$ has size $|A^{-1}A|\ge\frac53\,|A|$, unless $A$ is densely contained in a coset, or in a union of two cosets of a finite subgroup.