Sets of unit fractions without two members whose average is a unit fraction

We show that there is a constant $c>0$ such that, for all sufficiently large $N$, there is a subset $A \subseteq \{1,\dots,N\}$ of size $>cN$ such that for any two distinct elements $a,b$ in $A$, the average of $\frac{1}{a}$ and $\frac{1}{b}$ is not a unit fraction, negatively answering a question of Erd\H{o}s and Graham. This also gives the best known lower bounds on the maximum size of a set of unit fractions without non-trivial three-term arithmetic progressions.