We investigate the influence of Unruh radiation on matter-wave interferometry experiments using neutral objects modeled as dielectric spheres. The Unruh effect leads to a loss of coherence through momentum diffusion. This is a fundamental source of decoherence that affects all objects having electromagnetic interactions. However, the effect is not large enough to prevent the observation of interference for objects of any size, even when the path separation is larger than the size of the object. When the acceleration in the interferometer arms is large, inertial tidal forces will disrupt the material integrity of the interfering objects before the Unruh decoherence of the centre of mass motion is sufficient to prevent observable interference.