We theoretically study the effect of a material defect (material void) on switching errors associated with magneto-elastic switching of magnetization in elliptical magnetostrictive nanomagnets having in-plane magnetic anisotropy. We find that the error probability increases significantly in the presence of the defect, indicating that magneto-elastic switching is particularly vulnerable to material imperfections. Curiously, there is a critical stress value that gives the lowest error probability in both defect-free and defective nanomagnets. The critical stress is much higher in defective nanomagnets than in defect-free ones. Since it is more difficult to generate the critical stress in small nanomagnets than in large nanomagnets (having the same energy barrier for thermal stability), it would be a challenge to downscale magneto-elastically switched nanomagnets in memory and other applications where reliable switching is required. This is likely to be further exacerbated by the presence of defects.