Probing a nonminimal coupling through superhorizon instability and secondary gravitational waves

In this paper, we investigate the impact of scalar fluctuations ($\chi$) non-minimally coupled to gravity, $\xi\chi^2 R$, as a potential source of secondary gravitational waves (SGWs). Our study reveals that when reheating EoS $\wre < 1/3$ and $\xi \lesssim 1/6$ or $\wre > 1/3$ and $\xi \gtrsim 1/6$, the super-horizon modes of scalar field experience a \textit{Tachyonic instability} during the reheating phase. Such instability causes a substantial growth in the scalar field amplitude leading to pronounced production of SGWs in the low and intermediate-frequency ranges that are strong enough to be detected by Planck and future gravitational wave detectors. Such growth in super-horizon modes of the scalar field and associated GW production may have a significant effect on the strength of the tensor fluctuation at the Cosmic Microwave Background (CMB) scales (parametrized by $r$) and the number of relativistic degrees of freedom (parametrized by $\dneff$) at the time of CMB decoupling. To prevent such overproduction, the PLANCK constraints on tensor-to-scalar ratio $r \leq 0.036$ and $\dneff \leq 0.284$ yield a strong lower bound on $\xi$ for $\wre < 1/3$, and upper bound on the value of $\xi$ for $\wre > 1/3$. Taking into account all the observational constraints we found the value of $\xi$ should be $\gtrsim 0.02$ for $\wre =0$, and $\lesssim 4.0$ for $\wre \geq 1/2$ for a wide range of reheating temperature within $10^{-2} \lesssim \Tre \lesssim 10^{14}$ GeV, and for a wide range of inflationary energy scales. Further, as one approaches $\wre$ towards $1/3$, the value of $\xi$ remains unconstrained. Finally, we identify the parameter regions in $(\Tre,\xi)$ plane which can be probed by the upcoming GW experiments namely BBO, DECIGO, LISA, and ET.