Thermodynamic stability, compressibility matrix, and effects of mediated interactions in a strongly-interacting Bose-Fermi mixture

We theoretically investigate the thermodynamic stability of a normal-state Bose-Fermi mixture, with a tunable Bose-Fermi pairing interaction $-U_{\rm BF}<0$ associated with a hetero-nuclear Feshbach resonance, as well as a weak repulsive Bose-Bose interaction $U_{\rm BB}\ge 0$. Including strong hetero-pairing fluctuations associated with the former interaction within the self-consistent $T$-matrix approximation, as well as the latter within the mean-field level, we calculate the compressibility matrix, to assess the stability of this system against density fluctuations. In the weak- and the intermediate-coupling regime with respect $-U_{\rm BF}$, we show that an effective attractive interaction between bosons mediated by density fluctuations in the Fermi component makes the system unstable below a certain temperature $T_{\rm clp}$ (leading to density collapse). When $U_{\rm BB}=0$, $T_{\rm clp}$ is always higher than the Bose-Einstein condensation (BEC) temperature $T_{\rm c}$. When $U_{\rm BB}>0$, the density collapse is suppressed, and the BEC transition becomes possible. It is also suppressed by the formation of tightly bound Bose-Fermi molecules when the hetero-pairing interaction $-U_{\rm BF}$ is strong; however, since the system may be viewed as a molecular Fermi gas in this case, the BEC transition does not also occur. Since quantum gases involving Bose atoms are known to be sensitive to inter-particle correlations, our results would be useful for the study of many-body properties of a Bose-Fermi mixture in a stable manner, without facing the unwanted density collapse.