On Instability and Stability of Three-Dimensional Gravity Driven Viscous Flows in a Bounded Domain

We investigate the instability and stability of some steady-states of a three-dimensional nonhomogeneous incompressible viscous flow driven by gravity in a bounded domain $\Omega$ of class $C^2$. When the steady density is heavier with increasing height (i.e., the Rayleigh-Taylor steady-state), we show that the steady-state is linear unstable (i.e., the linear solution grows in time in $H^2$) by constructing a (standard) energy functional and exploiting the modified variational method. Then, by introducing a new energy functional and using a careful bootstrap argument, we further show that the steady-state is nonlinear unstable in the sense of Hadamard. When the steady density is lighter with increasing height, we show, with the help of a restricted condition imposed on steady density, that the steady-state is linearly globally stable and nonlinearly locally stable in the sense of Hadamard.