In this paper, we study the fluid–structure interaction of a three-dimensional (3-D) flexible membrane immersed in an unsteady separated flow at moderate Reynolds numbers. We employ a body-conforming variational fluid–structure interaction solver based on the recently developed partitioned iterative scheme for the coupling of turbulent fluid flow with nonlinear structural dynamics. Of particular interest is to understand the flow-excited instability of a 3-D flexible membrane as a function of the non-dimensional mass ratio ($m^{*}$), Reynolds number ($Re$) and aeroelastic number ($Ae$). For a wide range of parameters, we examine two distinct stability regimes of the fluid–membrane interaction: deformed steady state (DSS) and dynamic balance state (DBS). We propose stability phase diagrams to demarcate the DSS and DBS regimes for the parameter space of mass ratio versus Reynolds number ($m^{*}$-$Re$) and mass ratio versus aeroelastic number ($m^{*}$-$Ae$). With the aid of the global Fourier mode decomposition technique, the distinct dominant vibrational modes are identified from the intertwined membrane responses in the parameter space of $m^{*}$-$Re$ and $m^{*}$-$Ae$. Compared to the deformed steady membrane, the flow-excited vibration produces relatively longer attached leading-edge vortices which improve the aerodynamic performance when the coupled system is near the flow-excited instability boundary. The optimal aerodynamic performance is achieved for lighter membranes with higher $Re$ and larger flexibility. Based on the global aeroelastic mode analysis, we observe a frequency lock-in phenomenon between the vortex-shedding frequency and the membrane vibration frequency causing self-sustained vibrations in the dynamic balance state. To characterize the origin of the frequency lock-in, we propose an approximate analytical formula for the nonlinear natural frequency by considering the added mass effect and employing a large deflection theory for a simply supported rectangular membrane. Through our systematic high-fidelity numerical investigation, we find that the onset of the membrane vibration and the mode transition has a direct dependence on the frequency lock-in between the natural frequency of the tensioned membrane and the vortex-shedding frequency or its harmonics. These findings on the fluid-elastic instability of membranes have implications for the design and development of control strategies for membrane wing-based unmanned systems and drones.