It is known that for finite Rossby numbers geostrophically balanced flows develop specific ageostrophic instabilities. We undertake a detailed study of the Rossby–Kelvin (RK) instability, previously studied by Sakai (J. Fluid Mech., vol. 202, 1989, pp. 149–176) in a two-layer rotating shallow-water model. First, we benchmark our method by reproducing the linear stability results obtained by Sakai (1989) and extend them to more general configurations. Second, in order to determine the relevance of RK instability in more realistic flows, simulations of the evolution of a front in a continuously stratified fluid are carried out. They confirm the presence of RK instability with characteristics comparable to those found in the two-layer case. Finally, these simulations are used to study the nonlinear saturation of the RK modes. It is shown that saturation is achieved through the development of small-scale instabilities along the front which modify the mean flow so as to stabilize the RK mode. Remarkably, the developing instability leads to conversion of kinetic energy of the basic flow to potential energy, contrary to classical baroclinic instability.