The instability of a rotating-magnetic-field-driven liquid metal flow in a finite cylinder with respect to infinitesimal azimuthally periodic perturbations is studied numerically. This instability is observed to set in prior its axisymmetric counterpart with relatively low frequency at diameter-to-height ratios between 0.5 and 2. The axisymmetric and three-dimensional instabilities have similar characteristic features. The instability originates in the cross-section of the horizontal and vertical rotating boundary layers and excites inertial waves in the inviscid core.