The bistable fluttering response of heavy inverted flags with different aspect ratios ($AR$) is investigated to determine how the vortical structures affect the intermittent vibration response of the flag. A heavy inverted flag in a uniform flow may exhibit several response modes; amongst them are three major modes that occur over an extended velocity range: stationary, large-scale periodic oscillation and one-sided deflected modes. Significant hysteretic bistability is observed at the transition between these modes for all $AR$, which is notably different from the conventional flag vibration with a fixed leading edge and free trailing edge where no hysteresis is observed at the lower $AR$ limit ($AR<1$). The difference is associated with the distinct roles of vortices around the flag. Experiments with flags made of spring steel are conducted in a wind tunnel, where the flow speed is steadily increased and later decreased to obtain different oscillatory modes of the heavy inverted flags. The experimental results are used to validate the numerical model of the same problem. It is found that different critical velocities exist for increasing and decreasing flow velocities, and there is a sustained hysteresis for all $AR$ controlled by the initiation threshold and growth of the leading-edge and side-edge vortices. The effect of the vortices in the bistable oscillation regime is quantified by formulating a modal force partitioning approach. It is shown that $AR$ can significantly alter the static and dynamic vortex interaction with the flexible plate, thereby changing the flag's hysteresis behaviour and bistable response.