This study introduces an improved gesture-based human–swarm interaction control system for drone swarms that unifies motion and formation control while addressing the critical challenge of neglecting the impact of faulty robots on task performance. Unlike conventional approaches that treat these functionalities separately, our integrated framework enables robust task execution in challenging environments. Using 26 hand gestures (18 formations and 8 flight commands) processed by a variety of deep learning models, including convolutional neural networks (e.g., ResNet101 and MobileNetV2) and recurrent neural network-based approaches such as long short-term memory (LSTM), bidirectional LSTM, gated recurrent unit (GRU), and bidirectional GRU (BiGRU), the system provides comprehensive yet accessible control, notably through the BiGRU network with 99.4% accuracy in real-time gesture recognition tasks. Deployment testing conducted using the CrazyFlyt simulation platform demonstrated the system’s robustness in maintaining user-intended formations with low positional error and stable convergence times, even in the presence of sensor noise and faulty robots. Statistical analysis revealed highly significant main and interaction effects ($p\lt 0.05$) for swarm scale and scenario on performance metrics, indicating that the impact of swarm size is dynamically contingent upon environmental conditions. Post hoc evaluations predominantly highlighted significant performance degradation in faulty-robot scenarios compared to normal or noisy conditions, with swarm scale critically modulating these effects.