This paper presents a fractional-order sliding mode control scheme equipped with a disturbance observer for robust impedance control of a single-link flexible robot arm when it comes into contact with an unknown environment. In this research, the impedance control problem is studied for both unconstrained and constrained maneuvers. The proposed control strategy is robust with respect to the changes of the environment parameters (such as stiffness and damping coefficient), the unknown Coulomb friction disturbances, payload, and viscous friction variations. The proposed control scheme is also valid for both unconstrained and constrained motions. Our novel approach automatically switches from the free to the constrained motion mode using a simple algorithm of contact detection. In this regard, an impedance control scheme is proposed with the inner loop position control. This means that in the free motion, the applied force to the environment is zero and the reference trajectory for the inner loop position control is the desired trajectory. However, in the constrained motion the reference trajectory for the inner loop is determined by the desired impedance dynamics. Stability of the closed loop control system is proved by Lyapunov theory. Several numerical simulations are carried out to indicate the capability and the effectiveness of the proposed control scheme.