Safe and accurate navigation for autonomous trajectory tracking of quadrotors using monocular vision is addressed in this paper. A second order Sliding Mode (2-SM) control algorithm is used to track desired trajectories, providing robustness against model uncertainties and external perturbations. The time-scale separation of the translational and rotational dynamics allows to design position controllers by giving a desired reference in roll and pitch angles, which is suitable for practical validation in quad-rotors equipped with an internal attitude controller. A Lyapunov based analysis proved the closed-loop stability of the system despite the presence of unknown external perturbations. Monocular vision fused with inertial measurements are used to estimate the vehicle's pose with respect to unstructured scenes. In addition, the distance to potential collisions is detected and computed using the sparse depth map coming also from the vision algorithm. The proposed strategy is successfully tested in real-time experiments, using a low-cost commercial quadrotor.