The hypothesis of the small scales being passively swept along by the large-scale motions in turbulent flow is extended to passive scalars in isotropic turbulence. A theory based on strong mutual cancellation between local and advective derivatives and other assumptions is shown to capture the Reynolds and Schmidt number dependence of time scales characterizing Eulerian and Lagrangian rates of change. Agreement with direct numerical simulation data improves systematically with increasing Reynolds number. In accordance with the physics of random sweeping, the Eulerian frequency spectrum is very similar in shape to the wavenumber spectrum, but is broadened at higher frequencies compared to its Lagrangian counterpart. Overall the hypothesis appears to be even more valid for transported scalars than for the velocity field, which gives support to the use of Lagrangian approaches in the study of turbulent mixing.