Oversight, scholars argue, allows Congress to control the executive agents it empowers to implement law. Yet the tools of oversight are rather limited and debate continues as to how much influence oversight provides. How well can members of Congress motivate bureaucratic performance? To measure the efficacy of oversight, we create a new dataset matching oversight efforts to a bureaucratic deficiency Congress has sought to reduce since the early 2000s: improperly made payments to contractors and clients. We estimate the effect of congressional hearings, one of the most important tools of congressional oversight, as well as correspondence, appropriation committee reports, statutes, and executive action. We find that hearings lead to subsequent declines in improper payments. The magnitude of the effect, however, is small relative to the scope of the problem, suggesting strong limits on the efficacy of oversight. Our findings imply that America’s elected officials struggle to effectively manage implementation of policy.