Since the nineteenth century, Western societies have witnessed a dramatic expansion in the scope of state authority. Influenced by the emergence of industrial capitalism, popular reactions to economic crises, and the resource demands of war, contemporary states exercise wide-ranging regulatory powers that are unique in the history of human governance.Footnote 1 Given the scope and complexity of their actions, modern states have adopted a specific organizational form, using administrative bureaucracies to draft, implement, and adjudicate policy decisions.
However, contemporary administrative states are not merely distinguished by their bureaucratic organizational form; they also exhibit unique sources of legitimacy. While elected officials derive their authority from the consent of the governed, a long tradition in the social sciences argues that expert administrators are legitimized by the technical efficiency of their decisions.Footnote 2 Indeed, Max Weber famously argued that the triumph of legal-rational bureaucracies was due to their superior efficiency relative to rival forms of organization, and his pessimism regarding modern politics focused on the diminished prospects for individual freedom in the face of rationalized bureaucratic power.Footnote 3
Some, however, have criticized bureaucratic administrative authority for imposing instrumental rationality into social spheres that previously stood outside the realm of calculation and reason. Theorists such as Hannah Arendt and Jürgen Habermas have assumed that administrative states are instrumentally rational, and have focused on critiquing instrumental rationality as a form of rationality.Footnote 4 Such critics often accept the premise that administrative states successfully adopt the most efficient policies to achieve socially determined ends, and have focused their critiques on whether the administrative state’s expansion of instrumental rationality has come at the expense of other ends.Footnote 5 Indeed, public choice and regulatory capture theorists typically assume that regulatory failure is due to suboptimal incentives rather than information problems.Footnote 6
Given their wide-ranging powers, assessing the administrative state’s capacity for efficient action is a central question for contemporary politics and for our general understanding of the ongoing process of political rationalization.Footnote 7 This essay examines administrative states’ efforts to rationally regulate their societies, and identifies a general epistemic problem facing efficient administrative action. I argue that a fundamental feature of administrative states’ authority, namely the singular nature of their regulatory decisions for all members of society, creates barriers to efficient administrative action that exist independently of the personnel who staff the administrative state and the interests they serve.
Instead of emphasizing the incentives facing administrative actors, their policies’ dispersed costs and concentrated benefits, the advantages of organized interests over the wider society, or administrators’ self-interest, I examine how the singularity of administrative states’ regulatory decisions prevents observation of the counterfactual effects of policies that were possible, but were not implemented.Footnote 8 I argue that the unobservable nature of policy counterfactuals makes it difficult to assess the causal effects of administrative decisions, as the social conditions that would have emerged absent the administrative state’s actions cannot be observed. However, the absence of observable policy counterfactuals also frustrates efforts to assess the efficiency of administrative states’ decisions, as it is difficult to know whether policies that were possible, but which were not implemented, would have produced better social conditions at lower cost.
In an effort to clarify the nature of the epistemic problems facing administrative states, the rest of this essay is organized as follows. Section I provides my conceptual definitions and isolates the principal feature of administrative authority that I examine, namely the singularity of administrative decisions. This section also discusses the Weberian justification for legal-rational administrative legitimacy, namely the efficiency of bureaucratic rule making, and discusses how administrative states are distinguished from alternative forms of bureaucratic governance.
Section II introduces arguments regarding causal inference and experimental knowledge relevant for assessing administrative efficiency. This section focuses on the role of counterfactual knowledge in experimental research design, and argues that the inferential problems experiments are designed to overcome have implications for theories of organizational behavior as well. Specifically, I argue that the availability of opportunities to observe policy counterfactuals has implications for our ability to make accurate causal inferences regarding the consequences of organizations’ decisions.
Section III deploys arguments regarding experiments and counterfactuals to identify a general problem confronting efficient administrative action. This section examines how the singularity of administrative states’ decisions prevents observation of counterfactual social conditions that would have emerged absent the administrative state’s actions. Absent such counterfactual knowledge, it is difficult to assess whether regulatory policies’ costs and benefits are efficient or not, or whether other policies would have produced greater benefits at lower cost.
Section IV illustrates this argument by discussing the challenges facing the Food and Drug Administration’s (FDA) efforts to efficiently regulate drug safety. I focus on the Prescription Drug User Fee Act (PDUFA), a program allowing drug companies to pay fees to the FDA in exchange for guarantees that the FDA will reach a decision on their new drug applications in a given amount of time. Instead of examining the FDA’s incentives, or whether the FDA is captured by organized interests, I focus on identifying the inferential problems facing efforts to assess the efficiency of the FDA’s decisions.
I conclude with a general discussion of the epistemic problems facing administrative decision-making. I argue that although administrative actors can observe whether policy objectives are met, it is difficult to know whether such objectives were achieved efficiently. As this problem is caused by the singularity of administrative states’ decisions, and the corresponding problems this feature of politics creates for the generation of counterfactual knowledge, barriers to efficient administrative action exist independently of administrators’ preferences, the interests they pursue, or the public’s degree of involvement in their decisions.
I. Definitions and Assumptions
Prior to analyzing the problems facing the administrative state’s decisions, it is necessary to define the “administrative state.”Footnote 9 As Weber noted, defining the unique features of an organization as complex as the modern state constitutes “by far the most complicated and most interesting” challenge confronting clear conceptual definitions.Footnote 10 Hence I do not attempt to offer a general definition of the state, but instead focus on certain features of state decision-making and the implications of these features for political knowledge.Footnote 11
The administrative state constitutes the policy-making bureaucracy of modern governments that are staffed by appointed civil servants who enjoy tenure of office and are isolated from democratic elections and public opinion. However, administrative states can be distinguished from bureaucratic states, of which it is a specific type. While states have always employed bureaucracies to carry out their decisions, these bureaucracies have taken different forms (for instance, patrimonial, legal-rational), and have produced different types of goods (for instance, public or private goods).
Administrative states are conceptually distinguished from other public bureaucracies by the type of goods they produce. Specifically, I define administrative states as bureaucracies engaged in the production of singular and legally binding rules for all members of a geographically defined society. Thus instead of directly producing private goods for their societies, such as postal services or health care, administrative states produce rules governing the production, distribution, and consumption of goods by private actors. For example, although the United States Postal Service (USPS) is a public bureaucracy, it produces a good, mail delivery, which society is not legally required to consume, and faces direct competition from private firms (FedEx, UPS, DHL) whose services can be used as substitutes for those the USPS produces. Hence, while the USPS is a public bureaucracy, it is not part of the “administrative state” that I analyze.
The singularity of administrative states’ regulatory decisions describes the fact that administrative states face no rival organization with the authority to produce legally binding rules that are compulsory for all members of its geographically defined territory. For example, administrative states monopolize control over tax policy, and there is no rival organization whose tax regulations can be used as an alternative to those the administrative state imposes on its citizens.Footnote 12 Hence, if an American citizen is displeased with the IRS’s tax code, there is no rival organization whose taxation regulations they can use as a substitute for those the IRS produces. In this sense the IRS produces singular tax rules that apply to all Americans living within the jurisdiction it claims sovereignty over.
The singularity of the administrative state’s decisions distinguishes its authority from social organizations that lack singular decision authority over a geographically defined territory. Hence, if consumers are displeased with a private firm’s product, they can purchase a different product from a rival firm without having to physically move to another location. Conversely, if citizens are displeased with their administrative state’s policies, they cannot use a rival organization’s regulations as a substitute for those the state implements, but must abide by the administrative state’s decisions or try to change the singular decision the administrative state imposes.
While the singularity of such decisions prevents citizens from “exiting” from administrative states’ decisions, I focus on the epistemic implications of this feature of state authority.Footnote 13 I do not examine citizens’ opportunities for exercising voice, nor do I examine the epistemic advantages associated with incorporating diverse perspectives into deliberative political bodies. I ignore these issues because the singularity of administrative states’ decisions is a feature of politics that exists independently of the number of actors involved in the decision-making process, or whether or not such decisions are reached through open and fair deliberation.Footnote 14 Thus, even if administrative states are open to social actors being involved in reaching its regulatory decisions, perhaps by directly involving either citizens or social groups in its deliberations, such decisions remain singular once the actors involved have agreed to them.Footnote 15
I use several unrealistic assumptions to demonstrate that the singularity of political decisions creates specific epistemic problems that persist even absent other problems. I assume that administrative actors are solely motivated by the public interest, are not subject to principal-agent problems, and are immune from corruption or regulatory capture. As administrators may be self-interested, captured, and corrupt, these assumptions are unrealistic; I make these assumptions to demonstrate that the challenges facing efficient administrative decision-making will merely become more severe if greater degrees of realism are introduced.
While many political decisions are singular and lack rivals, a distinguishing feature of administrative states’ decisions is that they are legitimized by their instrumental rationality. The rational basis of administrative legitimacy is typically attributed to Weber, who argued that “the decisive reason for the advance of bureaucratic organization has always been its purely technical superiority over any other form of organization.”Footnote 16 Instrumental rationality is defined here in Weberian terms, as actions whereby “the end, the means, and the secondary results are all rationally taken into account and weighed. This involves rational consideration of alternative means to the end, of the relations of the end to the secondary consequences, and finally of the relative importance of different possible ends.”Footnote 17
While various normative standards may legitimize political action, I assume administrative actors are legitimized by the technical efficiency of their decisions. Indeed, the separation of political and administrative decisions, and the corresponding assumption that administrative personnel possess unique forms of expertise, is typically considered a hallmark of administrative authority.Footnote 18 While administrative states may receive democratic authorization from voters and elected representatives, their authority is distinguished from elected representatives by the emphasis on the technical efficiency of their decisions, and not by their correspondence with electoral preferences and public opinion. This makes the analysis of administrative bureaucrats simpler than other political actors whose authority is primarily legitimated by their democratic credentials. While many question whether politics and administration can be clearly distinguished, I assume political and administrative questions can be clearly separated, and that administrators simply seek policies that are efficient means to politically determined ends.Footnote 19
II. Administration and Counterfactual Knowledge
Assuming administrative states are legitimized by the instrumental rationality of their decisions, that is, their ability, relative to other forms of government, to adopt efficient means to politically determined ends, what challenges confront their efforts to engage in efficient regulatory action? This general question confronts all objectives that administrative states pursue; even if administrative states seek to promote egalitarian ends, or non-economic objectives, such as environmental protection, we are generally interested in whether such ends are produced efficiently, that is, at minimal cost, or whether alternative policies would generate, for example, greater amounts of equality at lower cost relative to the alternative policies.
Administrative states’ regulatory failings have often been attributed to the objectives or interests that administrative actors pursue. Hence, public choice and regulatory capture theorists argue that bureaucrats’ self-interest, and patterns in the concentrated benefits and dispersed costs of regulatory decisions, cause administrative actors to deviate from the public interest.Footnote 20 However, these explanations’ descriptions of bureaucrats’ self-interest, the causes and frequency of regulatory capture, and the costs and benefits that their policies generate, have been contested on empirical and theoretical grounds.Footnote 21
Instead of focusing on whether bureaucrats are self-interested or captured, or examining policies’ costs and benefits, I focus on how a general characteristic of administrative states’ decisions, their singularity, limits the production of counterfactual knowledge necessary to assess their efficiency. Given that many of the decisions of modern states are singular and lack rivals, the argument identifies a general problem that is not derived from bureaucrats’ self-interest: their tendency to be captured or corrupt, which also exists independently of the dispersion of their policies’ costs and benefits. As I attribute specific problems to the way administrative states limit the generation of counterfactual knowledge, it is necessary to discuss certain features of experimental research and causal inference.
III. Administrative Decision-Making and the Fundamental Problem of Causal Inference
While such issues are normally the subject of purely methodological considerations and may appear unrelated to questions of organizational theory, the challenges facing administrative states’ efforts to implement efficient policies resemble a general methodological problem confronting the social sciences. Just as administrative states seek to identify policies that will efficiently produce social conditions, social scientists are interested in identifying the independent variables responsible for causing specific outcomes, such as democratization or economic development. In both cases, social scientists and administrators are interested in causal questions regarding how a policy or variable will influence a social condition or outcome, such as whether certain policies influence unemployment or crime rates, or have implications for economic development.
However, given that an infinite number of independent variables may be correlated with fluctuations in a dependent variable, it is often difficult for social scientists to generate the variation necessary to make inferences regarding causal relationships among variables. Social scientists have increasingly used experimental research designs to generate valid causal inferences regarding the relationships among independent and dependent variables.Footnote 22 Experiments use an untreated control group to mimic the counterfactual conditions that exist absent a treatment condition, and randomized assignment of subjects to treatment and control groups is used to control for confounding variables.
However, experiments confront a basic inferential problem regarding treatment condition’s causal effects upon any individual subject (or unit) in an experiment. Paul Holland and others have argued that “The Fundamental Problem of Causal Inference” confronts efforts to observe the counterfactual behavior of individual experimental subjects, as it is impossible for individual participants in experiments to be exposed to both treatment and control conditions at the same time.Footnote 23
Thus, for example, if an experiment seeks to determine a new curriculum’s effect upon students’ test scores, it is impossible to assign individual students to both treatment and control groups. Individual students can only be exposed to either the treatment condition (that is, the new curriculum) or the control condition, the old curriculum. Since individual students can only be assigned to either the treatment or control group, it is impossible to observe the new curriculum’s causal effect on any individual student’s test performance because we cannot observe the test scores the student would have earned had the student not been assigned to the treatment condition, that is, the new curriculum. Thus we only observe the test scores earned by individual students assigned to either of the two experimental conditions, and we cannot recover the counterfactual scores they would have earned had they, for example, not been exposed to the treatment condition. This problem makes it difficult to observe how much the new curriculum influenced individual students’ test scores, as we simply cannot know what their scores would have been had they not been exposed to the new curriculum.
Experiments overcome the fundamental problem of causal inference by using a control group to simulate the counterfactual world that would have existed absent individual’s assignment to the treatment condition. If random assignment is used to eliminate the influence of confounding variables, experiments generate valid causal inferences regarding the treatment condition’s average causal effect relative to the control group.Footnote 24 The use of a control group thus helps the experiment overcome the inferential problems that the fundamental problem of causal inference creates for reaching valid inferences regarding the treatment’s causal effect on individual experimental subjects.
Although rarely discussed outside the context of research design methodology, the fundament problem of causal inference has implications for any organization possessing monopoly power.Footnote 25 Specifically, administrative states’ monopoly over the production of singular rules for a geographically defined society reproduces the fundamental problem of causal inference in empirical reality because the singularity of the administrative state’s regulatory decisions eliminates observable policy counterfactuals within the territory the state controls.
Given the absence of any rival organizations whose decisions can be used as substitutes for those the administrative state makes, it is impossible to observe the social conditions that would have been produced by regulatory decisions that the state did not impose. Societies merely observe the policy the state implemented and the social conditions that exist following this decision; it is impossible to observe what social conditions would have emerged had the state implemented a regulatory decision other than the one it implemented.
This condition exists regardless of the end the state seeks to achieve. Hence, for example, if the IRS alters the tax code in an effort to increase the level of economic equality in the United States, it is impossible to observe the level of inequality that would have emerged absent the IRS’s decision; we merely observe the rule change and the subsequent level of equality that exists after this change; it is impossible to determine whether more equality would have been created with a different change to the tax code, or whether the amount of equality that emerged following the rule change was caused by some exogenous change that was correlated with the IRS’s decision.
Unfortunately, the technique that experiments use to overcome the fundamental problem of causal inference, namely the creation of an untreated counterfactual control group that is identical to the treatment group on all covariates except exposure to the treatment condition, cannot be used in the administrative state’s policy decisions. Since states impose singular and legally binding decisions upon their societies, it impossible to observe an “untreated” society that was not subjected to the administrative state’s policy decisions. Rather, administrative states are administering policy “treatments” to their society, but the singular nature of their decisions prevents observation of the counterfactual social conditions that would have emerged absent their actions.
Hence, for example, when a central bank changes interest rates it is impossible to observe the economic conditions that would have emerged absent the central bank’s action. The singularity of such decisions mirrors our inability to observe a treatment’s causal effect on individual experimental subjects, as, in the context of regulatory politics, it becomes difficult to identify policy decisions’ causal effects because there is no means of observing the counterfactual social world that would have existed absent the state’s actions.
The singularity of administrative states’ decisions is fundamentally dissimilar to social organizations that face competition from rival organizations within the same geographic territory. This distinction is not due to the different motivations or incentives that exist in different realms of human action, but is caused by the different forms of counterfactual knowledge generated by social and political competition. Perhaps the clearest example of this distinction is the competitive market where multiple firms generate observable counterfactual data regarding economic performance through the simultaneous comparisons consumers can make of their products.Footnote 26
This distinction does not require an idealized conception of markets, which, in empirical reality, are often dominated by a small number of powerful firms. Indeed, the argument offers a critique of noncompetitive markets due to the reduced number of counterfactual comparisons that can be performed when a small number of powerful firms dominate a market. In this sense, the argument is applicable beyond politics, as it offers a basis for critiquing economic monopoly that is independent of the elevated prices and corresponding deadweight social welfare losses that are the normal focus of neoclassical economic theory. However, if one accepts that noncompetitive markets harmfully limit the production of counterfactual data, this condition is merely magnified in settings where an organization monopolizes the production of certain goods over a fixed geographic territory, as is often the default condition with modern states’ decision-making.
Thus, while competitive economic markets facilitate comparisons of firms’ performance in ways that mimic scientific experiments’ use of treatment and control groups to identify causal relationships, the singularity of administrative states’ decisions reproduces the fundamental problem of causal inference in the realm of public policy. Just as scientists cannot observe a treatment’s causal effect on individual subjects, the unobservable counterfactual social conditions that would have existed if the state had not acted makes it difficult to observe the causal effects administrative states’ policies have on their society.
It is important to emphasize that this argument recognizes that different political decisions can be compared, for example, by using state and local governments in federal systems, by observing nation-states that exhibit different degrees of centralization, or by using the history of prior policies, or variation in when a policy is implemented, to generate counterfactuals. The principal problem with such comparisons is that the political units used to generate the counterfactual, that is, cities or other nations, are not identical on all covariates except exposure to the policy “treatment.” Thus, while comparisons can be employed to support specific causal claims, it is difficult to rule out confounding variables with the same level of certainty that is possible with actual scientific experiments.
Furthermore, once one recognizes that comparisons across nations and societies reveal some information, one has accepted the basic premise of this essay’s argument. Specifically, the variation necessary for such comparisons to reveal knowledge about causality is eliminated by the singularity of the state’s decisions within its territory, because the homogeneous application of regulatory rules for all members of a society eliminates the variation that, for example, federalism exhibits at the local level.
Additionally, although nation-states can be compared, differences in their societies frustrate efforts to draw valid inferences from such comparisons. Thus, while we can observe that an economic crisis emerged in one country that may exhibit a more centralized political system, but not in a society with a less powerful centralized state, the large number of economic variables that may be responsible for this outcome makes it difficult to know how specific regulatory decisions may have contributed to such outcomes. Given that nations exhibit so many differences in both their political systems and in their economies, it is difficult to isolate how specific policy decisions caused the outcomes we observe, or whether such differences were due to social or economic variables that are unrelated to the society’s political system.
While the singularity of the administrative state’s policy decisions reproduces the fundamental problem of causal inference in ways that frustrate observation of regulatory decisions’ causal effects, this feature of administrative states’ decisions creates additional problems with assessing the efficiency of administrative states’ decisions. Specifically, the absence of policy counterfactuals ensures that the costs and benefits of regulations that were not implemented cannot be observed. The absence of such data makes it difficult to determine whether a policy is efficient or not, as it is impossible to observe whether rival policies would have generated greater benefits and lower costs than the policy the state implemented.
Thus, if a central bank responds to a recession by lowering interest rates, this decision is subsequently correlated with a given level of economic growth, inflation, and unemployment. While these economic conditions can be measured and quantified, we cannot observe what would have happened had the central bank lowered interest rates even further, held them constant, or taken some other action entirely. While these unobservable counterfactual scenarios frustrate our ability to identify the causal effects of central banking decisions, they create even more difficult problems for assessing their efficiency, as it is unclear whether the economic conditions emerged following the central bank’s intervention generated the highest benefits and lowest costs, or whether better economic conditions could have been produced with a different decision. While administrative actors can attempt to estimate the costs and benefits of different policy decisions, it is simply difficult to evaluate the accuracy of these estimates for policies that are not implemented.
Insofar as the inferential problems caused by the singularity of administrative states’ decisions frustrate efforts to assess the efficiency of their policies, barriers to efficient regulatory action exist that are independent of administrative actors’ motives, the interests they serve, the costs and benefits of the goods they produce, or the public’s degree of involvement in their decisions. Although these epistemic problems are particularly problematic for administrative actors whose authority is legitimated by the instrumental rationality of their decisions, this problem appears to confront any political decision that is singular and lacks rivals within a defined geographic territory.Footnote 27
IV. Application to the FDA
To illustrate the epistemic problems facing efficient administrative action, this section examines the Food and Drug Administration’s (FDA) regulation of drug safety, and specifically focuses on the FDA’s Prescription Drug User Fee Act (PDUFA), a regulatory program that charges companies fees in exchange for guarantees that their new drug applications will be evaluated by the FDA within a fixed period of time. Standard criticisms of the FDA suggest that it has low incentives for completing quick safety reviews, and is subject to capture from the funding it receives from the industry it regulates. While such problems may certainly exist, I focus on how the difficulty in conducting counterfactual comparisons of regulatory rules creates challenges for assessing the FDA.
In its efforts to efficiently regulate drugs, the FDA monopolizes production of regulatory rules that are binding for all members of American society. Thus, if citizens wish to consume drugs that have not been approved by the FDA, or if pharmaceutical companies are dissatisfied with the FDA’s regulatory requirements, they are legally prevented from using a rival organization’s rules as substitutes for the FDA’s. It is in this sense that the FDA’s authority generates singular rules that lack rivals for all members of American society.
The FDA faces a basic trade-off between the speed with which new drugs are evaluated and its error rate in either mistakenly approving drugs that are unsafe, or in delaying approval of drugs that are safe and effective. If the FDA is too cautious, it will unnecessarily delay patients from consuming drugs they need, and reduce pharmaceutical companies’ profits. If it is too hasty, it may approve unsafe drugs that harm consumers. Thus the FDA seeks to implement regulations that balance accuracy with speed of approval to maximize the benefits both to drug consumers and to producers, at minimum cost.
The FDA stipulates that any pharmaceutical company wishing to bring a new drug to market must conduct three-stage experimental trials. Phase I trials evaluate a drug’s safety with small numbers of healthy volunteers, Phase II uses a slightly larger number of sick volunteers to establish a drug’s efficacy and determine what constitutes a safe dosage of the drug to treat a disease, Phase III enrolls larger numbers of sick patients to try to reach definitive conclusions regarding efficacy and appropriate drug dosages.Footnote 28 After an application reporting the results of clinical trials is sent to the FDA, the FDA either approves or denies the company’s application to sell the drug on the market.
In the 1970s and 1980s there were no restrictions on how long the FDA could take to test drug safety, and the losses to profits resulting from the FDA’s lengthy review times caused many new drugs to be initially released outside the United States.Footnote 29 In response to criticisms that the FDA was taking too long to approve drugs during the AIDS crisis, in 1992 the FDA adopted the Prescription Drug User Fee Act (PDUFA) to expedite drug testing and increase the number of staff reviewing new drug submissions.Footnote 30 PDUFA requires the FDA to review new drug applications within a fixed period of time in exchange for fees pharmaceutical companies pay the FDA.Footnote 31 PDUFA I (1992–1997) established that most new drug applications carry a $100,000 fee, and additional annual fees of $60,000 and $6,000 were charged for each manufacturing facility and manufactured drug, respectively, in order to smooth fluctuations in funding.Footnote 32 By 2001 the new drug application fee rose to $310,000, and user fees equaled 13 percent of the FDA’s budget appropriation from Congress, and 50 percent of FDA spending on drug review activities.Footnote 33
The PDUFA requires the FDA to issue decisions on 90 percent of new drug applications within either six or ten months, depending on whether applications are granted “priority” or “standard” status.Footnote 34 Once a drug is submitted to the FDA, a review clock is initiated and if the FDA fails to review the application on time, it loses the user fee.Footnote 35 In an effort to shorten drug review times, the FDA used PDUFA funding to increase the number of personnel evaluating drug safety. In 1993, the FDA employed 1,408 full time employees in the Center for Drug Evaluation and Research (CDER) to review new drug applications, by the year 2000 this number had risen to 1,780.Footnote 36 In 1993, a year after passage of PDUFA I, the FDA collected $29 million in annual user fees; by 2018 this figure had increased to $908 million.Footnote 37
The combination of Phase I–III testing, and the use of PDUFA to incentivize timely FDA review of new drug applications, creates an average cost and approval time to develop new drugs, which are correlated with a given error rate for drugs that were approved but were subsequently removed from the market due to adverse health effects. Under the PDUFA regulatory regime the FDA rarely approves drugs that wind up being unsafe, only about 2.5 percent of drugs approved for the market are subsequently withdrawn, while the average time for a drug to be developed and then released to the market is between 8 to 12 years and costs approximately $802 million.Footnote 38
Taken together, Phase I–III testing and PDUFA constitute a set of regulatory standards governing all new drugs that pharmaceutical companies seek to market within the United States. If the singular nature of the FDAs decisions creates epistemic problems with assessing their effects, there should be difficulties in assessing both the causal effects of PDUFA, and the efficiency of the FDA’s regulation of drug safety. I address each of these below, and specifically focus on the challenges facing efforts to assess the efficiency of PDUFA’s effects on consumer welfare.
In assessing PDUFA’s causal effects, most statistical analyses make comparisons between drugs approved before and after passage of PDUFA in 1992, and use the historical period prior to PDUFA’s passage as the counterfactual. Generally, PDUFA is believed to have shortened the length of time it requires for most drugs to be evaluated, and expedited the release of new drugs to the market.Footnote 39 Prior to PDUFA I drug approval times fell about 2 percent annually, while after passage of PDUFA I and II, review times fell by 6–7 percent, and 3–4 percent, respectively.Footnote 40 Given that PDUFA mandates that the FDA must meet specific review time targets, and has doubled the number of FDA staff reviewing new drug applications, these findings are not surprising.
While the decrease in review times following PDUFA’s passage is apparent, there are inferential problems associated with using the drugs approved prior to PDUFA’s passage in 1992 as the comparison group to evaluate drug approvals after PDUFA’s passage. Specifically, the drugs submitted to the FDA prior to, and then after PDUFA, are not identical across all other characteristics, but reflect differences in medical knowledge, learning from prior drug trials and experiences, as well as the development of new ideas and technologies for treating diseases, the understanding of which is itself subject to change.
Such changes frustrate interpretation of how PDUFA has influenced the quality of the FDA’s regulatory decisions. For example, some argue that the number of drug recalls has increased after PDUFA’s passage, while others argue that the overall number of drug recalls is not statistically different between the two regulatory periods.Footnote 41 However, the drugs submitted for FDA review during the two periods differ from each other in ways that make it difficult to interpret quantitative differences in the number of drug recalls before and after PDUFA’s passage.
Specifically, the different drugs submitted during these two periods embody different kinds of scientific knowledge and understanding of the appropriate methods for the treatment of disease. Given the resulting differences in risk profiles for the drugs submitted for FDA approval prior to and after PDUFA, it is difficult to interpret quantitative differences in the number of drug recalls in the two periods. Indeed, if the riskiness of the drugs submitted to the FDA decreased after PDUFA’s passage, and yet the number of drug recalls is equal to the number issued prior to PDUFA’s passage, the simple number of recalls does not reveal the FDA’s effectiveness.
It is possible that changes in the forms of scientific understanding that existed prior to PDUFA passage may have caused the drugs submitted to the FDA for review to have been fundamentally more- or less risky than those submitted after PDUFA’s passage, due to changes in the medical understanding of disease or the development of novel medicines. Changes in the frequency of drug recalls that followed PDUFA may be a result of the forms of scientific understanding embodied in the drugs submitted to the FDA for review, and not to factors associated with the changes PDUFA introduced, such as the increased number of personnel the FDA hired to evaluate new drug applications.
A lack of understanding of the changes in the risk levels of the drugs submitted for FDA approval makes it difficult to interpret any quantitative differences in drug recalls between the two periods. Indeed, even if the number of recalls fell after PDUFA, without an understanding of the underlying risk profiles of the drugs submitted for review during the two periods, the decline in drug recalls would be difficult to interpret. Changes in the state of medical knowledge, the invention of drugs as the result of new scientific discoveries, the development of new technologies, improvements in computing speed, changes in the types of firms submitting drug approval applications, changes in the profitability of the pharmaceutical industry relative to other sectors of the economy, and drug companies’ focus on the United States as the market for new drug releases, make it difficult to isolate PDUFA’s effects from other changes following PDUFA’s passage.Footnote 42
Hence, the basic inferential problem with efforts to compare the review times and safety records of drugs approved before and after PDUFA is that “for a national policy change such as PDUFA, there is unfortunately no U.S.-based control group of drugs to which to compare the experience of the Act.”Footnote 43 The problems created by the absence of a control group are not an inevitable characteristic of the good the FDA produces, but stem from the way the FDA’s decision authority prevents the generation of counterfactual knowledge.
While the FDA can be compared to other countries’ regulatory agencies, such as the EU’s European Medicines Agency (EMEA) or Health Canada, the complexity of such organizations, and the number of ways they are both similar to and different from each other, makes it difficult to isolate the specific features of agencies’ organizational structure and regulatory processes that are responsible for causing given outcomes. This problem is exacerbated when regulatory agencies rely on similar funding structures, as is the case with the EMEA’s collection of user fees in a manner similar to the FDA’s use of PDUFA.Footnote 44
One method for mitigating these inferential problems is to use different regulatory agencies’ experience with reviewing the same drugs to create near-identical counterfactual comparisons.Footnote 45 However, comparisons of countries’ regulatory outcomes are complicated by differences in the societies that rival agencies regulate. In the context of drug safety regulation, certain societies may exhibit different public health problems in ways that have implications for how we evaluate the trade-off between the accuracy of drug safety evaluation and drug review times. For example, Americans’ high obesity rates may make quicker review times for new anti-obesity drugs more efficient, even if doing so increases the risk of approving drugs that subsequently prove to be unsafe.
Thus, if we compare rival countries’ error rate in drug approvals, and find that a higher level of regulatory errors exists under PDUFA relative to, for example, the UK’s Medicines and Healthcare Products Regulatory Agency (MHRA), the different public health problems facing the United States and the UK may complicate interpretation of whether this difference is efficient given the different needs and characteristics of each country’s populations. These difficulties are exacerbated if agencies do not operate on stable biological processes that are relatively similar across societies, as is the case in drug regulation, but instead deal with fundamentally dissimilar social conditions, as is often the case with the varied characteristics of the economies different states regulate, the divergent interests they advance in international trade policy, or the interest rate decisions made by central banks.Footnote 46
While using compliance rates, or black market sales, as a proxy for regulatory quality may generate information about regulations, drawing correct inferences from these comparisons may be frustrated by the absence of experimental control. Indeed, societies have different historical experiences with law enforcement or state capacity, and these differences may frustrate efforts to determine whether compliance rates are caused by a specific regulation’s characteristics or characteristics of the societies regulations are imposed upon. For example, black markets in Italy differ from those in other European countries due to the presence of organized crime networks, and the nature of different crime networks within different regions of Italy may make it difficult to understand what a black market may mean for a policy’s efficiency even in the Italian context.
However, while our inability to compare drugs submitted under PDUFA to a group of identical drugs that were not subject to PDUFA frustrates identification of PDUFA’s causal effects, the absence of experimental control also makes it difficult to assess whether the FDA’s decisions are efficient. Evaluating PDUFA’s efficiency requires assessing whether PDUFA’s benefits are greater than those produced by alternative regulatory policies, and whether PDUFA’s costs are lower than other methods the FDA could have implemented for testing drug safety.
While PDUFA’s shortened review times helps pharmaceutical companies earn higher profits and aids consumers by expediting access to drugs that are quickly released to the market, these benefits come at a cost in terms of the deaths, and years of life lost, as a result of unsafe drugs that were approved under PDUFA that had to be withdrawn from the market. As the FDA loses the fees it charges pharmaceutical companies if it fails to review drugs by stipulated deadlines, there are concerns that PDUFA creates perverse incentives that negatively impact the quality of the reviews the FDA performs.Footnote 47
For example, statistical analyses of drugs approved in the two months immediately prior to the expiration of PDUFA deadlines are nearly seven times more likely to subsequently be withdrawn from the market due to safety issues relative to drugs approved in the earlier months leading up to the deadline.Footnote 48 While such safety problems may be due to a number of factors, it is possible that the threat of losing PDUFA funding alters the review process in ways that harm drug consumers.
However, it is worth noting that the overall number of drugs that are recalled under PDUFA is very small. For example, during PDUFA I–II (1992–2002) only nine of the drugs the FDA approved had to subsequently be withdrawn from the market.Footnote 49 Although this may indicate that the FDA is effective in preventing unsafe drugs from reaching consumers, the infrequency, and statistically indistinguishable number of drug recalls before and after PDUFA’s passage, does not demonstrate that PDUFA is as safe as the regulatory regime it replaced.Footnote 50 Although the FDA rarely withdraws drugs from the market, one of the largest public health disasters in U.S. history, the Vioxx scandal, occurred under PDUFA.
Vioxx was a COX-2 inhibitor intended to ease arthritis pain that the FDA approved in 1999 but was subsequently withdrawn from the market in 2004 due to evidence that it significantly increased the odds of having a stroke or a heart attack.Footnote 51 While Vioxx was one of only nine drugs withdrawn during PDUFA I–II (1992–2002), Vioxx is estimated to have caused between 88,000 to 140,000 heart attacks in the United States alone, of which 44 percent were probably fatal.Footnote 52 This indicates that under PDUFA II a single statistically unlikely regulatory error caused between 38,720 and 61,600 deaths, nearly the number of U.S. troops killed during the entire Vietnam War.
While the number of lives lost due to Vioxx is disturbing, it is difficult to assess whether PDUFA was causally responsible for this error. Although Vioxx was approved under PDUFA, it is simply unclear whether the FDA would have approved Vioxx had it been submitted for FDA review prior to PDUFA’s implementation, or whether the pressure to respond to PDUFA funding deadlines caused Vioxx to be approved when it otherwise would not have been. Any assessment of whether PDUFA was responsible for the deaths Vioxx caused hinges upon an unobservable counterfactual scenario regarding whether Vioxx would have been approved had it been submitted to the FDA prior to PDUFA’s implementation, yet this counterfactual simply cannot be empirically assessed.
This is not to suggest that it is impossible to try to estimate PDUFA’s cost-effectiveness. The most sophisticated cost-benefit analysis of the PDUFA has concluded PDUFA is net-positive in its dollar value and life-years saved, generating between $14 billion and $31 billion in total social surplus. While there are serious problems with the data these estimates are based upon, the critical question from an efficiency standpoint is not whether the net benefits exceed the policy’s costs, but whether an alternative policy would have generated greater benefits and lower costs.Footnote 53
In the context of the Vioxx episode, it is possible that a different funding mechanism would have prevented Vioxx from being approved and released to the market, and hence would have prevented tens of thousands of unnecessary deaths from occurring. However, it is also possible that Vioxx would have been approved regardless of the regulatory rules in place simply because the science behind the drug was poorly understood. Yet the singularity of the FDA decisions causes such questions to remain hypothetical, as it is impossible to observe the costs and benefits of regulatory decisions other than those the FDA implemented.
This feature of the FDA’s decisions makes it difficult to interpret the cost-benefit calculations that can be constructed regarding PDUFA, as any evaluation of PDUFA’s efficiency must try to assess whether a different regulatory policy would have generated greater benefits, in terms of faster drug evaluation, and fewer costs, such as the unsafe drugs the FDA mistakenly approved. As the consequences of policies that were not implemented cannot be observed, it is difficult to evaluate PDUFA’s costs and benefits. Since the inferential problems caused by this feature of the FDA’s regulatory decisions are derived from a fundamental feature of state authority, namely the singularity of the state’s decisions, epistemic problems confront efforts to assess the FDA’s efficiency that exist independently of the interests the FDA serves or the characteristics of its personnel.
V. Conclusion
While the study of administrative states often focuses on the interests they pursue, the motives of their personnel, or the concentrated benefits and dispersed costs of their decisions, this essay has examined the epistemic problems caused by the singularity of administrative states’ regulatory decisions, and considered how this feature of politics limits the generation of counterfactual knowledge necessary to assess the efficiency of their actions.
These problems are not derived from administrative actors’ interests, whether they are self-interested, or subject to regulatory capture. Indeed, even if states are staffed by public-spirited experts and are immune to organized groups’ influence, the singularity of their decisions frustrates efforts to assess their efficiency. Thus, while one can observe whether a given objective has been achieved, such as whether a rival army has been defeated or a certain inflation or unemployment rate has been maintained, it is difficult to know whether these outcomes were generated at minimum cost, or whether different policies would have generated greater benefits than those that were produced.
While such uncertainty frustrates administrative actors’ efforts to approach the ideal of scientific expertise that legitimates their authority in the minds of the public, these limits generate endless speculation regarding the efficiency of administrative states’ actions, whether, for example, certain policies prevented a recession from becoming worse than it would have become, or whether such policies were unnecessarily costly and actually prolonged economic hardship. If experiments are a critical component necessary for generating scientific understanding of the world, the singularity of the state’s decisions frustrates efforts to apply the experimental method to political questions.Footnote 54
The difficulty in conducting experimental tests of policies’ consequences makes it difficult to falsify contending claims regarding their effectiveness, or to decisively refute contending explanations regarding the causes of the social conditions states seek to influence. Thus, instead of exhibiting the progressive forms of understanding and control that scientific understanding has generated over certain realms of the natural environment, modern states limit the generation of counterfactual knowledge in ways that cause politics to exhibit the forms of irrationality and myth that persisted in scientific disciplines prior to the invention of the experimental method.Footnote 55
While these problems are exacerbated by suboptimal incentives, partisan bias, and bureaucratic intransigence, even if these features of modern politics were somehow eliminated, the singularity of modern states’ decisions would still frustrate application of the experimental method to political questions. Unfortunately, the methodological techniques social scientists use to mitigate the inferential problems created by the absence of observable counterfactual data exhibit their own problems and shortcomings. For example, the differences between historical periods may frustrate efforts to draw correct inferences from prior regulatory experience, and differences among states—political sub-units—may frustrate efforts to determine whether social conditions and regulatory outcomes were caused by unobserved variables. Even if states and regulatory systems are comparable, differences in the societies political systems regulate may make it difficult to assess the efficiency of the administrative state’s policy decisions.
These problems frustrate the development of scientific consensus regarding the appropriate policy means for given ends, and make it difficult to determine whether the administrative state could have produced better regulatory outcomes at a lower cost. Hence, the singularity of states’ decisions, and the way this feature of modern politics reproduces the fundamental problem of causal inference in empirical reality, causes well-meaning experts, elected officials, and citizens alike to exhibit intractable disagreements over questions regarding rival policies’ efficacy—disagreement that would persist even if there were widespread social agreement on the ends public policy should advance.
Many recommendations for reforming administrative pathologies merely transfer these problems to a different set of actors, either laymen citizens or elected officials, whose decisions confront the same epistemic challenges facing expert bureaucrats. These problems are merely exacerbated if society evolves over time, if novel innovations in knowledge or technology generate new wants, opportunities, and pathologies, or if social preferences are themselves subject to change.
However, this indicates that political disagreement may persist, not due to self-interest, corruption, or value conflicts, but because it is difficult to falsify hypotheses about policy effectiveness with the same forms of certainty that controlled experimentation grants over other realms of scientific knowledge. Hence a specific feature of the administrative state’s power frustrates its efforts to identify and adopt rational regulatory policies and prevents the analysis of public policy from developing the forms of consensus exhibited in disciplines that are more amenable to the experimental method, such as the medical treatment of disease. As these problems are derived from the specific way the state’s authority limits the scientific understanding of politics, it is difficult to see how they can be ameliorated or overcome.
Open access
