Procedure versus Performance

Recent years have witnessed growing efforts to identify institutional qualities of IOs with implications for citizens’ legitimacy beliefs toward these organizations. The starting point for most of this literature has been the distinction between input- and output-based legitimacy, introduced by Scharpf (Reference Scharpf1970, Reference Scharpf1999). This dichotomy originally served to identify two alternative normative grounds for justifying the authority of the EU. In Scharpf’s view, the EU could earn its normative legitimacy either from democratic participation by the people or from problem-solving outcomes for the people. This distinction fed into a broader debate about the normative credentials of European and global governance (e.g., Zürn Reference Zürn2000; Held and Koenig-Archibugi Reference Held and Koenig-Archibugi2005; Archibugi et al. Reference Archibugi, Koenig-Archibugi and Marchetti2012). Scharpf’s dichotomy was also picked up by researchers interested in establishing institutional sources of sociological legitimacy for IOs. Over the past decade, a growing literature has distinguished between procedure (input) and performance (output) as two generic institutional sources of legitimacy for regional and global IOs (Bernauer and Gampfer Reference Bernauer and Gampfer2013; Binder and Heupel Reference Binder and Heupel2015; Dellmuth and Tallberg Reference Dellmuth and Tallberg2015; Tallberg and Zürn Reference Tallberg and Zürn2019).

The premise of procedural accounts is that process criteria are important for perceptions of IO legitimacy. On this argument, actors support an institution’s exercise of authority because of how it is set up and operates. Procedural accounts have an early antecedent in Weber’s (Reference Weber1922/1978) notion of legal-rational sources of legitimacy. On these lines, governance is regarded as appropriate because properly appointed authorities follow properly formulated decision-taking processes. So, for example, audiences might accord legitimacy to the UNFCCC because its policy-making is perceived to involve a broad range of stakeholders. Alternatively, actors might deny legitimacy to the IMF because its decision-making process is seen to give some states disproportionate weight. For procedural accounts, the legitimacy of an IO derives from the way that the institution functions, irrespective of the consequences of its policies.

Recent research offers several examples of procedural accounts. For instance, Bernauer and Gampfer (Reference Bernauer and Gampfer2013) focus on whether procedures that allow for greater civil society involvement also translate into greater legitimacy for global environmental governance. They find this to be the case: Citizens tend to favor civil society engagement, and therefore, procedures that provide for such participation are rewarded with higher legitimacy. Similarly, Johnson (Reference Johnson2011) studies how procedures giving certain states particular advantages (e.g., through vetoes) influence the legitimacy of IOs. She finds that IOs which grant major states such as the US and Russia a special say in decision-making suffer in terms of perceived legitimacy.

In contrast, other accounts emphasize performance as an institutional source of IO legitimacy. On these lines, legitimacy beliefs derive from audience evaluations of a governing institution’s outcomes. With a focus on performance, IOs might gain or lose legitimacy depending on whether audiences see them as enhancing or undermining desired conditions in society. For example, the WHO might gain legitimacy if actors perceive that it effectively prevents epidemics. Meanwhile, the World Bank might lose legitimacy if subjects believe that this institution fails to reduce poverty. For performance approaches, the legitimacy of an IO derives from its impacts, irrespective of how the institution formulated and executed the relevant policy.

Existing research provides many examples of this type of argument. Multiple studies of public opinion toward the EU highlight the importance of policy-making outcomes for people’s legitimacy beliefs. These investigations show that citizens evaluate the EU’s legitimacy in relation to costs and benefits, both for their personal well-being and for their country (Gabel Reference Gabel1998; Hooghe and Marks Reference Hooghe and Marks2005; Rohrschneider and Loveless Reference Rohrschneider and Loveless2010). Edwards (Reference Edwards2009) advances a similar argument to explain public opinion toward the IMF, the World Bank, and the WTO. He finds that people’s legitimacy beliefs toward these IOs are primarily driven by the perceived implications of these organizations for their country’s economy.

While this existing work has expanded our understanding of how institutional features of IOs may affect people’s legitimacy perceptions, it suffers from two key limitations. First, efforts to compare the causal significance of communication about procedures and performances for popular legitimacy beliefs are missing. Especially in global governance, where citizens tend not to have direct experiences with IOs, elite communication is central as a mediating factor between institutional qualities and legitimacy beliefs. However, the few contributions testing factors drawn from both categories rely on observational methods such as textual analysis and public opinion surveys that do not allow for inferences about communication (e.g., Edwards Reference Edwards2009; Binder and Heupel Reference Binder and Heupel2015; Dellmuth and Tallberg Reference Dellmuth and Tallberg2015; Schlipphak Reference Schlipphak2015).

Second, existing research has not systematically unpacked procedure and performance to consider the more specific qualities of processes and outcomes that may generate beliefs in the legitimacy of an IO. Procedure and performance are broad and encompassing categories which, while conceptually useful, evade the deeper question: What is it more particularly about an institution’s conduct and results that generates perceptions of legitimacy? Each category encompasses a diversity of more specific qualities that may be important for legitimacy assessments. In terms of procedure, is it civil society involvement, decision-making rules, legality, efficiency, or other features that elicit legitimacy beliefs toward IOs? In terms of performance, is it problem-solving capacity, distributional consequences, implications for democratic governance, or other outcomes that generate perceptions of IO legitimacy?

As a result of these limitations, important questions remain unanswered. Is communication about some institutional qualities more significant than communication about others in shaping citizens’ legitimacy beliefs regarding IOs? Do effects of communication about specific institutional features vary across countries, IOs, and issue areas?

This chapter moves beyond these limitations. The remainder of this section develops a conceptualization that unpacks procedure and performance to identify specific qualities of each that often feature in elite communication about IOs. Subsequent sections then describe the design and results of a survey experiment that evaluates the causal effects of specific communicated institutional qualities on popular legitimacy beliefs across IOs in three policy fields.

Unpacking Institutional Qualities

Building on the distinction between procedure and performance, this section develops a richer and more systematic typology of institutional qualities to which citizens may be receptive. As the preceding discussion indicates, there is more to the sources of IO legitimacy than procedure and performance per se. To get at the institutional sources for legitimacy beliefs we must also examine the specific qualities of procedure and performance.

Starting from Scharpf’s (Reference Scharpf1999) work, previous research has tended to equate procedure with qualities of democracy and performance with qualities of effectiveness. However, this conflation excludes the possibility that democracy and effectiveness could each be features of both procedure and performance (Tallberg and Zürn Reference Tallberg and Zürn2019). For example, democratic governance could be an outcome in cases where an IO’s activities (such as election monitoring) generate more transparency and participation in politics. Likewise, effectiveness could be a quality of policy processes if a given institutional procedure (such as use of expert committees) allows an IO to produce more and faster decisions. In addition, it is plausible, for reasons elaborated below, that other qualities of procedure and performance besides democracy and effectiveness, such as fairness, could shape legitimacy perceptions.

Based on these considerations, we introduce a new typology of institutional sources of legitimacy by means of a 2×3 matrix (Table 6.1). In this schema, the two rows make the distinction between procedure and performance that has evolved out of earlier research. Hence, the matrix distinguishes between sources of legitimacy that pertain to the processes and to the consequences of IO policy-making. Meanwhile, the columns in the matrix highlight a threefold distinction between democratic, technocratic, and fair as three generic qualities that may apply to both the procedures and the performances of IOs.

Table 6.1

Institutional sources of legitimacy

This new typology uses the category of democratic procedure and performance to cover perceptions that affected publics have due voice in and control over governance arrangements. The category of technocratic procedure and performance is taken to encompass perceptions that a governing authority is effective in the light of best available knowledge and policy instruments. The term “technocratic” is not invoked here with any pejorative connotations and rather seeks to convey a sense of expertise-based problem-solving (Fischer Reference Fischer1989). Finally, the category of fair procedure and performance captures perceptions that process and outcome are just, equitable and impartial vis-à-vis implicated actors. Fairness is a feature of both procedure and performance that has obtained limited independent attention in previous research and is distinctive from both democratic and technocratic qualities (for an exception, see Hurd Reference Hurd2007).

The 2×3 schema yields six types of institutional features that may generate perceptions of legitimacy for IOs when communicated by elites. The expectation that people care about these qualities is anchored in social-psychological research. The right to voice opinions, a key political norm held by individuals in democratic societies (Held Reference Held1995), has been shown to drive perceptions of legitimacy toward domestic political institutions (De Cremer and Tyler Reference De Cremer and Tyler2007). Similarly, outcome favorability is well known to influence how people evaluate policy decisions and institutions (Skitka Reference Skitka2002; Dohertly and Wolak Reference Dohertly and Wolak2012; Esaiasson et al. Reference Esaiasson, Persson, Gilljam and Lindholm2019). Finally, extensive research has demonstrated that fairness is a fundamental concern in politics and matters to people’s perceptions of legitimate institutions (Tyler Reference Tyler1990; Tyler et al. Reference Tyler, Boeckmann, Smith and Huo1997).

The following paragraphs specify and exemplify the six institutional qualities in the context of IOs. First, the category democratic procedure covers frameworks and practices that bring affected people into IO policy-making processes. In this category, one prominent institutional quality is participation: namely, where implicated actors have due involvement in the formulation, implementation, and review of IO decisions (Steffek et al. Reference Steffek, Kissling and Nanz2007; Macdonald Reference Macdonald2008). Another significant feature of democratic procedure is transparency: namely, where affected publics can access full information about an IO’s activities and policies, making it easier to hold the organization accountable (Scholte Reference Scholte2011; Tallberg Reference Tallberg, Cogan, Hurd and Johnstone2016). Several studies argue that democratic procedure has become the foremost source of legitimacy in global governance (e.g., Held Reference Held1995; Bernstein Reference Bernstein2011). More specifically, some research finds that civil society involvement strengthens popular legitimacy in global environmental governance (Bernauer and Gampfer Reference Bernauer and Gampfer2013). Other work identifies dissatisfaction with allegedly nondemocratic decision-making as a crucial motivation for public contestation of IOs (Della Porta and Tarrow Reference Della Porta and Tarrow2005; Norris Reference Norris2011).

The category democratic performance captures the consequences of policy-making for people’s power in national and international governance. Important in this respect are IO activities that safeguard or improve domestic democracy, for instance, by demanding accountability of national governments or by protecting civil rights (Pevehouse Reference Pevehouse2005; Keohane et al. Reference Keohane, Macedo and Moravcsik2009). The expectation that democratic performance matters for IO legitimacy is frequently expressed in the negative: namely, that people withhold support from IOs because of their negative implications for domestic democracy (Hooghe et al. Reference Hooghe, Lenz and Marks2019). Such thinking also figures prominently in populist discourse that argues for a repatriation of powers from IOs to democratic domestic governance (Inglehart and Norris Reference Inglehart and Norris2017).

The category technocratic procedure encompasses practices that bring efficiency and expertise to policy-making processes. Efficiency can lie in the number and speed of an IO’s decisions (Hardt Reference Hardt2014; Tallberg et al. Reference Tallberg, Sommerer, Squatrito and Lundgren2016), while expertise can involve basing IO decisions on the best available knowledge and skills (Majone Reference Majone1998; Bernstein Reference Bernstein2005). Along these lines, Chan et al. (Reference Chan, Carlo Carraro, Kolstad and Stavins2016) argue that the legitimacy of the Intergovernmental Panel on Climate Change (IPCC) rests primarily on the scientific expertise that this body brings to global environmental governance. Conversely, slow decision-making, mismanagement of funds, and organizational dysfunction are frequently invoked when explaining legitimacy difficulties for organizations such as the UN and the EU (Reus-Smit Reference Reus-Smit2007).

The category technocratic performance refers to effectiveness in achieving policy ends. Qualities under this heading include problem-solving (i.e., successfully addressing a policy challenge) and collective gains (i.e., achieving benefits for society) (Keohane Reference Keohane1984; Scharpf Reference Scharpf1999). It is commonly claimed that IOs earn their popular legitimacy through the collective advantages they produce for states and societies. In this vein, Lindberg and Scheingold (Reference Lindberg and Scheingold1970) introduced the term “permissive consensus” to describe a situation where populations enjoy the fruits of international cooperation and support its broad goals, while taking little concrete interest in IO policy-making processes. Likewise, functionalist accounts assume that states and their domestic constituencies support IOs because of the collective benefits they generate (Keohane Reference Keohane1984). In line with these expectations, some recent empirical research concludes that citizen perceptions of successful IO problem-solving constitute a strong base for legitimacy beliefs (Dellmuth and Tallberg Reference Dellmuth and Tallberg2015; Ecker-Ehrhardt Reference Ecker-Ehrhardt2016).

The category fair procedure refers to policy-making practices that give equitable treatment to all concerned. This quality is exhibited in impartiality (i.e., decision-taking processes are followed consistently and without discrimination) and proportionality (i.e., members contribute to IO resourcing in accordance with their relative means). For some types of IOs, such as international courts, fair procedure may be the chief institutional source of legitimacy. For instance, low legitimacy for the ICC among many African governments arises from their perception that the ICC imposes double standards between African and other leaders (Helfer and Schowalter Reference Helfer and Schowalter2017).

The category fair performance involves consequences of policy-making in terms of equitable outcomes. This quality can be judged in relation to IO practices that advance human dignity (i.e., norms of basic cultural, economic, and political livelihood) and distributive justice (i.e., equitable sharing of benefits and burdens) (Tyler Reference Tyler1990). For example, global justice protests have often targeted international economic institutions for allegedly producing unacceptable inequalities in society (O’Brien et al. Reference O’Brien, Goetz, Scholte and Williams2000; Scholte et al. Reference Scholte, Fioramonti and Nhema2016). Conversely, IOs with poverty alleviation profiles – such as multilateral development banks and the United Nations Development Programme (UNDP) – often legitimize themselves in terms of promoting fairness for underprivileged people. The expectation is that citizens are more likely to accord legitimacy to IOs that are perceived to reduce injustice within and between countries.

In sum, the typology developed above disaggregates the conventional distinction between procedure and performance to bring out six more specific institutional sources of IO legitimacy. Each of the six qualities gives rise to a hypothesis about effects on legitimacy beliefs when this quality of an IO is communicated to citizens (see Table 6.2). Together, these distinctions permit a more fine-grained assessment of how communicated institutional features shape citizen perceptions of legitimacy in global governance. Social psychology, earlier political research, and anecdotal examples suggest that each of these institutional features may matter for legitimacy beliefs toward IOs. However, it is an empirical question for systematic investigation to establish in what ways and to what extent this is actually the case.

Table 6.2

Hypotheses and corresponding treatments

Notes: Wording of vignettes for the IMF. The wordings change depending on the IO, as they are active in different issue areas. For exact wordings in the second and third round, see Online Appendix N. After receiving the treatment, people were asked how much confidence they have on IOs: “How much confidence do you personally have in the UN?” Answer categories range from 0 (no confidence at all) to 10 (complete confidence); “don’t know.” The control group receives the question about confidence without a vignette.

Survey Design

The survey experiment was conducted in Germany, the Philippines, South Africa, and the US. These countries were selected based on two criteria. First, they lie on different continents, thereby reducing the risk of regional biases and covering countries with different experiences of the IOs covered. Second, the four selected countries all have relatively high levels of Internet penetration (almost 90 percent in Germany and the US, over 50 percent in South Africa, and more than 40 percent in the Philippines), thereby increasing the representativeness of the sample to the whole populations of those countries.

The questionnaire was implemented by YouGov from September to November 2016 (see Online Appendix A). A total of 1,586 interviews were conducted in Germany, 1,358 in the Philippines, 1,384 in South Africa, and 1,393 in the US. After respondents accessed the online survey, they filled in a questionnaire in English (except in Germany, where it was translated into German). The survey took about 5 minutes to complete. It started with several “warm-up” questions, then moved to the survey experiment, and ended with several additional questions (see Online Appendix N for the questionnaire). The warm-up and supplementary questions were included to enable balance tests and robustness checks (see below). Questions were sequenced in ways to avoid priming the respondents inappropriately.

Experimental Design

The experimental part of the questionnaire aimed to isolate the causal effects on legitimacy beliefs of communication regarding the six institutional qualities discussed above. To this end, individuals were randomly assigned to groups that received different experimental treatments, as well as to a control group that did not receive any treatment (Mutz Reference Mutz2011, 9). The randomized design consisted of two factors that varied across the respondents of the survey: (a) dimension of policy-making (procedure or performance) and (b) institutional quality (democratic, technocratic, or fair). The resulting 2×3 factorial design yielded six conditions in total. Each combination of factors was presented to a similar number of individuals (around 820).

Table 6.2 summarizes the hypotheses-related treatments tested in the experiment. Taken together, these treatments allowed us not only to identify and compare the respective causal impacts of procedure and performance on legitimacy beliefs but also to disaggregate these dimensions to evaluate the importance of democratic, technocratic, and fair qualities of procedure and performance.

The experimental treatments were operationalized in vignettes. It was vital that respondents should react to the precise prompt in the respective vignettes. We therefore kept vignette formulations as short and straightforward as possible, and similar in strength, since longer and more complex vignettes lead to a greater risk of ineffectual treatments (Mutz Reference Mutz2011, 64–65). The vignettes involved hypothetical rather than actual scenarios. Although using real-world information can increase the credibility of vignettes, using hypothetical vignettes makes it easier to ensure that treatments are of similar strength.

The vignettes exposed respondents to a description of an IO (see Table 6.2). For instance, respondents were presented with the information that “the United Nations (UN) holds its meetings about military conflicts in public,” “United Nations (UN) actions on climate change strengthen democracy in affected countries,” “the International Monetary Fund (IMF) responds on time to financial crises,” or “European Union (EU) actions on regional cooperation benefit everyone equally” (Online Appendix N). The use of framed descriptions as vignettes builds on the assumption in our theory (Chapter 3) that citizens on average are not very well informed about political matters and therefore use communicated information to update their opinions on issues. Survey data suggest that this assumption is reasonable in the present case: Citizens are typically aware of the existence of the IOs used in the experiment, but lack a detailed understanding of their decision-making procedures and policy performances (Gallup 2005; Dellmuth Reference Dellmuth2016). When respondents are presented with framed descriptions of IOs, the expectation is therefore that they integrate the new information into their opinions. If this new information relates to features of IOs that respondents care about, it should also influence their perceptions of the organization’s legitimacy.

In the treatment groups, people received first a vignette and then a question about their “confidence” in IOs. In contrast, the control group received only the question of how much confidence the respondent has in an IO, without the preceding vignette. Confidence – our preferred measure of legitimacy, as set forth in Chapter 3 – was measured on a scale from no confidence at all (0) to complete confidence (10). Most respondents answered on this scale, although a small minority (close to 10 percent) selected a “don’t know” option.

The experiment used a randomized factorial design that systematically varied vignettes on combinations of the first factor (procedure or performance) and the second factor (democratic, technocratic, or fair). This made it possible to establish the distinct causal effects on IO legitimacy of information regarding each of the six institutional qualities. Each vignette was formulated to operationalize one central institutional quality in each of the six categories summarized in Table 6.1: transparent decision-making (democratic procedure), effects on domestic democracy (democratic performance), efficient decision-making (technocratic procedure), effective problem-solving (technocratic performance), equal say in decision-making (fair procedure), and equal distribution of benefits (fair performance). The effects of information regarding these six institutional qualities were evaluated by comparing mean confidence in the treatment groups with mean confidence in the control group. Since people are psychologically more responsive to negative information than to positive prompts (Kahnemann and Tversky Reference Kahneman and Tversky1979; Baumeister et al. Reference Baumeister, Bratslavsky, Finkenauer and Vohs2001; see also Chapter 4), vignettes were constructed using a positive formulation in order to enable a hard theory test. Positive treatments also speak to IOs’ efforts to appear more democratic, effective, and fair in their procedures and performances.

In sum, we establish whether communication about an institutional quality matters for respondents’ legitimacy beliefs toward an IO by first exposing them to a treatment containing information on this particular quality, then asking them about their confidence in the IO, and finally comparing the average level of confidence among the respondents who receive this particular treatment with that of respondents who receive no treatment (the control group). A statistically significant difference in the average level of confidence between the two groups allows us to conclude that the institutional quality, which is manipulated in the particular treatment, likely contributes to respondents’ confidence in the IO.

The experiment was conducted in four rounds designed to capture institutional sources of legitimacy for seven IOs: three global organizations (IMF, UNFCCC, and UNSC) and four regional organizations (ASEAN, AU, EU, and NAFTA). This design permits the experiment not only to test expectations about the general effects of these six institutional qualities but also to show the occurrence of such effects across countries and IOs. The first round presented vignettes related to the UN’s actions on military conflict, speaking to the UNSC, the primary IO in the area of security. The second round presented vignettes about the UN’s actions on climate change, speaking to the UNFCCC, the central IO on this issue. The third round addressed a prominent economic IO, the IMF. The fourth round focused on one regional organization relevant for each of the four included countries (ASEAN in the Philippines, AU in South Africa, EU in Germany, and NAFTA in the US). Respondents were never exposed to the same treatment more than once. Respondents who were placed in the control group remained there throughout the three rounds. The order of the experimental rounds was randomized in order to avoid potential priming effects.

Effects of Communication about Institutional Qualities

The key finding is that communication about all six procedure- and performance-related institutional qualities affected respondents’ legitimacy beliefs. Figure 6.1 sets out differences in average levels of confidence between the control group and the respondents receiving treatments. All of the treatments generated statistically significant effects on respondents’ confidence in IOs, be they about democratic, technocratic, or fair qualities of procedure and performance.

Figure 6.1

Effects of communication about institutional qualities

Notes: Average treatment effects with their respective 95 percent confidence intervals. Weighted data. See Online Appendix O1 for detailed results.

The size of the treatment effects varies from 0.241 to 0.419 on an 11-point scale of confidence. These effects are substantively important. For example, an effect size of 0.419 for “democratic performance” is similar in size to the average difference in confidence between the UNFCCC and the IMF in the control group (Figure 6.2). In addition, the experimental setting likely underestimates the corresponding real-world impact. After all, the experimental effects result from a one-shot treatment, rather than from continuous exposure to a particular institutional quality, as would be the case in actual situations (Gaines and Kuklinski Reference Gaines, Kuklinski, Druckman, Green and Lupia2011). Moreover, it should be recalled that positive treatments usually generate smaller effects than negative treatments, since people are psychologically less responsive to positive information than to negative prompts.

Figure 6.2

Average confidence in IOs in the control group

Notes: Weighted means, based on data from the control group. N = 676.

We also assess if cues about procedure and performance are more effective among the politically more aware, given that they are more likely to use new information efficiently when updating their opinions, as theorized in Chapter 3. We therefore test if the treatment effects depicted in Figure 6.1 are conditional on our two awareness indicators: education and discussion of politics with friends. The results suggest that more politically aware citizens did not respond differently than less aware citizens when confronted with the treatments (Online Appendices O2–O3).

Taken together, these results demonstrate that both procedure and performance can affect people’s confidence in IOs, and this conclusion holds across democratic, technocratic, and fair qualities of procedure and performance. Hence, claims that legitimacy rests with either procedure or performance, or with either democratic, technocratic, or fair features of IO policy-making, would appear to be misguided.

The finding that democratic, technocratic, and fair qualities all matter speaks to earlier research in interesting ways. First, this result supports previous research which holds that democratic concerns are central to people’s evaluations of IOs (Held Reference Held1995; Zürn Reference Zürn2000; Bernstein Reference Bernstein2011). Second, this finding shows that the importance of fairness for legitimacy beliefs applies to international as well, ant not only domestic political institutions (Lind and Tyler Reference Lind and Tyler1988; Tyler Reference Tyler1990; Gibson et al. Reference Gibson, Caldeira and Kenyatta Spence2005; Esaiasson et al. Reference Esaiasson, Persson, Gilljam and Lindholm2019). Third, this outcome cautions against the proposition that people are nowadays less concerned with technocratic aspects of IOs (Scharpf Reference Scharpf1999; Hooghe and Marks Reference Hooghe and Marks2009), instead pointing to the sustained importance of efficiency and effectiveness considerations for legitimacy perceptions.

Disaggregating Treatment Effects across IOs

Next, we examine whether the effects of communication regarding procedure- and performance-related institutional qualities vary across IOs, considering the results from each experimental round separately. Figure 6.3 shows the results for the global organizations, while Figure 6.4 depicts the results for the regional organizations.

Figure 6.3

Effects of communication about institutional qualities, by global organization

Notes: Average treatment effects with their respective 95 percent confidence intervals. Weighted data. See Online Appendix O4 for detailed results.

Figure 6.3 suggests that all treatments are effective in relation to the IMF, with the exception of the treatment on fair procedure. By contrast, in the context of the UNSC and UNFCCC, fewer treatments are statistically significant. In the UNSC, democracy-related qualities matter, and in the UNFCCC, we find no significant effects. Figure 6.4 shows that no significant treatment effects were found for the regional organizations. However, it should be noted the analyses of the regional organizations are based on a considerably smaller sample of respondents – only one country per IO.

Figure 6.4

Effects of communication about institutional qualities, by regional organization

Notes: Average treatment effects with their respective 95 percent confidence intervals. Weighted data. See Online Appendix O4 for detailed results.

These results suggest four observations. First, they confirm that communication regarding both procedural and performance qualities matters for the perceived legitimacy of IOs, also when we break down the effects by IO. In the context of IOs with significant treatment effects (IMF, UNSC), both procedure and performance are important. Second, treatments are most effective in the US, when compared to Germany, the Philippines, and South Africa. Third, the findings suggest that communication regarding institutional qualities is of varying importance for global organizations, mattering most for legitimacy perceptions toward the IMF, less for the UNSC, and least for the UNFCCC. Finally, the varying occurrence of treatment effects at the level of countries and IOs reduces potential concerns that the homogenously positive and significant effects established earlier at the aggregate level would result from a general framing effect in the experiment.

Focusing specifically on the variation in treatment effects across IOs, five interpretations are conceivable. First, variation may reflect the respective missions of the IOs. As previously noted, it may be that some organizational purposes generate legitimacy beliefs more than others because these mandates are perceived to be intrinsically good and uncontestable (Scott Reference Scott, Walter and DiMaggio1991; Lenz and Viola Reference Lenz and Viola2017). If this logic is at play, it may affect the relative importance of other institutional qualities as sources of legitimacy. For example, the UN’s attention to climate change may generate legitimacy beliefs by itself and thereby reduce the relative importance of procedural and performance features. In contrast, the IMF pursues objectives that are perceived to be more contestable, which may elevate the relative importance of procedural and performance sources of legitimacy.

Second, differences across issue areas could reflect varying levels of authority among the IOs in question. According to one argument, the standards that an IO must reach to be considered legitimate depend on its level of policy competence (Tallberg and Zürn Reference Tallberg and Zürn2019). Institutions that possess higher levels of authority must meet more demanding standards of procedure and performance or else they will suffer a legitimacy deficit. This logic may help to explain why the full range of institutional qualities matter for legitimacy beliefs toward the IMF, which has far-reaching formal authority. Consistent with this interpretation, fewer institutional qualities matter in the case of the UNSC, which possesses medium authority, and none at all in the case of the UNFCCC, which has limited formal authority (Hooghe et al. Reference Hooghe, Marks, Lenz, Bezuijen, Ceka and Derderyan2017; Zürn Reference Zürn2018). In Chapter 7, we will systematically examine effects of purpose and authority in conjunction with procedure and performance.

Third, citizens could evaluate some IOs on a broader range of parameters than others. This interpretation ties in well with previous anecdotal evidence on sources of contestation in global governance. Consistent with our experimental findings, public opposition to IOs in the area of economic governance invokes a broad range of criticisms, including democratic shortcomings, perceived injustices of neoliberal doctrine, and ineffective policies (O’Brien et al. Reference O’Brien, Goetz, Scholte and Williams2000; Rauh and Zürn Reference Rocabert, Schimmelfennig, Crasnic and Winzen2020). Similarly, and in keeping with the results here, earlier research on the legitimacy of the UNSC shows that its democratic limitations have attracted particular criticism (Binder and Heupel Reference Binder and Heupel2015).

Fourth, the observed pattern of variation across IOs is consistent with the possibility that citizens respond more strongly to new information about IOs they know less well (cf. Chong and Druckman Reference Chong and Druckman2007a). As we theorized in Chapter 3, the object of messages may matter for communication effects if citizens have stronger priors toward some IOs than others. Earlier survey data suggest that the UN is more known to citizens in a global sample of countries than the IMF (Gallup International Association 2005; Dellmuth et al. Reference Dellmuth, Scholte, Tallberg and Verhaegen2022b, ch. 2). This finding is further corroborated by evidence from a question in our survey on factual knowledge about the IMF and the UN (Figure 6.5) (Mondak Reference Mondak1999; Jessee Reference Jessee2017). In addition, the four regional IOs are likely better known among citizens than the global IOs, as suggested by data from the Gallup International Association (2005, 2011) analyzed in Chapter 4. Thus, the broader range of treatment effects for the IMF may partly reflect respondents’ greater sensitivity to new information about this IO, compared to the other two global IOs, and the absence of treatment effects for the regional IOs may partly reflect citizens’ more well-developed priors regarding these organizations.

Figure 6.5

Political knowledge about IOs

Notes: Weighted percentages. Left panel shows responses to a question about knowledge about where the headquarters of the IMF are located. Answer categories: “A) Washington, DC, B) London, C) Geneva, D) Don’t know.” Right panel shows responses to a question about knowledge about which of the following countries does not have a permanent seat on the Security Council of the United Nations. Answer categories: “A) France, B) China, C) India, D) Don’t know.”

Fifth, and related, variation in treatment effects may be shaped by contextual factors, such as whether an IO is subject to intense public debate at a certain point in time. Much like general knowledge about an IO, intense public debate around a particular organization can lead to more developed opinions (cf. Bakaki and Bernauer Reference Bakaki and Bernauer2017) and therefore less responsiveness to treatment manipulation. For example, the timing of our survey experiment less than a year after the signing of the Paris Agreement might have contributed to the lack of treatment effects in relation to climate governance. Conversely, contextual events can sensitize people to a particular aspect of an IO’s work, leading to larger treatment effects on that dimension. For instance, the results for problem-solving (technocratic performance) in respect of the UNSC might have been different twenty years ago in the aftermath of its shortfalls in Rwanda and Bosnia. Similarly, the treatment effects in relation to the IMF could potentially have been even stronger in the 1990s when the Fund’s structural adjustment programs were intensely contested.

Disaggregating Treatment Effects across Countries

Next, we examine whether the effects of communication regarding procedure- and performance-related institutional qualities vary across countries. Figure 6.6 calculates the differences in average confidence between the control group and the respondents receiving treatments for each country separately. It suggests several interesting patterns.

Figure 6.6

Effects of communication about institutional qualities, by country

Notes: Average treatment effects with their respective 95 percent confidence intervals. Weighted data. See Online Appendix O for detailed results. Numerical results in Online Appendix O5.

First, the results at the country level are heterogeneous and quite weak. Citizen opinions generally move in the expected directed in two countries (South Africa and US), move little at all in one country (Germany), and move in the opposite direction than expected in one country (Philippines). However, only few treatments are significant. This may partly be the result of lower statistical power, since the N in each category is considerably smaller when we analyze these treatment effects by country (Online Appendix O5).

Second, if we focus on the significant effects, Figure 6.6 reveals that information about technocratic procedure and fair performance leads US citizens to update their opinions about IOs. By contrast, we do not find any treatment effects in Germany and South Africa. In the Philippines, we observe one treatment effect, albeit in the negative direction: Information about technocratic performance appears to weaken legitimacy beliefs in IOs. This deviant result could potentially be understood against the background that public opinion toward IOs in the Philippines is more positive than in other countries (see also Dellmuth et al. Reference Dellmuth, Scholte, Tallberg and Verhaegen2022a). In our data, mean confidence in IOs in the Philippines is 6.2, while it is much lower in the other three countries (4.8 in South Africa, and 3.9 in Germany and 3.6 the US).Footnote ² The high degree of confidence in IOs in the Philippines is also a consistent pattern in WVS data over time. WVS data from 1994 through 2019 indicate that respondents in the Philippines on average have particularly high levels of confidence in the UN compared to respondents in other countries.Footnote ³ Thus, it could be that citizens in the Philippines, because of the high degree of confidence in IOs, have particularly high expectations and therefore are little moved by positive treatments – indeed, may even be disappointed by the additional information.

Validity and Robustness Checks

We conclude this analysis with a number of validity and robustness checks. To test the internal validity of the experiment, we report a series of balance tests. These tests are based on the responses to several additional questions asked in the survey, capturing, inter alia, political knowledge about global governance. The tests check if the randomized allocation of respondents across treatment groups has worked by assessing if there is a statistically significant difference in mean confidence across levels of these variables. The tests reveal only eight imbalances for sixty tests, which should not compromise causal inference (Online Appendix P).

Moreover, we examine if the results could have been moderated by people’s confidence in the domestic government or the interest they have in specific IOs. To begin with, we examine for each country separately if treatment effects depend on confidence in domestic government, as more trusting people might have more fixed opinions about institutions in general and therefore be less swayed by elite communication. However, we do not find evidence for this intuition (Online Appendix Q1). Moreover, the results do not depend on respondent’s interest in the respective global organizations (UN or IMF) (Online Appendix Q2).