Research design and hypotheses

Our pre-analysis plan was uploaded to the Evidence in Governance and Politics (EGAP) registry on April 30, 2020, and can be found in Online Appendix F. The study relies on Facebook’s split test feature for advertisements to implement the experiments. This feature allows advertisers to assign several ads to randomly constructed “audiences” of Facebook profiles for the purpose of comparing the relative effectiveness of the ads in achieving some desired outcome (Orazi and Johnston Reference Orazi and Johnston2020). Because the profiles are randomly assigned to view one of the ads but not the others, this relative effectiveness is causally identified. Within the ad, users are directed to click the ad to show their support for refugees. Thus, clicking on ads represents a behavioral measure of public engagement with the efforts of refugee advocates: this is our primary outcome measure.

Once a Facebook user clicks on one of the ads, they are redirected to a short Qualtrics survey, allowing us to perform additional analysis of the mechanisms underlying our treatment effect. Give the selection bias in who consents to completing the survey, this analysis would remain exploratory; and, as we later explain, we obtained a very low response rate on the survey, rendering this analysis infeasible. Consort diagrams of the research design are shown in Figures A.1 and A.2 in Online Appendix A.

Our first hypothesis draws on recent work demonstrating that there is higher support for refugees when they are seen as contributing to society (Adida, Lo, and Platas Reference Adida, Lo and Platas2019). Given the extent of the pandemic’s impact on the economy and politics of the USA, as well as the significant anxiety about the crisis felt by much of the public (McCarthy Reference McCarthy2020), it is plausible that refugees will benefit from emphasizing their contributions to the pandemic response. Indeed, refugee organizations have been pushing narratives of refugee solidarity efforts during the crisis. Our first hypothesis tests whether this strategy effectively increases public engagement with refugee advocacy organizations. There are at least two mechanisms related to emotions that could increase engagement with refugee organizations. The first is anxiety. Mentioning coronavirus could increase users’ anxiety and cause them to seek out more information on what refugees are doing to help during the pandemic (Albertson and Gadarian Reference Albertson and Gadarian2015). The second mechanism is gratitude, which has been shown to increase pro-social political behaviors like voting (Panagopoulos Reference Panagopoulos2011). Some users may feel gratitude for the efforts of refugees to make a difference during this time of crisis, particularly since helping during the crisis requires a larger sacrifice than helping at other times.

Second, social scientists debate whether individual support for policies toward migrants is driven by self-interest or sociotropic concerns. In times of crisis, such as natural disasters, parochial concerns may be particularly salient (Chang Reference Chang2010). These more parochial interests may lead Americans to place a higher value on the actions of solidarity taken by refugees in their own communities over actions that benefit individuals in other parts of the USA (Kustov Reference Kustov2020). This leads to our second hypothesis, tested in the context of COVID-19.

Finally, recent work suggests that individuals are more supportive of migrants with severe vulnerabilities (Bansak, Hainmueller, and Hangartner Reference Bansak, Hainmueller and Hangartner2016). Refugee migration is more likely to meet this criterion than is immigration, which is typically understood to be voluntary rather than forced. Additionally, refugee advocates have communicated to us that they wonder whether labeling a migrant as a refugee or immigrant makes a significant difference in how the public responds. Our third hypothesis tests this labeling effect in the context of the pandemic.

Experimental design details

Our study requires two sets of experiments because the Facebook platform constrains us to a maximum of five conditions per experiment. The first experiment features Mustafa, a former refugee from Somalia who now lives in Lancaster, PA. Mustafa has been volunteering to deliver groceries to elderly people who cannot leave their residences easily. As shown in Figure 1, the ads include a photo of Mustafa with bags of groceries, an ad headline, and a brief description that includes our experimental manipulations. To test H1, the headline says either: “Refugees help America fight coronavirus” or “Refugees help America.” To test H2, the description notes that “Mustafa volunteers to deliver groceries [in “location”]. Click to support refugees helping us,” but randomly assigns users to view “location” as “the USA,” “Pennsylvania,” “Lancaster,” or to see no mention of geography.

Figure 1.

Example of Facebook advertisement for Condition 1 in Table 1.

The second experiment references Dr. Heval Kelli, a refugee from Syria who now treats coronavirus patients (see Figure 2). All five ads feature the same picture of Dr. Kelli in his medical coat. The ad headlines display some version of “Refugee doctors are fighting coronavirus.” To test H1, we vary whether the headline says “Refugee doctors are fighting coronavirus” or “Refugee doctors are helping America.” To test H3, we vary whether the headline starts with “Refugee doctors” or “Immigrant doctors.” Additionally, the ad description says some version of “Dr. Heval Kelli fights for his coronavirus patients. Click to support refugees helping us.” We drop “coronavirus” if the virus is not mentioned in the headline. We replace “refugees” with “immigrants” when the headline includes immigrants, and with “doctors” when the headline mentions just doctors. The narratives in the two experiments are not fictional as they contain real information about the actual actions of these two individuals.

Figure 2.

Example of Facebook advertisement for Condition 6 in Table 1.

The two experiments use different target audiences on Facebook. For our first experiment about Mustafa, we specified that any adult Facebook profile within 35 miles of Lancaster, PA – the city where Mustafa lives – could be included in the split test. This provides a potential audience of 450,000 profiles and allows us to test H2 about whether the refugee’s actions are taking place within the respondent’s local community. For our second experiment about Dr. Kelli, we specified that any adult Facebook profiles in the USA could be included in the split test. This provides a significantly larger potential audience of 160,000,000 profiles. Of these potential profiles, the actual number who viewed our ads was determined by our budget.

Table 1 above describes our full set of experimental conditions. The Facebook platform allows only a maximum of five treatment arms per study. Study 1 (conditions 1 through 5) allows us to test H1 and H2. Study 2 (conditions 6 through 10) allows us to test H1 and H3. All ads appear in Online Appendix C.

Table 1.

Facebook Experimental Design

We conduct simulations of the proposed design as detailed in Online Appendix B to assess power and find our estimators likely to be well-powered (above 0.80) for the estimands of interest.

Results of preregistered analysis

This study was a preregistered report conditionally accepted by the Journal of Experimental Political Science and uploaded with the EGAP/OSF Registry (https://osf.io/bc2m8). The rollout of our experiment took longer than expected: we fielded the study later in the Summer of 2020 than we originally planned, but with no apparent implications for our research design or results. Facebook blocked our ads several times for purported policy violations that did not appear relevant to our situation. We struggled to receive clarification from Facebook but were eventually able to implement the study after linking the ads to a Facebook ad account with a longer history of running ads on political and social issues. We also made slight changes to implementation, deviating from the Stage 1 Registered Report. We had initially planned to run our ads in the right-hand column, so that ad recipients could not “engage” with the ads by liking or commenting on them. However, as of the time of implementation, Facebook did not allow ads on social and political issues to be run in this position, requiring us to distribute ads to the news feed instead. This change meant slight modifications to how ads looked to respondents. It also gave us longer character limits for the ad text, which allowed us to include “Pennsylvania” in our test of H2 instead of “PA” as we had preregistered. We felt this adjustment provided a cleaner test of H2. Running ads in the news feed also enabled respondents to like or comment on the ads, but the change should not have affected our ability to test the hypotheses. ^{Footnote 3}

Table 2 summarizes the number of impressions (total number of times the ad was shown on a user’s screen), reach (number of unique profiles that viewed the ad), and results (total number of user clicks) for each ad. The Mustafa ads were distributed to 130,367 total Facebook users in the vicinity of Lancaster, PA and the Kelli ads were distributed to 229,695 total Facebook users in the USA. Click rates ranged from approximately 1.9% to 2.9%, a fairly high range for Facebook ads, which typically generate a click rate of about 1% (Irvine Reference Irvine2020). We reran our power analyses using actualized sample sizes and find that our design is still well-powered.

Table 2.

Summary Statistics per Ad

The click rates for the Mustafa and Kelli ads are displayed in Figure 3, and in Figure 4 we present results from our hypothesized difference-of-means tests for H1 in the left panel, H2 in the middle panel, and H3 in the right panel. We apply Benjamini–Hochberg corrections for multiple hypothesis testing as preregistered in Stage 1. Recall that H1 was tested in the context of both the Mustafa (everyday refugee) and the Kelli (refugee doctor) ads. None of the results are statistically significant at conventional levels. While the top and middle coefficients are in the expected direction, the bottom coefficient is not. As such, we do not find support for H1: in a variety of different contexts, mentioning COVID does not increase click rates.

Figure 3.

Click rates for ads. Experiment 1 Mustafa ads are on the left-hand side, while Experiment 2 Kelli ads are on the right-hand side. Mean proportions of ad clicks within each arm are presented, with 95% confidence intervals on the mean proportions.

Figure 4.

Ad treatment effects. Each panel represents difference in means tests conducted under H1 through H3 families of hypotheses, where the outcome is whether respondents clicked on the ad. Individual tests are listed on the y-axis and colored according to whether the data used are from the Mustafa (Experiment 1) or Kelli (Experiment 2) ads. Difference in means estimates and corresponding 95% confidence intervals are presented for each test.

In the middle panel, we present results from our hypothesized difference-of-means tests for H2 – the location effect. In descending order, the coefficients compare Lancaster to the USA, Lancaster to no specified location, Lancaster to Pennsylvania, Pennsylvania to the USA, Pennsylvania to no specified location, and the USA to no specified location. None of the results are statistically significant, but they are consistently in the expected direction, with higher click rates in ads that mention a more local connection. ^{Footnote 4} These results suggest that H2 may be worth exploring further in future research.

In the right-hand panel, we present results from our hypothesized difference-of-means tests for H3 – the migrant label effect. The top coefficient compares the ad in which Dr. Kelli is labeled as a refugee doctor to the ad in which he is only labeled as a doctor, the middle coefficient compares the ad in which Dr. Kelli is labeled as an immigrant doctor to the ad in which he is only labeled as a doctor, and the bottom coefficient compares the refugee label to the immigrant label. The refugee label resulted in a higher click rate than the immigrant label, as expected by H3, but this difference was small and insignificant. However, we do find that both the refugee doctor (“COVID-Refugee vs COVID-Neither”) and immigrant doctor (“COVID-Immigrant vs COVID-Neither) ads generated higher click rates than ads in which Dr. Kelli was only described as a doctor (difference of 0.003 with a p = 0.0006 and difference of 0.002 with a p = 0.032, respectively, though the former persists after multiple hypothesis correction and the latter does not).

Results of exploratory analysis

As discussed, we initially planned to implement our study by placing our ads in the Facebook right-hand column, but we were compelled by Facebook to run them in the news feed instead. These ads can be liked or commented on by Facebook users, providing an additional measure of engagement to the click rates. Because we only learned of this requirement after preregistering our study, we did not register these outcomes. The results are relevant to our hypotheses, so we report them here. Specifically, we construct an outcome for the number of positive “reactions” to each ad, which includes the number of likes, loves, or cares left by users who viewed the ads.

The results for the Mustafa ads are shown in the left panel of Figure 5. They indicate that users in the Lancaster and Pennsylvania treatment groups were substantially more likely to leave a positive reaction on the ad. Approximately 0.9% of profiles who viewed these ads registered a positive reaction, vs. 0.4–0.5% of profiles in the ads that mentioned the USA or no specific location. These results are consistent with H2 and the direction of the click-rate effects reported in Figure 4, suggesting that a more local connection can increase pro-refugee engagement.

Figure 5.

Positive reactions to ads. Positive reactions are defined by “like,” “love,” and “care” Facebook reactions (under Facebook settings, each user is constrained to one possible reaction). Mustafa ads are presented on the left side panel and Kelli ads on the right side panel.

The results for the Kelli ads are shown in the right panel in Figure 5. Here, the results provide evidence consistent with H1. The ad that mentions COVID and labels Dr. Kelli only as a doctor received positive reactions from 1.6% of viewers, compared to a reaction rate of 0.8% for the ad that did not mention COVID and labels Dr. Kelli only as a doctor. Likewise, of the two ads that labeled Dr. Kelli as a refugee, the ad that mentioned COVID had a reaction rate of 0.7% and the ad that did not mention COVID had a reaction rate of 0.5%. These differences are statistically significant.

On the other hand, the ad reactions differed from the ad clicks with regard to H3. Here, the immigrant doctor label produced a higher reaction rate than the refugee doctor label, and the ads with the doctor only label produced higher reaction rates than the immigrant doctor or refugee doctor labels. One interpretation of this divergence is that reactions and clicks capture different types of engagement with the ads. The reactions are a public endorsement of the ad content, while the clicks are more likely to capture a private desire to learn more about how to help the relevant group, given the wording of our ads. It is plausible that individuals would be more willing to show public support for a doctor generally as opposed to a doctor defined as an immigrant or refugee, but that they would be more motivated to seek additional information about helping refugees and immigrants when the doctor was defined as part of those specific groups. This interpretation remains speculative and requires further research that explicitly tests the difference between clicking through to an ad vs. engaging publicly with an ad.

As mentioned in our description of the research design, the landing page for our ads was a Qualtrics survey about attitudes toward migrants and COVID-19. We hoped to use this survey to analyze further how Facebook users were engaging with our ads. However, only 40 users who clicked on the ads completed the survey, making this analysis infeasible.

Conclusion: ethical considerations and interpretation of effects

Our research design and study was approved by the Institutional Review Boards at each author’s institution. Any social science experiment that manipulates a user’s experience on social media should be the subject of ethical scrutiny. Indeed, social scientists raised a number of red flags upon the publication of the results of an experiment that manipulated the extent to which more than 680,000 people were exposed to emotional expressions in their FB news feed (Kramer, Guillory, and Hancock Reference Kramer, Guillory and Hancock2014). That study tested emotional contagion online, manipulating users’ exposure to positive and negative emotional content and then measuring their own online posting behavior. Here, we follow Albertson and Gadarian’s framework for thinking through the ethical considerations in this study. ^{Footnote 5}

1. 1. Was there consent? The emotion study never obtained informed consent from its users, such that more than 680,000 subjects in the study never knew – and never will know – that they were part of a scientific study. This was particularly problematic at the time the study was conducted because Facebook’s Data Use Policy did not mention anything to users about their data being used for research. Yet, partly as a result of the uproar over the ethics of that particular study, Facebook has since updated its Data Use Policy to explicitly include research. ^{Footnote 6} Still, some argue that this is not enough because Facebook’s Data Use Policy does not communicate in a responsible and accurate manner, opting for “full disclosure saturation of information” instead (Flick Reference Flick2016). Although we have no control over the content or format of Facebook’s Data Use Policy, and cannot seek informed consent at the moment we introduce the ads into the user’s feed, we do immediately seek informed consent as soon as the user clicks on the ad. We therefore have two groups of subjects. The subjects who never click on the ad or who click and refuse to give consent, and the subjects who click and give consent. For the former, we never obtained individual-level data: the Facebook A/B testing feature will only allow us to conduct difference-of-means tests. And for those users whose individual-level data we did collect, we have obtained informed consent.

2. 2. Was there deception? In our study, users are never exposed to deceptive content: the ads themselves are all truthful, obtained with consent from Refugee International, RCUSA and from the individuals in the ads themselves. Additionally, organizations like RCUSA already engage in the kind of online outreach we are testing, communicating narratives of refugee solidarity on social media outlets such as Twitter and Facebook. We have checked with our partner organizations and they have confirmed that they are already buying FB ads to promote refugee issues. Finally, deviating again from the Facebook emotion experiment, our ads clearly state that they are sponsored by an organization: users explicitly know they are seeing paid ads.

3. 3. Does the Facebook study cause harm? This question is a legitimate concern in the Facebook emotion experiment, because the study manipulated emotions both in the positive and in the negative direction. By contrast, our intervention presents only positive stories of individuals helping out in the time of COVID19.

In designing our study, we were acutely aware of the ethical considerations involved in an online Facebook experiment. We believe we defined a scope of study and took the necessary precautions to abide by our discipline’s ethical standards.

Beyond the ethics of the research design, we add one more point of ethical consideration related to the theory and findings of the study. Our hypotheses build on the idea that the contributions migrants make to society will increase engagement with refugee advocacy organizations, particularly in a time of national crisis. However, while these narratives of refugees as heroes may increase engagement, they may also create or sharpen the distinction between refugees with the capabilities to contribute and those who may not have such capabilities. Particularly concerning is the possibility that such narratives create an expectation of refugees as heroes and deserving of American altruism and support. In the long run, this could be detrimental as it may undermine a broad commitment to the original purpose of the UN Refugee Convention to support refugees regardless of their ability to contribute to their host society. Thus, we suggest future work should seek to explore ways in which inclusive attitudes toward refugees can be increased without relying on promoting only an ideal refugee type.

In summary, the COVID-19 pandemic presented both an opportunity and a risk to organizations advocating on behalf of migrant communities in the USA. We partnered with refugee advocacy organizations to study whether engagement with these organizations could be increased using narratives of refugees helping society during the pandemic. We developed three hypotheses related to anxiety about the pandemic (H1), geographic proximity of the helping behavior (H2), and the perceived vulnerability of the helper whether refugee, immigrant, or neither (H3). Our innovative research design used experiments embedded in advertisements on Facebook to test the hypotheses.

Our results suggest avenues for increased engagement as well as dead-ends. Using preregistered classical t-tests, we find no support for the idea that mentioning the coronavirus pandemic in our ads explicitly increased engagement on Facebook compared to ads that did not mention the pandemic. ^{Footnote 7} Organizations do not seem to receive an additional boost in engagement from supporters by mentioning coronavirus directly in the ad. We suggest this may be because at the time our ads appeared on Facebook – in August 2020 – COVID-19 was already exceptionally salient: whether or not our ads mentioned coronavirus, individuals in both treatment and control would view the ads with the coronavirus pandemic already on their minds. More generally, these results imply that during extremely salient national or international events, there may be no additional impact of including language about the event explicitly in public relations materials.

On the other hand, the results for H2 and H3 do suggest plausible strategies for organizations to increase involvement. As regards H2, the preregistered and exploratory analysis both point in the direction that “going local” is effective at increasing engagement. When refugees are described as helping in the participants’ city, these participants are more likely to click on the ad or engage positively. Together with recent work showing that local officials are more receptive to local refugee resettlement when refugees are perceived as contributing to their local community (Shaffer et al. Reference Shaffer, Pinson, Chu and Simmons2020), our results suggest that emphasizing geographically narrow contributions to their relevant audience may effectively mobilize support for refugees. Similarly, the tests of our third hypothesis showed that identifying doctors as either immigrant or refugee increased engagement relative to an ad that did not use either of those terms. We take this as some evidence that migrant advocacy organizations do not need to shy away from including these terms in their advertisements and that relative to more neutral phrasing, these descriptors may actually increase engagement with their cause.

The effect of the COVID-19 pandemic on prejudice and inclusion of the most vulnerable groups among us is not well known. Our study tested the effectiveness of a series of narratives on which refugee advocates rely to boost public engagement with refugee advocacy organizations. Our results confirm that narrative frames matter for public engagement. But the combination of our registered and exploratory results also raise an important avenue for future research: audiences engage in different ways with the efforts of refugee advocates, and some of these are more publicly visible than others. Refugee advocates may want to consider the ways in which the public visibility of a response might interact with various migrant frames to boost public engagement with their cause.