1. Conceptual framework: defining and measuring sources of survey error in public opinion samples

Our definition of survey quality builds on prior assessments of cost-quality trade-offs in public opinion samples. Scholars have assessed the quality of samples recruited through Amazon’s Mechanical Turk (MTurk) (Berinsky et al., Reference Berinsky, Huber and Lenz2012; Huff and Tingley, Reference Huff and Tingley2015), Prime Panels (Litman et al., Reference Litman, Robinson and Abberbock2017), Lucid (now Cint) (Coppock and McClellan, Reference Coppock and McClellan2019; Ternovski and Orr, Reference Ternovski and Orr2022), Google Consumer Surveys (Santoso et al., Reference Santoso, Stein and Stevenson2016), and Facebook Advertising (Kosinski et al., Reference Kosinski, Matz, Gosling, Popov and Stillwell2015; Jäger, Reference Jäger2017; Boas et al., Reference Boas, Christenson and Glick2020; Zhang et al., Reference Zhang, Mildenberger, Howe, Marlon, Rosenthal and Leiserowitz2020; Neundorf and Öztürk, Reference Neundorf and Öztürk2023). Much of this work focuses on the quality of U.S. online samples, typically by comparing sample demographic statistics with the U.S. Census (Berinsky et al., Reference Berinsky, Huber and Lenz2012; Huff and Tingley, Reference Huff and Tingley2015; Coppock and McClellan, Reference Coppock and McClellan2019; Zhang et al., Reference Zhang, Mildenberger, Howe, Marlon, Rosenthal and Leiserowitz2020). These assessments are relatively straightforward in the U.S., where high-quality census data are frequently updated and many probability and quota samples are available as benchmarks.

We extend this work to three settings in the Global South, while also conceptually disaggregating and empirically assessing the types of errors that threaten the validity of conclusions drawn from Facebook surveys. We define error as unobserved disturbances that influence a statistical quantity of interest. Such error is problematic if it causes survey estimates to differ systematically from true population parameters. These systematic differences reflect bias in our estimates.

We use the Total Survey Error framework, first developed in the 1940s (Deming, Reference Deming1944) and used by modern survey researchers (e.g.,Ansolabehere and Schaffner, Reference Ansolabehere and Schaffner2010; Groves and Lyberg, Reference Groves and Lyberg2010; Groves et al., Reference Groves, Fowler, Couper, Lepkowski, Singer and Tourangeau2011; Lyberg and Weisberg, Reference Lyberg, Weisberg, Wolf, Joye, Smith and Fu2016), to define the distinct sources of error that threaten the external validity of conclusions derived from Facebook-recruited surveys. Since the framework’s introduction, survey methodologists have defined two groups of quality indicators for survey statistics: measurement errorFootnote ¹ and representation error (Groves and Lyberg, Reference Groves and Lyberg2010; Groves et al., Reference Groves, Fowler, Couper, Lepkowski, Singer and Tourangeau2011). Here we focus on the latter. Errors of representation are systematic or random imperfections in the relationship between a target population, sampling frame, and sampling units, and they threaten external validity.

Figure 1, adapted from Groves et al. Reference Groves, Fowler, Couper, Lepkowski, Singer and Tourangeau2011 and Groves and Lyberg (Reference Groves and Lyberg2010), shows the inferential steps required to draw population conclusions from a Facebook-recruited survey sample. First, coverage error is defined as the gap between a target population and a sampling frame. In our case, the target population is a country’s national adult population, and the sampling frame is the national adult population with a Facebook account in that country. The existence of adult residents without Facebook accounts will generate “undercoverage.” Coverage bias is quantified as the difference between the mean value of a descriptive statistic in the national population and the Facebook population. Second, sampling error arises because not all individuals in the sampling frame are surveyed. Random sampling variation occurs since many different sets of individuals could be drawn from the sampling frame, simply by chance. Sampling bias arises when individuals in the sampling frame have unequal chances of being included in the sample. Third, unit nonresponse error arises when some sampled individuals fail to record meaningful and complete survey responses. Unit nonresponse is the gap between people who click on the advertisement for our survey and those who finish the survey. Nonresponse bias arises if certain types of people are more likely to finish the survey than others, and if there is a relationship between the likelihood of finishing a survey and survey responses. Finally, adjustment error arises when researchers weight data to give greater representation to cases that are underrepresented in the sample. These weights are used to reduce coverage, sampling, and nonresponse bias, but they can also increase these biases (Bailey, Reference Bailey2024).

Figure 1.

Components of representation error in Facebook samples.

We examine total survey error and disaggregate its components with two sets of empirical analyses. Our analyses build on prior work that compares survey-derived statistics to external benchmarks, such as a “gold-standard” survey, census, or re-interview data (Ansolabehere and Schaffner, Reference Ansolabehere and Schaffner2010; Groves and Lyberg, Reference Groves and Lyberg2010; Berinsky et al., Reference Berinsky, Huber and Lenz2012; Kennedy et al., Reference Kennedy, Hatley, Keeter, Mercer, Igielnik and Traylor2018; Coppock and McClellan, Reference Coppock and McClellan2019; Zhang et al., Reference Zhang, Mildenberger, Howe, Marlon, Rosenthal and Leiserowitz2020; Holliday et al., Reference Holliday, Reny, Rossell Hayes, Rudkin, Tausanovitch and Vavreck2021). We compare, first, the demographic compositions of our samples and, second, the public opinions measured in our surveys with those derived from other high-quality samples. Our disaggregated assessment of survey error allows researchers to gauge Facebook’s suitability for specific research applications and highlights ways to minimize biases.

2. Data collection

3. Comparing demographics

To evaluate the quality of the statistical summaries derived from Facebook samples, we first examine total survey error by comparing the demographic characteristics of our samples with benchmark surveys and national census data. Our quantity of interest is the weighted sample mean survey response.

Figure 2 plots the distribution of demographic characteristics, compared with benchmark surveys and national census data. In all three countries, the weighted Facebook samples (filled, dark blue squares) differ most from the census (gold crosses) with respect to age and education. In Kenya and Indonesia, Facebook survey samples are younger than the national population and the barometer samples (red triangles). By contrast, the Mexican Facebook sample (similar to LAPOP) contains a greater share of respondents over 50, compared with the national population.

Figure 2.

Demographic benchmark comparisons. Note: Means are plotted with 95% confidence intervals for demographics reported in the national census (no confidence intervals), nationally representative survey samples (LAPOP/Afrobarometer/Asian Barometer), the Facebook population (no confidence intervals), our original survey recruited with Dynata, and our Facebook samples. Facebook samples are weighted using raking to match the national census on gender, education, age cohort, and geography.

In general, Facebook samples are more educated than national populations and the barometer surveys. However, the Indonesian Facebook sample is less biased on education than the commercial online sample. The commercial online sample also underrepresents lower education respondents more severely than our Facebook sample.

We find little gender bias across all three countries. The distribution of religions in Kenya and Indonesia is also fairly representative, although some bias is evident in Mexico. Marital status in Mexico and Indonesia, and tribe in Kenya, also show minimal bias. However, there is substantial bias toward urban populations in Kenya.

3.2. Sampling error

While quota-based sampling can help researchers mitigate coverage bias, we need to examine whether the Facebook ad platform introduces sampling bias by limiting researchers’ ability to reach particular groups. Sampling error arises because only some of the individuals in the sampling frame (Facebook users in each country) are included in the potential sample (Facebook users who are shown the ad and click on it). Sampling bias could arise at two points in the sampling process. First, ad design may appeal to some individuals more than others. Researchers have control over this step. They can, for example, experiment with the ad design to appeal to different respondent types. Second, the Facebook advertising platform may systematically fail to reach some individuals. While researchers can target certain individuals with recruitment quotas, Facebook’s back-end data and algorithms (which are beyond researchers’ control) determine whether ads reach targeted individuals.

To assess sampling bias, we focus on individuals who clicked our survey ad (“ad clickers”). We can observe Facebook’s back-end demographic information for ad clickers, even if they did not complete the survey, because the survey records the sampling stratum of the ad they clicked on. We use this Facebook-inferred demographic information to check whether we recruited at least one individual from each stratum and, conversely, whether any individual types were systematically excluded. In Kenya and Mexico, we received responses from every quota, although in Kenya we did not reach quota targets in 49 of 66 strata.Footnote ³ In Indonesia, we were unable to recruit respondents from 67 (25%) of 272 targeted strata. The vast majority of these strata contained men and women over 50 years old, though we also failed to reach women between 30 and 49 years in three provinces. This suggests that in Indonesia, sampling bias contributes to the underrepresentation of older individuals in our sample.

Of course, if Facebook’s back-end data are inaccurate, then even perfect success at recruiting people from all survey strata will not ensure that all individuals have a chance of being sampled. To examine this possibility, we compare self-reported demographics with those reported by Facebook. Table 1 provides the percentage of people in each category for which self-reported and Facebook-targeted characteristics match.

Table 1.

Accuracy of Facebook targeting, as defined by the percent match between Facebook- and self-reported data

Note: *In Kenya, a match was defined as whether respondents’ self-reported age was 32 years old or above, for those who responded to an ad targeting this age group. The self-reported ages of these respondents ranged from 19 to 48 years old, with a mean of 31.

$^{\ddagger}$In Kenya, a match was defined as whether self-reported educational attainment was secondary school or less, for individuals who responded to an ad targeting respondents with an “unspecified” level of education.

$\diamond$For location matches, we aggregate self-reported and targeted location to the largest administrative unit used for targeting. In Kenya, this corresponds to the former provinces used as administrative units, matched from the county respondents say they live in and the locations used for ad targeting. For Indonesian respondents, we checked whether respondents self-reported living in the same province as the targeted province for the ad through which they were recruited. For Mexican respondents, we checked whether respondents self-reported living in the same group of municipalities targeted by the ad through which they were recruited. The “Recruitment and sampling” section of the SI provides additional clarity on the administrative units used for sampling in each country.

The accuracy of Facebook’s ad targeting varies across demographics and between countries. Facebook’s targeting was remarkably accurate in correctly identifying respondents’ gender in Mexico and Kenya (99% and 91% match, respectively) and slightly less accurate in Indonesia (76%). In both Indonesia and Kenya, the 10–20% of respondents recruited from an ad targeting the opposite gender might have resulted from respondents sharing ads with friends. It would not be surprising that greater sharing would have occurred in the context of the incentivized surveys in Kenya and Indonesia, compared to the non-incentivized survey in Mexico. Targeting by age was quite accurate in Mexico (87% match) and Indonesia (77% match), but not in Kenya (44%). Geographic targeting was similarly accurate in Mexico and Kenya but less so in Indonesia. Furthermore, education targeting was not very successful in Kenya, which is in part due to the fact that we attempted to reach less-educated respondents by targeting those with an “unspecified” level of education—meaning they had not reported their level of education on their profile. In contrast, education targeting in Mexico was reasonably accurate. We targeted respondents with no higher than a high school degree, and 70% of our respondents self-reported an education level consistent with this targeting criterion. We report more details on these comparisons in the SI.

These findings indicate that inferred demographics are not always accurate, highlighting the importance of collecting self-reported demographic data. We also recommend that researchers periodically examine the composition of the sample of ad clickers to gauge success at recruiting respondents from each quota (see SI Figure S4 for an example). To increase entrants from underrepresented groups in the sample of ad clickers, researchers could increase spending for these groups’ ad sets, modify ads to increase appeal for these groups, or leave ads running for a longer period.

Overall, quota sampling helps us recruit a more diverse sample of respondents. Left to its own devices, Facebook’s algorithm would optimize ad targeting to recruit the least expensive sample. This optimization entails recruiting respondents that are most similar to those who have already entered the survey. Running an ad campaign across quota sampling cells weakens this regression toward the most prevalent (or cheapest) respondent types. Still, the errors in Facebook’s back-end inference of demographic attributes, as well as the fact that some people are excluded from the Facebook platform altogether, mean that researchers cannot fully overcome the gap between the Facebook and national populations through quota sampling alone.

3.3. Nonresponse error

Unit nonresponse is common across survey contexts and causes bias when it is nonrandom and correlated with survey answers (Bailey, Reference Bailey2024). We define attrition and nonresponse interchangeably, as entering the survey but failing to provide meaningful and complete responses.Footnote ⁴ Overall, the attrition rate was 46%, 31%, and 25% in Mexico, Indonesia, and Kenya, respectively. For context, attrition from the online sample for the American National Election Study was 14% in 2020 (DeBell et al., Reference DeBell, Amsbary, Brader, Brock, Good, Kamens, Maisel and Pinto2020, p. 74), and attrition from the Cooperative Election Study was 21% in 2022 (Ansolabehere et al., Reference Ansolabehere, Schaffner and Shih2023, p. 11).Footnote ⁵ We examine whether nonresponse is systematic using the same Facebook-assigned demographic information used in the previous section. We first assess whether the likelihood of entering but dropping out of the survey was correlated with users’ demographic characteristics (assigned by Facebook). Figure 3 shows that attrition is systematic, but the predictors of attrition vary across samples.

Figure 3.

Predictors of nonresponse. Note: The figure shows the results from a linear regression of attrition (attrition = 1, completion = 0) on demographic characteristics used by Facebook to target individuals. Omitted categories are Female and 18–29 for the Mexican regressions, Female and 21–29 for the Indonesian regressions, and Female and younger than 32 years old for the Kenyan regressions. 95% confidence intervals are reported using heteroskedasticity-robust standard errors.

Incorporating age, gender, and education into adjustment weights helps to alleviate nonresponse bias associated with these observable characteristics. We reduce nonresponse bias by upwardly weighting individuals with underrepresented characteristics and down-weighting those with overrepresented characteristics. Researchers could also adjust their sampling strategy on the fly to improve response rates among high-attrition groups. For instance, researchers might increase ad spending or introduce incentives for high-attrition groups. Of course, unobserved sources of nonresponse present a more worrisome problem for descriptive inference, since researchers cannot adjust for unobserved and non-ignorable sources of nonresponse. To account for non-ignorable nonresponse, scholars can use bounds (Manski, Reference Manski1990), sensitivity analysis (Hartman and Huang, Reference Hartman and Huang2024), selection models (McGovern et al., Reference McGovern, Canning and Bärnighausen2018; Gomes et al., Reference Gomes, Radice, Camarena Brenes and Marra2019), nonresponse weights (Sun et al., Reference Sun, Liu, Miao, Wirth, Robins and Tchetgen2018), or other methods (Bailey, Reference Bailey2024).

4. Replicating classic survey-experimental findings

Most social science researchers are primarily interested in public attitudes or behaviors, rather than demographic summaries. We now turn to these quantities of interest. As an initial check on the face validity of survey results derived from our samples, we use a canonical behavioral experiment—the Tversky and Kahneman (Reference Tversky and Kahneman1981) “disease problem” used to test prospect theory. Since other low-cost platforms have been shown to produce reliable replications of this and other survey-experimental findings (Berinsky et al., Reference Berinsky, Huber and Lenz2012; Mullinix et al., Reference Mullinix, Leeper, Druckman and Freese2015; Coppock and McClellan, Reference Coppock and McClellan2019), we include this analysis as a low-bar, benchmark test with which many scholars are already familiar.

This survey experiment asks respondents to “Imagine that your country is preparing for the outbreak of an unusual disease, which is expected to kill 600 people. Two alternative programs to combat the disease have been proposed.” Then respondents are randomly assigned to see two program options which are framed either in terms of the number of lives that will be saved (the “save” condition) or the number of lives that will be lost (the “die” condition) under the two options. Within each condition, the difference between the two programs is whether the outcome is stated in certain or probabilistic terms. In expectation, the payoffs of both policies are equal. The finding that has been replicated in many samples is that respondents are more likely to choose the probabilistic (risky) option to avoid losses (in the “die” condition), and they are more likely to choose the certain option to accrue gains (the “save” condition).

Table 2 shows the original results from Tversky and Kahneman (Reference Tversky and Kahneman1981) among their sample of U.S. students, a replication by Berinsky et al. (Reference Berinsky, Huber and Lenz2012) using a U.S.-based MTurk sample, and results from our Facebook samples in Kenya and Indonesia.Footnote ⁶ Our results are largely consistent with these studies and others that have replicated this experiment across cultural contexts (Ruggeri et al., Reference Ruggeri, Alí, Berge, Bertoldo, Bjørndal, Cortijos-Bernabeu, Davison, Demić, Esteban-Serna, Friedemann and Gibson2020; Im and Chen, Reference Im and Chen2022). For instance, Im and Chen (Reference Im and Chen2022) report that the percent of respondents who pick the certain option when framed in a positive (save) vs. negative (die) framing in South Africa is 57% and 37%, in Mexico is 59% and 37%, and in Indonesia is 51% and 33%.

Table 2.

Replication of Tversky and Kahneman (1981) disease problem

Note: The table shows the proportion of respondents choosing “certain” and “risky” policies to manage a disease, when the policies are framed in terms of lives saved vs. deaths. When the policies are framed in terms of the number of lives saved, a majority of respondents prefer the certain policy. When the policies are framed in terms of the number of people who will die, the majority prefers the risky option.

These preferences manifest in the same direction in our samples, with one notable exception: the application of weights skews the result in the “die” condition in Indonesia. Table S3 shows that the unweighted result is consistent with our expectations in the Indonesian sample. However, the result does not hold for respondents over the age of 50 (Table S4), which suggests that the older individuals in our sample are distinctive in how they respond to this experiment. Recall from Section 3.2 that we were unable to recruit respondents over the age of 50 from several of the strata in our sample (sampling bias). Up-weighting these individualsFootnote ⁷ in our estimates of experimental effects pulls the average effect toward the biased (relative to the benchmark) result among older individuals. This reflects adjustment bias. We highlight this finding because it reinforces our suggestion (introduced in Section 3.2) to monitor sampling quotas and dedicate resources to ensure sufficient recruitment within all strata. The finding also cautions against the uncritical use of weights to address underrepresentation of certain groups.

5. Comparing public opinion estimates

We next report descriptions of public opinions and political behaviors. We compare responses from our Facebook samples to identical opinion questions fielded in benchmark surveys (Figure 4). In general, where there are gaps between the Facebook-derived estimates and benchmark samples, Facebook samples report greater political activity. Note that the application of weights tends to move Facebook-derived estimates toward benchmark survey estimates. Applying weights tends to improve the accuracy of the estimates.

Figure 4.

Political attitudes and behaviors. Note: This figure shows self-reported behaviors including identifying with a political party, voting, and engaging in activities such as community meetings and protests, from Afrobarometer/Asian Barometer and Facebook samples in Kenya and Indonesia. Estimates for both samples have been weighted using individual weights provided by Afrobarometer/Asian Barometer surveys or the raking procedure described above (for Facebook samples).

Persistent gaps in Figure 4 are likely attributable to some combination of sample composition and survey mode. Representation bias could account for some upward bias in estimates of political activity, since Facebook samples overrepresent highly educated respondents who are likely to be more politically engaged (Verba et al., Reference Verba, Schlozman and Brady1995). Still, survey mode could also play a role here. In an in-person survey context, social desirability bias may deter respondents from reporting activities, such as protesting, that challenge the government. Consistent with this tendency, barometer survey respondents report lower levels of these types of activities and, in Kenya, higher levels of approval of the President. Conversely, evidence suggests that turnout is overreported in surveys (Holbrook and Krosnick, Reference Holbrook and Krosnick2010), and this bias in self-reporting may be larger for in-person surveys (Jackman and Spahn, Reference Jackman and Spahn2019). Accordingly, voter turnout is the only place where Facebook samples report lower activity than corresponding barometer estimates. Moreover, the Facebook sample estimates are closer to the official turnout figures reported in each country (indicated by the green crosses in Figure 4).

Consideration of political context reinforces the idea that different estimates of political activity may be in part attributable to mode effects. The Asian Barometer survey was fielded in July 2019 in Indonesia, just a few months after President Joko Widodo’s reelection sparked protests and claims of election fraud from the opposition Gerindra party candidate (Suhartono and Victor, Reference Suhartono and Victor2019). In this context, only 1% of Asian Barometer respondents reported affiliation with the Gerindra party candidate, who received 44% of the vote. In the Facebook sample, many more respondents (16%) reported affiliation with the Gerindra party. Respondents may have been less willing to express their true political views to an enumerator.

To illuminate the platform’s utility for assessing policy views, we examine environmental and climate change opinions in Mexico and Indonesia. In Mexico, we replicated the question wording and response options from a LAPOP question regarding whether economic growth or environmental protection should be prioritized. Facebook respondents were more likely than LAPOP respondents to answer that environmental protection should be given the highest priority (44% vs. 20%) and less likely to choose responses at the economic-growth end of the Likert scale (9% vs. 18%) (Figure 5, left panel). In Indonesia, we asked multiple questions about climate change to our Facebook-recruited respondents and to respondents recruited from Dynata’s online panel. Facebook respondents were less worried about climate change and less likely to understand that it is human-caused, compared with the Dynata sample (Figure 5, right panel). These differences likely stem from sample composition. The Facebook sample is more highly educated than the LAPOP sample in Mexico but less highly educated than the Dynata sample in Indonesia, and education is positively correlated with environmental concern in global surveys (Lee et al., Reference Lee, Markowitz, Howe, Ko and Leiserowitz2015).

Figure 5.

Public opinion estimates. Note: This figure shows responses from LAPOP and Facebook samples, to the question of whether environmental protection (1) or economic growth (7) should be given priority. Facebook results are weighted, and LAPOP results are unweighted, consistent with LAPOP documentation.

6. Survey costs

Researchers typically weigh survey quality concerns against the relative costs of different survey tools, and we find that Facebook sampling is quite cost-effective. The costs of Facebook sampling include platform advertisements and (optional) incentives paid to respondents. Not including incentives, the mean cost per completed survey was $0.16 in Mexico, $0.85 in Kenya, and $0.91 in Indonesia.Footnote ⁸ Including incentives, the surveys cost an average of $1.03 per completed survey (ranging from $0.16 in Mexico to $1.57 in Indonesia). This is incredibly inexpensive in Mexico. In Kenya or Indonesia, the cost is comparable with the cost of recruiting online convenience samples using platforms such as MTurk or Cint. Of course, MTurk, Cint, or similar platforms do not enable researchers to contact survey respondents in every country, and their user populations are much smaller than Facebook’s. Realistically, in-person field surveys or online panels provide the most feasible alternative in most parts of the Global South. The costs of our Facebook samples are substantially cheaper than these alternatives. For instance, the Indonesia Dynata sample (n = 1,130) cost $5.75 per completed survey. In-person surveys are even more expensive to field.

7. Conclusion

Facebook is the most popular social media platform in the world with 3 billion monthly users as of 2024 (Dixon, Reference Dixon2023). It’s extensive user base presents an opportunity for researchers to quickly and reliably recruit subjects globally, including from contexts underrepresented on existing online subject recruitment platforms. Our replication of the “disease problem” experiment suggests that, similar to other low-cost online survey platforms (Berinsky et al., Reference Berinsky, Huber and Lenz2012; Mullinix et al., Reference Mullinix, Leeper, Druckman and Freese2015; Coppock and McClellan, Reference Coppock and McClellan2019), Facebook is a cost-effective tool for recruiting survey-experimental subjects. The platform’s near-global reach makes it particularly useful where other online recruitment platforms are unavailable. Our analysis evaluates the quality of Facebook-recruited samples in three countries across the Global South, highlighting some of the advantages and shortcomings of this method and providing practical guidance to researchers.

We have assessed total survey error and its components, to provide practical insights into where and how researchers can minimize bias in estimates derived from Facebook-recruited surveys. Coverage error favors respondents who have achieved higher levels of education, on average, than the general population. This overrepresentation of highly educated respondents is a structural feature of the Facebook platform in our case countries. By extension, researchers should investigate these imbalances in other target countries before deciding to use Facebook as a survey recruitment tool. Researchers can mitigate bias associated with coverage error by targeting recruitment resources toward respondents who are underrepresented on Facebook. In our case, quota sampling through the ad platform allowed recruitment of a more representative sample than leaving the advertising algorithm to its own devices. However, we do find some evidence of sampling bias in Indonesia, and up-weighting the underrepresented individuals introduces adjustment bias in our experimental results from Indonesia. Together, these findings highlight that researchers should monitor entries into the survey from each stratum and adjust their advertising campaign to ensure recruitment in all strata. Still, the back-end demographic data that Facebook uses for targeting are noisy, particularly for education and location. As a result, quota sampling cannot substitute for the use of design weights based on self-reported demographics. In general, weights reduced demographic imbalances and affected descriptive inferences only slightly. Our weighted estimates tended toward benchmark estimates, compared with unweighted estimates. Researchers should also assess nonresponse rates across quota cells during data collection to inform an iterative sampling strategy that maximizes response rates among high-attrition groups.

Practically, Facebook recruitment costs can be quite low but vary by targeting strategy and survey incentives. In Mexico, we recruited respondents without incentives, while in Kenya and Indonesia we provided a modest airtime credit to encourage participation. This incentive may have encouraged greater participation among resource-constrained respondents, but it also led to instances of gaming and link sharing. Future research could explore sampling strategies that leverage Facebook’s social nature through snowball sampling.

The effectiveness of Facebook recruitment for social science research in a particular context depends on several factors. The platform will be particularly successful where (1) phone and internet penetration are widespread, (2) literacy rates are high, and (3) recent census (or other benchmark) population data are available to enable weighting. Researchers should also consider how users’ interactions with and expectations of Facebook might influence internal validity. While we tested this method in competitive democracies, concerns about government surveillance might influence responses in authoritarian states. Researchers should consider these concerns in the survey design process, and future research could consider how internal validity varies across political contexts. The popularity of Facebook also changes over time, and the platform may not retain the large and broad user base that makes it an attractive recruitment platform. Changes to Meta’s advertising policies may also change researchers’ ability to target ads. Such changes are beyond researchers’ control but within their ability to monitor in order to use the platform wisely. Moreover, our conceptual framework can be applied to other platforms that may supplant Facebook in the future.

Of course, Facebook should not replace gold-standard, in-person field surveys for recruiting nationally representative samples in the Global South. Our Facebook samples tend to be more educated and slightly younger than the general population, skewing descriptive inferences on political engagement and public policy views. Nonetheless, Facebook outperforms a commercial online survey firm, at considerably lower cost. Researchers can refine their sampling approach, model nonresponse bias, and apply weighting to mitigate sample biases. We further recommend that researchers employ benchmark comparisons similar to those we have performed here. Used effectively, Facebook can be a valuable tool for expanding public opinion research.