Fraud in Online Surveys: Evidence from a Nonprobability, Subpopulation Sample

Abstract We hired a well-known market research firm whose surveys have been published in leading political science journals, including JEPS. Based on a set of rigorous “screeners,” we detected what appears to be exceedingly high rates of identity falsification: over 81 percent of respondents seemed to misrepresent their credentials to gain access to the survey and earn compensation. Similarly high rates of presumptive character falsification were present in panels from multiple sub-vendors procured by the firm. Moreover, we found additional, serious irregularities embedded in the data, including evidence of respondents using deliberate strategies to detect and circumvent one of our screeners, as well as pervasive, observable patterns reflecting that the survey had been taken repeatedly by a respondent or collection of respondents. This evidence offers reasons to be concerned about the quality of online nonprobability, subpopulation samples, and calls for further, systematic research.

These irregularities point to the potential for significant identity misrepresentation rates in online nonprobability, subpopulation surveysrates that are orders of magnitude greater than those typically reported in standard online surveys (Callegaro et al. 2014;Cornesse et al. 2020;Kennedy et al. 2020a;Mullinix et al. 2016). Although online survey firms can vary markedly in their regulation of quality controland all should be assessed for data problems (Kennedy et al. 2016)risks for exceedingly high levels of fraud could be heightened under the conditions present in our study. Our findings call for further, systematic research into the validity of nonprobability online surveys, particularly those that sample specific subpopulations. They also underscore the imperative for researchers to develop clear tools and strategies (prior to statistical analyses and within preregistration plans) to ensure data integrity based on expert knowledge of research subjects.

Survey details
We contracted a nationally recognized market research firm, 1 whose samples have formed the basis of numerous, widely-cited political science studies, including articles published in Journal of Experimental Political Science, American Political Science Review, and Journal of Politics. The firm, which used multiple sub-vendors, fielded the survey over two separate rounds in April-May 2021. 2

Screening process
We employed two screeners to confirm the authenticity of respondents with self-identified Army experience. First, we asked a "knowledge" question about the practice of saluting, one of the most essential elements of military protocol. 3 The question required knowing both the Army's rank hierarchy and that enlisted soldiers salute first. Multiple former Army officers consulted for this study validated the screen, with one stating: "Anyone who is answering that question incorrectly is either not reading the question or has not served in the military, let alone the Army." 4 Second, respondents reported specific information on their Army background including: highest rank achieved, source of officer commission, deployment years and locations, and unit type. 5 We coded "non-viable" responses as those that provided information contravening federal law or Army personnel policy. We coded a small number of responses "highly improbable" based on being contrary to Army personnel practices or historical evidence, but which could be theoretically plausible. 6 1 We withhold the survey firm's name and its associated sub-vendors due to liability reasons. 2 Round 2 was launched after Round 1 was halted due to concerns about data quality. For detailed information on the survey, see Appendix (Supplementary Material).
3 For text, see Appendix. Army rank structure and saluting protocols are so central to Army service that they are among the first subjects recruits learn in basic training (Army, 2019). 4 Email from former US Army major, April 20, 2021. 5 Information such as military rank is so defining of an Army member's career and identityakin to a civilian's jobthat respondents should almost never report these details incorrectly (Bell & Terry, 2021). 6 For examples of "non-viable" and "highly improbable" responses, see Appendix.

Key findings
• Total invalid: 81.8 percent -43.3 percent of total respondents failed the Army knowledge question.
-35.5 percent of respondents passed the knowledge screen but gave answers about Army service that were non-viable under federal law or military administrative rules. -3.0 percent of respondents reported information about an Army background and career that was highly improbable. • Total valid: 18.2 percent Notably, the number of invalid responses remained consistent across multiple vendors, each operating in a different survey round (83.7 in Round 1 and 78.9 in Round 2), suggesting that the problem was not isolated to a specific panel (Table 1).
Additional irregularities 1. Respondents appeared to employ deliberate tactics to circumvent the knowledge screener.
Evidence. Beginning on April 17 in Round 1, the proportion of respondents who passed the knowledge question with non-viable demographic and career answers jumped sharply from 15.4 to 65.9 percent (Table 2). Possible explanations for this unexpected "break" in the data might include collusion (via the sharing of information on an email listserv or crowdsourced website), the retaking of surveys on devices with unique IP addresses, or the use of survey "bots." 2. Respondents appeared to use deliberate tactics to detect the knowledge question as a screener.
Evidence. In Round 1where failure to pass the knowledge question terminated the survey immediately -49.8 percent passed the knowledge question but provided non-viable career responses. Conversely, in Round 2where we modified the setup so that failure resulted in termination later in the survey, thereby "hiding" the screenerthe number of non-viable fell to 13.7 percent (Table 1). Conversely, the percentage of respondents who failed the screen rose from 30.6 percent (Round 1) to 62.7 percent (Round 2). This suggests that, absent a "hidden" screener, later respondents in Round 1 may have gleaned information from earlier respondents or survey attempts to identify the screener and defeat it. 7 3. A respondent, or collection of respondents, appeared to take the survey numerous times.
Evidence. Categorizing conservatively, we identifiedat a minimum -73 suspicious instances of repeated (and unusual) responses regarding Army background and deployment experience. 8 The sequential clustering of these responsessix 7 Duplicate IP addresses accounted for 11 of the total responses. These responses are included in the analysis. distinct waves of repeating answers across the 3 days of responsessuggests that such repetition was not coincidental. We also observed obvious repetitive patterns of survey takers seeming to misrepresent personal demographic information.

Conclusion
We see this analysis as an opportunity for learning. Despite taking precautions to screen out invalid respondents, we found high rates of presumptive fraud. This reinforces that researchers should be especially cautious when employing online surveys using nonprobability samples of specific subpopulations. Given that only about 7 percent of the US population is military or ex-military (Vespa 2020), our results are consistent with incentives for fraud increasing as the size of the subpopulation qualifying to participate in surveys decreases (Chandler and Paolacci 2017). Combined with other techniques, employing a diversity of screeners predicated on expert understanding of research subjectsincluding factors like demographics and content knowledgecan improve the odds of detecting falsified responses. Future research should systematically assess the quality of nonprobability surveys (Hauser and Schwarz 2016;Lopez and Hillygus 2018;Kennedy et al. 2020b;Thomas and Clifford 2017). By implementing rigorous screeners on diverse