Who Participates in Focus Groups? Diagnosing Self-Selection


 Focus groups have become increasingly popular in political science alongside the growth in field experimental and other causal inference-oriented work in comparative politics. Yet, scholars rarely provide details about recruitment processes and descriptive statistics on focus-group participants. This situation is problematic given the likelihood of self-selection and the fact that scholars often use focus groups to pretest or refine experimental treatments or survey questionnaires. By leveraging a series of focus groups that were recruited from a pool of large-N survey respondents, this article demonstrates a method for assessing which variables drive the decision to participate. I recommend that scholars diagnose self-selection into focus groups whenever possible; that they compare participants to relevant baselines when working with samples of convenience; and that they always provide descriptive statistics and details on how focus-group members were recruited.

To assess how scholars publishing in top political science journals use focus groups in their research, I examined all articles published in print in the American Political Science Review, American Journal of Political Science, and Journal of Politics between 2013 and 2022.These journals are consistently ranked as the top three in reputational surveys of political scientists (Garand and Giles, 2003;Garand et al., 2009;McLean et al., 2009).I searched the full text of these three journals (using ProQuest for APSR, Wiley Online Library for AJPS, and University of Chicago Press Journals for JOP) for articles containing the phrase "focus group" or "focus groups."Reviewing the resulting list of articles, I identified 36 that conveyed findings from original focus groups, along with 21 that only use the phrase in passing or in the bibliography or that analyze focus groups conducted by others.
The 36 articles using original focus groups, and the variables on which I coded them, are listed in Table 1.Recruitment Details and Descriptive Statistics report whether the article or its online appendix provide any information about the recruitment process or any descriptive statistics on focus group participants.I also coded whether articles compared focus group participants to a relevant baseline, such as sampling frame, national census, or representative survey, but none did, so this column is omitted.Purpose describes the analytical purpose that focus groups serve: to inform a study's research design, such as developing or pretesting experimental treatments or survey instruments; to triangulate other sources of evidence, including Cyr's (2016) category of "integration"; to measure the study's dependent variable; and to administer an experimental treatment.Combo With describes the main method of the study: field experiment, survey experiment, lab-in-the-field experiment, or quantitative analysis of observational data.Finally, Percent of Text reports what percentage of the article's overall word count conveyed focus group findings and design details; online appendices are excluded from both the numerator and denominator.0.17

Descriptive Statistics
Table 2 contains descriptive statistics on the three focus groups conducted for the Boston Brazilians project.I separately characterize survey respondents who did and did not leave their contact info to potentially receive an invitation to a focus group, and, conditional on receiving an invitation, those who did and did not attend.P-values are for two-tailed difference in means t-tests.

Alternative Specifications
Table 3 shows results from two alternative regression specifications: a linear probability model, and a logistic regression with penalized log likelihood using the Firth (1993) method for bias reduction with rare events (given that relatively few invitees showed up for the focus group).These specifications yield substantively identical results as the conventional logistic regressions reported in the main text.

Note:
Entries are regression coefficients with standard errors in parentheses.Columns 1-2 estimate a linear probability model; column 3 estimates a logistic regression with penalized log likelihood using the Firth (1993) method for bias reduction with rare events.Group fixed effects estimated but not reported.* p<0.05; * * p<0.01; * * * p<0.001.

Table 2 :
Boston Brazilians Focus Group Selection: Descriptive Statistics Note: P-values are from two-tailed difference in means t-tests.

Table 3 :
Predictors of Boston Brazilians Project Focus Group Selection (Linear Probability/Firth  Logistic)