2.1 The use of NLP for quantifying responses from open-ended survey questions

Several natural language processing (NLP) techniques have been used to analyze open-ended survey data. These range from unsupervised approaches such as topic modeling, which automatically uncover latent themes (Chen et al. Reference Chen, Vidden, Nelson and Vriens2018; Pietsch and Lessmann Reference Pietsch and Lessmann2018; Roberts et al. Reference Roberts2014), to supervised or semi-automatic coding strategies that rely on predefined coding schemes and human-annotated training data to estimate statistical annotation models (Haensch et al. Reference Haensch, Weiß, Steins, Chyrva and Bitz2022; Gweon and Schonlau Reference Gweon and Schonlau2024). More recently, researchers have also prompted generative large language models (LLMs) to classify responses into predefined categories without additional task-specific training (Mellon et al. Reference Mellon, Bailey, Scott, Breckwoldt, Miori and Schmedeman2024).

Among unsupervised models, one widely used approach is the structural topic model (STM) (Roberts et al. Reference Roberts, Stewart, Tingley and Airoldi2013), applied in comparative survey research to model topics in the open-ended response (Mourtgos and Adams Reference Mourtgos and Adams2019; Roberts et al. Reference Roberts2014), while also incorporating covariates reflecting other attributes expressed in closed-ended questions.

While topic models are valuable for exploratory analysis, their data-driven nature poses a challenge for multilingual text collections and may result in different topic distributions for responses in different languages, even when respondents discuss similar underlying concepts. Such variability complicates systematic cross-national comparisons, as topics are not inherently aligned across different linguistic and cultural groups (Roberts et al. Reference Roberts2014). STMs remain useful for exploratory research in single-country studies, but their lack of predefined topic structures and cross-linguistic alignment limits their effectiveness in comparative survey settings.

Mellon et al. (Reference Mellon, Bailey, Scott, Breckwoldt, Miori and Schmedeman2024) praise LLMs as a powerful tool for social science researchers that could allow for increased use of open-ended survey questions, given their relative ease of use as compared to supervised modeling requiring expensive annotations. At the same time, they caution against blindly trusting the LLM output without proper validation and warn that annotation quality may be worse in languages other than English. Lin and Zhang (Reference Lin and Zhang2025) also raise the complexity of the annotation task as a factor in deciding whether LLMs should be used directly as primary coders or merely as secondary coding assistants.

Supervised methods require the development of an appropriate coding scheme, training of human annotators and manual annotation of a representative sample, in order to train a model. In their analysis, Mellon et al. (Reference Mellon, Bailey, Scott, Breckwoldt, Miori and Schmedeman2024) find LLMs to outperform their supervised models despite training the latter on a substantial amount of annotated examples. However, incorporating pre-trained language models (LMs) in the supervised model may allow for better annotation in multilingual settings, when a PMLM is chosen.

PMLMs leverage extensive pre-training on diverse, multilingual corpora, enabling them to generalize more effectively across languages and to capture semantic nuances. This capability facilitates consistent categorization of key concepts across different cultural contexts, which often vary across countries.

Given these trade-offs, supervised approaches, combining open-ended responses with structured coding techniques, offer a promising solution.

2.2 Structured coding frameworks for open-ended responses

A well-defined coding scheme ensures consistency, validity and theoretical alignment in classifying open-ended responses, whether manually by coders or automatically by PMLMs, and is essential for maintaining comparability across countries (Nelson et al. Reference Nelson, D.Burk, Knudsen and McCall2021).

In comparative survey research, coding schemes must be conceptually robust yet flexible enough to capture linguistic and cultural variation in how respondents express their views (Aizpurua Reference Aizpurua, Sha and Gabriel2020). Human coders play a crucial role by providing expert-annotated validation datasets that train models to distinguish between response types. This hybrid approach—combining human expertise with computational efficiency—improves classification accuracy and comparability in large-scale, cross-national research. While PMLMs scale the classification of open-ended responses, their reliability ultimately depends on the quality and theoretical soundness of the coding scheme.

This article employs the Democracy Tree framework (Dahlberg and Mörkenstam Reference Dahlberg and Mörkenstam2024) to classify conceptions of democracy across diverse linguistic and cultural contexts. The framework categorizes democracy-related discourse into procedural, substantive and outcome-based understandings. Originally developed for analyzing media discourse across 93 countries using distributional semantics, the Democracy Tree offers a systematic, empirically grounded and theoretically informed approach to structuring democratic conceptions.

We next describe the data, preprocessing and implementation of the Democracy Tree framework for classifying democratic conceptions across languages and cultures. We then compare PMLM performance to traditional coding and discuss implications for comparative survey research.

3.1 Survey data collection

The data collection, conducted between July and October 2020 via the panel provider Mantap Global AB, involved a master questionnaire translated by researchers at GESIS Leibniz Institute for the Social Sciences.Footnote ¹ The survey was fielded in ten countries—Estonia, France, Germany, Hungary, Italy, Poland, Russia, Spain, the United Kingdom and the United States—targeting Internet users aged 18–65. Stratified samples of approximately 2,000 respondents per country were drawn according to a predefined quota plan, yielding around 2,000 written responses per country.Footnote ² The survey covered topics of democracy and migration and included both closed- and open-ended questions; this article focuses on two open-ended questions about democracy.

The first open-ended question (Q1) asked the respondents to elaborate on what they think about when they hear the word democracy.Footnote ³ The second open-ended question (Q2) followed directly after a close-ended question asking the respondents to rate how satisfied they are with the way democracy works in their country on an 11-point numeric scale, from 0 labeled “Extremely dissatisfied” to 10 labeled “Extremely satisfied.”Footnote ⁴ Footnote ⁵ The first question captures abstract associations with democracy, while the second focuses on its perceived performance in each country.

While the responses in most of the countries were collected in the official language only, responses from Estonia were collected in either Estonian or Russian. Altogether, this gives responses from ten countries in nine languages (Russian is common to Estonia and Russia, English to the UK and the US). Figure 1 shows the number of responses for each question in the different countries and languages.

Figure 1

Number of responses per question (Q1 and Q2) and country/language.

3.2 Manual annotation

The data were annotated by international students—all native speakers of the language they annotated, and with an educational background in social sciences. The task consisted of annotating spans of text, that is, sequences of consecutive words, within the responses corresponding to categories in the coding scheme. The coding scheme, labeled the Democracy Tree, was developed both inductively and deductively based on online media data from the Online Distributional Semantic Lexicon created by the Linguistic Explorations of Societies (LES) project at the University of Gothenburg (see Dahlberg and Mörkenstam Reference Dahlberg and Mörkenstam2024; Dahlberg et al. Reference Dahlberg2023). Despite methodological differences, studies commonly converge on procedural or substantive understandings of democracy (Lu and Chu Reference Lu and Chu2021; Shin and Kim Reference Shin and Kim2018; Wu and Wu Reference Wu and Wu2022; Zhai Reference Zhai2019). Procedural democracy emphasizes electoral processes and institutional frameworks, while substantive democracy encompasses values and outcomes. This distinction forms the cornerstone for delineating three abstract categories: democracy as a system of governance, democracy as values and democracy as outcome. These categories were instrumental in constructing a Democracy Tree, aligning with findings from a previous study (Dahlberg, Axelsson, and Holmberg Reference Dahlberg, Axelsson and Holmberg2020), thereby ensuring the reliability and validity of the coding. The integration of these categories with prior research findings indicates the robustness of the coding, which is also demonstrated by correlations with previous studies on conceptions of democracy from open-ended survey questions (Dalton, Shin, and Jou Reference Dalton, Shin and Jou2007a). Finally, the Democracy Tree expands upon these dimensions to provide comprehensive insights into democracy, thereby rendering our results comparable to other cross-cultural survey studies (Dahlberg and Mörkenstam Reference Dahlberg and Mörkenstam2024). See Appendix 2 of the Supplementary Material for an overview of the Democracy Tree.

The coding scheme is made up of three main domains:

1. 1. Governance.

2. 2. Values.

3. 3. Outcome.

Each domain comprises a maximum of three subdomains, allowing a multilevel coding process. For example, 1. Governance represents the main domain (level 1), 1.1 Governance – Politics a subdomain at level 2, 1.1.1 Governance – Politics – National a subdomain at level 3 and 1.1.1.1 Governance – Politics – National – National Actors a subdomain at level 4. Besides the three main domains, the coding scheme includes three categories for coding text inapplicable to any of these domains:

1. 4. Other.

2. 5. Missing/Non-Response.

3. 6. Linguistically Unclear.

The annotators were instructed to only use domain 4. Other for spans of text that were fit for annotation but did not apply to any of the three main domains. Domain 5. Missing/Non-Response was used when text was missing or considered as an evident non-response (e.g., “I don’t know”), and domain 6. Linguistically Unclear was used when the annotator could not make sense of the text (e.g., “Yu es hey war”).

The annotators were encouraged to select the most specific category possible; they were allowed to assign multiple labels to the same span but were in general discouraged from doing so. However, of all labeled spans in the five languages about 40% still overlap with another span at level 1.

We selected a subset of five languages to annotate: Hungarian, Italian, Polish, RussianFootnote ⁶ and Spanish. The selection of languages was motivated by an interest in linguistic diversity (two Romance, two Slavic and one Finno-Ugric language) and in having some variation in political regime types in the respective countries.

Annotation was conducted in three phases. First, two annotators for each language independently annotated the two open-ended responses of (the same randomized) 500 respondents. After computing inter-annotator agreement (IAA) on these 500 instances, the two annotators were asked to reach a consensus annotation on the instances where annotations diverged the most, which amounted to about 300 instances per language. Figure 2 shows IAA in the first stage for categories at level 1 of the Democracy Tree across the five languages according to Cohen’s $\kappa $ (Cohen Reference Cohen1960). The agreement has been calculated separately for the divergent instances (red bars), which were later consolidated with a consensus annotation, and the remaining instances (blue bars), which were used unchanged. For the latter, the $\kappa $ values range from 0.70 (Polish) to 0.99 (Russian), which is regarded as substantial to almost perfect agreement. Finally, after the consolidation phase, the two annotators separately annotated 250 instances each. This gives a total of 1,000 annotated responses (for both open-ended questions) in each language, where 500 are annotated by a single annotator, about 300 are annotated by a consensus of two annotators and about 200 are annotated by both annotators. For the experiments, we only use one of the annotations for the doubly annotated instances, selecting the annotator that shows the highest IAA with the consensus annotations.

Figure 2

Inter-annotator agreement at Democracy Tree level 1. Blue bars correspond to examples with high enough agreement to be included for further use, while red bars show examples selected for consolidation by both annotators in each language.

Although we cannot measure the agreement on the annotations from stages two and three (because we only have single annotations), we expect it to resemble the unchanged annotations in Figure 2 more than the divergent annotations. IAA in the same splits at lower levels in the Democracy Tree is lower (Figures 7 and 8 in the Supplementary Material), which is why we only use annotations at level 1.

3.3 Automatic annotation

To automatically annotate the remaining responses in the five countries with manual annotations (roughly 1,000 responses per country), as well as all responses in the other five countries (roughly 2,000 responses per country), we fine-tune XLM-RoBERTa (Conneau et al. Reference Conneau, Jurafsky, Chai, Schluter and Tetreault2020)Footnote ⁷ using a subset of our annotated data. XLM-RoBERTa (XLM-R for short) is a multilingual pre-trained encoder model trained on text from the CommonCrawl repository in 100 languages, including all nine languages in our study. The multilingual pre-training enables classification after fine-tuning not only on the five languages included in the training data (Hungarian, Italian, Polish, Russian and Spanish) but also on the remaining four languages (English, Estonian, French and German). This technique is known as zero-shot cross-lingual transfer and has been studied extensively for PMLMs (see, e.g., Pires, Schlinger, and Garrette Reference Pires, Schlinger, Garrette, Korhonen, Traum and Màrquez2019; Wu and Dredze Reference Wu, Dredze, Inui, Jiang, Ng and Wan2019). To be able to assign different categories to different segments of a response, we fine-tune XLM-R for token classification, following common practice for related annotation tasks such as named-entity recognition (Liang et al. Reference Liang, Yu, Jiang, Er, Wang, Zhao and Zhang2020). This means that the model is trained to assign a label to each token (a word or a part of a word) in the text.

The choice of model and annotation approach is based on a pilot study conducted on the subset of the data available before the third phase of annotation (Zhao Reference Zhao2023).Footnote ⁸ Note that this approach involves fine-tuning a relatively small encoder LM (XLM-R) instead of prompting a much larger generative decoder model, like ChatGPT, Gemini or Llama. The motivation for this choice is threefold: (i) fine-tuning may improve performance when adapting the model to the specific coding scheme; (ii) annotating open-ended survey responses requires token-level classification, for which fine-tuned encoder models still often outperform much larger generative decoder models (Arzideh et al. Reference Arzideh2025; Nielsen, Enevoldsen, and Schneider-Kamp Reference Nielsen, Enevoldsen and Schneider-Kamp2025); and (iii) smaller models are more resource-efficient.

In preparation for the classification, annotated spans are converted into token-level labels using the BIO format (Ramshaw and Marcus Reference Ramshaw and Marcus1995), where B- and I- indicate the beginning and inside of a span, respectively, and tokens outside any span are assigned the label O. Figure 3 provides an example of both span and BIO annotations.

Figure 3

Example of a response annotated as spans (top row) and in BIO token classification format (bottom row).

Considering the low IAA for lower levels of the Democracy Tree mentioned earlier, we focus on the three broad level 1 categories, Governance (GOV), Values (VAL) and Outcomes (OUT), and map the remaining top-level categoriesFootnote ⁹ to Other (O). We split the data for each language by respondents into 850 training pairs, 50 development pairs and 100 test pairs. We randomly sample the development and test sets from the final round of 500 annotations and use all remaining data for training in order to have as much data as possible for training while retaining a reasonably sized test set.

To annotate responses in the four languages for which we do not have any annotations, we use cross-lingual zero-shot learning to fine-tune an instance of XLM-R on all the data from the other five languages and apply it to the unseen languages. We call this the multilingual model. For the five languages with annotations, we can either use this model or a monolingual model fine-tuned only on annotated data of that language.Footnote ¹⁰ To select the best model for the seen languages and to estimate the expected accuracy on the unseen languages, we perform a series of experiments with mono- and multilingual models fine-tuned only on the training sets and evaluated on the test sets:

• • Monolingual models, trained on one language, evaluated on the same language.

• • Multilingual models, trained on all five languages, evaluated on the same languages.

• • Cross-lingual models, trained on subsets of four languages, evaluated on the fifth.

While the cross-lingual experiments are the most relevant to our analysis on the full survey, monolingual and multilingual experiments indicate how well models perform on the five languages with annotations and how performance on individual languages may be helped by learning from additional annotations in other languages.

We estimate hyperparameters like learning rate and number of training epochsFootnote ¹¹ for the three scenarios on a subset of languages and settle on a learning rate of 2 $\times $ 10 $^{-5}$ and training up to five epochs with early stopping. Appendix 5 of the Supplementary Material gives more details on hyperparameter experiments, and Appendix 6 of the Supplementary Material shows the amount of data used at different stages of the experiments.

Building on the insights gained from the multilingual and cross-lingual experiments, we finally train a new multilingual model on all 5,000 annotated responses, which we use to annotate all survey responses from all ten countries to estimate the proportions of different Democracy Tree categories in individual countries and in the survey as a whole (see Section 4.2).

5.1 Analytical strategy

To test these predictions using our annotated survey data, we include three dependent variables corresponding to the three democracy categories: democracy as governance, as values and as outcome. Each variable is binary, coded as 1 if the classifier detects the respective category in a respondent’s answer, and 0 otherwise; coefficients therefore capture associations with detection probabilities. Since responses may reference multiple aspects of democracy, the classifications are not mutually exclusive—a single response can be assigned to one, two or all three categories.

To capture nativist attitudes, we construct an additive index ( $\alpha $ = 0.79) based on three survey items: (1) “Immigrants are generally good for [country]’s economy,” (2) “[Country]’s culture is generally undermined by immigrants,” and (3) “Immigrants increase crime rates.” Each item uses a five-point Likert scale ranging from “strongly agree” to “strongly disagree.” Responses were recoded such that higher values indicate stronger anti-immigrant or nativist sentiment.