3.1 Word Embedding

Word-embedding techniques represent an important recent advance in the large-scale analysis of text and, in particular, semantic meaning (Caliskan, Bryson, and Narayanan Reference Caliskan, Bryson and Narayanan2017; Charlesworth, Caliskan, and Banaji Reference Charlesworth, Caliskan and Banaji2022; Garg et al. Reference Garg, Schiebinger, Jurafsky and Zou2018). The basic requirement for training a word embedding layer is to convert a corpus of text into a term co-occurrence matrix. With this matrix we are then able to exploit pre-packaged algorithmic architectures to learn the pattern of co-occurrences and derive a distributional representation of each word in the corpus in vector space. To date, most practitioners use one of the GloVe (Pennington, Socher, and Manning Reference Pennington, Socher and Manning2014) or Word2Vec (Mikolov et al. Reference Mikolov, Chen, Corrado and Dean2013) modeling approaches. In the following, we use GloVe to train our embedding layer. We do so by sampling a maximum of 1.5m news articles across all countries combined and estimating a single embedding layer using the combined data from all countries. We also detail below experiments in the minimum effective training data sample size for this procedure.

While a promising agenda, studying semantic change over time in this way is confronted with two problems: 1) computational inefficiency; and 2) identification. Training an embedding layer is computationally expensive. As such, examining shifts in the relationship between words (proximity in vector space) over covariates of interest (such as time) requires a large amount of compute power, especially for large corpora. A corollary problem is that when embedding layers are trained over different temporal units, this means that over-time comparisons are no longer robust due to lack of identification in the underlying vector space (Hamilton, Leskovec, and Jurafsky Reference Hamilton, Leskovec and Jurafsky2016; Rodriguez et al. Reference Rodriguez, Spirling and Stewart2023).

3.2 ALC Word Embeddings

A recent innovation by Khodak et al. (Reference Khodak, Saunshi, Liang, Ma, Stewart and Arora2018), and implemented and extended by Rodriguez et al. (Reference Rodriguez, Spirling and Stewart2023), helps solve these problems. The technique—“ALC on Text”—provides a computationally efficient way to identify semantic change over time. The advantage of this technique is that we are able to use a single pre-trained embedding layer, and accompanying transformation matrix, to induce embeddings for a given target word over time without having to retrain an embedding layer for each unit of time.

The efficiency gains of the ALC approach come from the realization that embeddings for a particular (even very rare) target word may be derived by averaging the vectors of embeddings for words within its (here: six-word) context window from a pre-trained embedding layer (Arora et al. Reference Arora, Li, Liang, Ma and Risteski2018; Khodak et al. Reference Khodak, Saunshi, Liang, Ma, Stewart and Arora2018; Rodriguez et al. Reference Rodriguez, Spirling and Stewart2023). Once we have the embeddings of context words, we can then take the average of these vectors to derive our distributional representation of the target word. A transformation matrix—required to downweight words (such as stop words) that appear with high frequency—is then computed using the term co-occurrence matrix used to generate the embedding layer as well as the embedding layer itself (Khodak et al. Reference Khodak, Saunshi, Liang, Ma, Stewart and Arora2018; Rodriguez et al. Reference Rodriguez, Spirling and Stewart2023). Unlike other, e.g., dictionary-based methods, then, ALC embeddings do not rely on specific words appearing within the text corpus to derive a measurement nor does it rely on our target word appearing with high frequency in the embedding layer.Footnote ³

In our application, we train an embedding layer across a combined sample of all newspaper sources in all countries that make up our sample. We refer to this as our “reference embedding.” We pre-process the text by removing numbers, stopwords, and punctuation. We then use the GloVe algorithm, the R packages quanteda (Benoit et al. Reference Benoit2018) and text2vec (Selivanov, Bickel, and Wang Reference Selivanov, Bickel and Wang2025). We set vector dimensionality to length 300, and use a window size of six. The maximum number of iterations for training the embedding layer was set to 100, and the models all converged under this threshold for each country. We pruned the vocabulary over which to train the embedding layer such that the overall dimensionality of the resulting co-occurrence matrices was $\sim $ 30000x30000 (i.e., 30000 unique words). We then compute the transformation matrix required for the ALC approach using the R package conText developed by (Rodriguez et al. Reference Rodriguez, Spirling and Stewart2023). This is used to reweight words appearing with high frequency in the corpus. We also conduct experiments to determine the size of the feature space required to reliably detect signal.

3.3 Criticism Index

Unlike word frequency or topic modelling approaches, which use a bag of words as their foundation, word-embedding techniques retain the context and order of the text. One advantage of this is that the embedding layers retain information on the semantic associations between words, which means we can use matrix arithmetic to perform analogy tasks or derive index (vector) representations of concepts of interest (Bolukbasi et al. Reference Bolukbasi, Chang, Zou, Saligrama and Kalai2016). Our target leader words are detailed in Table B.2 in the Supplementary Material. We then calculate a criticism dimension by subtracting the vector for the word “opposition” ( ) from the vector for the word “support” ( ) in our reference embedding. The rationale for subtracting one word from the other is that this means the both poles of the index are interpretable. Ultimately, it also reduces the number of operations as it means projecting target words onto one index rather than two. This gives us a single “criticism index,” which will be used to capture coverage of events or editorial opinions that are critical of the political executive. We infer that text with high cosine similarity to the opposition pole is likely to be critical.

By criticism, we mean both news of events that are critical of the leader and editorial articles that are directly critical of the executive and its policies.Footnote ⁴ Here, our index captures one or all of three different things, which we understand to denote criticism of the executive:

1. 1. Reporting on events that target the figure of the leader, e.g., protests against the leader or their policies.

2. 2. Opinion articles and editorials detailing failings and allocating blame to the leader or his/her government.

3. 3. Second-hand criticism of the figure of the leader, e.g., reporting on public opinion and soundbites of citizens or other figures critical of the leader.

3.4 Projecting Words Over Time

We can observe temporal trends by calculating the cosine similarities between our target words of interest and our criticism index. To recover the over-time cosine similarities, we first split our observation period into year-week slices, and then get the context words around our target leader words for each country week. Using the ALC approach, we then estimate a time-period-specific embedding for the leader from the words appearing around their name over this time period. We do so by taking the average of the vectors of surrounding context words from our pre-trained references embedding layer for each of the leaders in each country in our sample respectively. We then combine these context words and apply the transformation matrix to downweight commonly appearing words. The weighting specified for the transformation matrix determines the extent to which commonly appearing words are penalized. A larger weighting means fewer words are downweighted.Footnote ⁵ Here, we set our transformation matrix weighting at 100, which is similar to other published work (Rodriguez et al. Reference Rodriguez, Spirling and Stewart2023). Below, we also detail experiments to determine the influence of the transformation matrix weighting on results.

From this procedure we are able to induce a single period-specific embedding for each leader over each time period. Once we have recovered these embeddings, we can then project them onto our criticism index by calculating the (l2-normalized) cosine distance between the vectors for each of our leaders and our criticism index over time. To aid applied researchers implement our approach, we have summarized this process in Figure 1.

Figure 1 Data analysis pipeline for measurement of media criticism in Arabic-language news media. This figure, illustrates the key steps in our methodology. (1) We collect news articles from publicly available sources across multiple countries and preprocess the text by removing stopwords, punctuation, and irrelevant content. (2) We identify mentions of political leaders and extract context words appearing within a six-word window around each mention. (3) Using pre-trained GloVe embeddings, we generate a reference embedding layer for all articles, which forms the basis for estimating media criticism. (4) We apply the ALC embedding method to construct time-specific word embeddings for each leader, allowing us to track changes in media discourse over time. (5) We compute a criticism index by projecting leader embeddings onto a semantic dimension spanning words associated with support and opposition. This pipeline enables us to estimate media criticism dynamically and at a granular temporal scale.

4.1 Changepoint Analysis

To provide a diagnostic routine for detecting signal of abrupt change in the levels of observed media criticism, we use a conventional cumulative sum (CUSUM) changepoint approach to detect structural changes in the time-series data (Zeileis et al. Reference Zeileis, Leisch, Hornik and Kleiber2002). The CUSUM approach works by estimating model residuals as a function of the time parameter of interest. It does so by estimating an OLS model of the outcome of interest, calculating the cumulative sum of standardized residuals over time, and comparing these to a null hypothesis of no change. Here, our specification is: ${cos\_sim}_t$ = $\beta _0$ + $\beta _1$ , ${week}_t$ + $\epsilon _t$ where ${cos\_sim}_t$ is the dependent variable (cosine similarity) at time t; β ₀ is the intercept; β ₁ is the slope coefficient for the predictor week _t ; and ϵ _t is the error term at time t. The F-statistic in this approach, provides us with an over-time estimate of model fit under two competing hypotheses: one of no structural change and another of structural change. A high F-statistic at a given point in time provides evidence of improved model fit when accounting for some structural change in over-time variation.Footnote ⁶

4.2 Synthetic Difference-in-Differences

We envisage that researchers will want to use our estimation approach to ask counterfactual questions. In particular, they may look to make causal estimates of the effect size of political events on media criticality of the executive. For our cases, the most obvious is the 2013 coup in Egypt. Here, the counterfactual question is: what would media criticism in Egypt have looked like had a coup not happened? The natural comparison case is Tunisia. Both Egypt and Tunisia experienced democratic breakthroughs in early 2011. Both are Arabic-speaking, Muslim-majority republics where entrenched dictators were overthrown through street-level mobilization within a few months of the other (Brownlee, Masoud, and Reynolds Reference Brownlee, Masoud and Reynolds2015; Ketchley and Barrie Reference Ketchley and Barrie2020). In both cases, precarious and highly polarized democratic transitions unfolded, with secular political forces competing for electoral power against organized Islamist movements (Nugent Reference Nugent2020). Crucially, both cases also saw a proliferation of new independent media organizations and the lifting of long-standing restrictions on media reporting during the post-breakthrough democratic transitions (El- Issawi Reference El- Issawi2016). However, unlike in Tunisia, Egypt’s democratic transition was abruptly ended in mid-2013, when a military coup overthrew the country’s first democratically-elected president, sparking thousands of anti-government street protests and a cycle of contention that targeted Egypt’s post-coup leadership (Ketchley Reference Ketchley2017, chapter 6).

For our main counterfactual analysis, we exploit the availability of news media from Tunisia to implement the synthetic difference-in-differences estimation procedure as described in Arkhangelsky et al. (Reference Arkhangelsky, Athey, Hirshberg, Imbens and Wager2021). Our analysis uses a panel of ten Egyptian and ten Tunisian newspapers (i) observed at weekly periods (t) beginning at the start of Egypt’s democratic transition in February 2011.Footnote ⁷ Following the case literature, we assume that Egyptian and Tunisian newspapers observed prior to the coup are operating in transitional democratizing contexts where they are more able to report on news and opinion that criticizes the executive, while newspapers in Egypt after the coup were more constrained in their reporting. Our econometric specification is thus: ${cos\_sim}_{it}$ = $L_{it}$ + $\tau _{it}W_{it}$ + $\epsilon _{it}$ , where $\tau _{it}$ is the effect of the coup on the cosine similarity score of newspaper (i) at week (t), and we estimate the average of $\tau _{it}$ over the observations where $W_{ij}$ =1. The matrix $L_{it}$ are simple two-way fixed effects at the unit and week level. Unit weights ( $\hat {\omega }$ ) match the pre-trend of the treated newspapers with the untreated controls (here: Tunisian newspapers), while time weights ( $\hat {\lambda }$ ) minimize the differences between the pre and post-treated periods for the controls.Footnote ⁸ In Section H of the Supplementary Material, we also estimate an interrupted time series model. This strategy is useful when applied researchers want to estimate the effect of an event on media criticality, but lack media articles from a comparison case.Footnote ⁹ We can also imagine that researchers might use newspaper media criticality scores from different contexts to estimate a comparative interrupted time series.

4.3 Synthetic Data Simulation

Applied researchers will also want to implement our proposed technique in other languages. To assess this, we innovate by generating synthetic data using two OpenAI large language models (LM) (specifically, gpt-3.5-turbo and gpt-4o). The prompts and code we used to generate these data are in code block 1 in the Supplementary Material. We used a limited prompting design, asking the model to generate a series of 500 articles that were “critical” and 500 “not critical” articles of a political figure we refer to as POLITFIG.Footnote ¹⁰ We use this neutral denotation to mitigate against activating any biases baked into the training data. We iterate over seven additional languages as well as Arabic. These data are useful for two key reasons: 1) they provide evidence of the generalizability of our technique to other languages; 2) they are designed specifically to include articles that are variously “critical” or “not critical” meaning we are able to determine whether our criticism index is actually capturing this concept. We translate words for support and opposition into each of these languages (see Figure C.1 in the Supplemenatry Material for the translations used). Instead of re-estimating embedding layers for these languages, we use the pre-trained embeddings for each language provided by Wirsching et al. (Reference Wirsching, Rodriguez, Spirling and Stewart2025). We select those languages that Wirsching et al. (Reference Wirsching, Rodriguez, Spirling and Stewart2025) have validated with human coders: Arabic, Chinese, English, French, Japanese, Korean, Russian, and Spanish.Footnote ¹¹

5.1 Detecting Changepoints

The above estimates demonstrate a high correlation between our text-based measures of media criticism in our change cases and those deriving from expert surveys measured at the country year level. If we are to incorporate text-based indicators into standard metrics of media freedom, we need a method to detect changes and a measure of the uncertainty associated with these changes. These rule-of-thumb metrics are central to other contributions using large text data as the foundation for early-warning systems (Balashankar, Subramanian, and Fraiberger Reference Balashankar, Subramanian and Fraiberger2023; Stolerman et al., Reference Stolerman2023). To achieve this, we implement our changepoint procedure that estimates an F-statistic of model fit under assumptions of structural change in the level of media criticism. The point where the F-statistic peaks can be understood as the most probable changepoint.

Figure 3 displays the weeks with the highest probability across our two change cases. The most pressing concern for applied researchers will likely be the size of training data required to detect signal. As such, we estimate our changepoint models over all six versions of our training data, i.e., from 10k to 1.5m unique news articles. We keep the feature size constant at 30k.Footnote ¹³ For both Egypt and Tunisia, we see that all versions, with the exception of the smallest in Tunisia, recover estimates of structural change points that align with case knowledge. In Egypt, this is at the beginning of July 2013 following a military coup; in Tunisia this is in late-December of 2010 and January of 2011 when Ben Ali’s dictatorial regime was ousted from power.

Figure 3 Top panel: F-statistics over time for changepoint procedure across versions; Bottom panel: text-based criticism scores for Egypt and Tunisia and breakpoints for each version. Breakpoints displayed with slight offset for visibility.

5.2 Counterfactual Estimation

Figure 4 shows the results of our synthetic difference in difference analysis to estimate the effect of the military coup in July 2013 in Egypt on criticism of the executive. As noted, to generate a plausible pre-coup trend, we use media criticism scores from newspapers from nearby Tunisia—which was also undergoing a democratic transition during this period—to construct unit and time weights in a two-way fixed effects model. The outcome measure is a newspaper’s criticism score assigned to the year-week from the period of democratic breakthrough in early 2011 through to 2019. The results suggest that the treatment in Egypt led to a substantive, enduring, and statistically significant diminution in media criticism, roughly equivalent to a one standard deviation decrease relative to pre-coup media criticism scores ( $p < .001$ ). This marked reduction in media criticism of the post-coup executive comes despite Egypt experiencing rampant inflation, currency devaluation and a foreign exchange crisis, thousands of anti-coup protests that continued for years after the military’s seizure of power, other episodes of street-level mobilization including against food prices and unpopular foreign policy decisions, and protracted insurgencies in the country’s border provinces (Grimm Reference Grimm2019; Ketchley and El-Rayyes Reference Ketchley and El-Rayyes2017; Ketchley Reference Ketchley2017; Nugent and Siegel Reference Nugent and Siegel2024).

Figure 4 Treatment effect of the coup (black arrow) on media criticism in Egypt using Tunisian newspapers as a counterfactual. Dashed line marks the coup.

Figure 5 Normalized cosine similarity criticism scores over time across eight languages using synthetic articles generated by gpt-3.5-turbo. Coloured lines represent the linear best fit over each period. The line at 0 is the point at which the LLM shifts to producing “not critical” articles.

Figure 6 Normalized cosine similarity criticism scores over time across eight languages using synthetic articles generated by gpt-4o. Coloured lines represent the linear best fit over each period. The line at 0 is the point at which the LLM shifts to producing “not critical” articles.

5.3 Synthetic Data

Figure 5 displays the results of re-estimating our main analysis across seven additional languages as well as Arabic using synthetic data. The text data is split into ten time units corresponding to 100 articles each. The call to the OpenAI API specified that language should become less critical after 500 of the 1000 total runs. The imagined time point at which the API shifts to producing “not critical” articles is displayed as 0 in Figures 5 and 6. Reassuringly, we observe that across all languages, there is a marked and statistically significant change at the midway point that is detected by our ALC word embedding approach for both LM. These results demonstrate that our approach is not sensitive to the input language and may be applied to other cases of authoritarianism and democratization.

5.4 Robustness

In the Supplementary Material, we provide a number of additional tests to ensure the robustness of our approach. These include comparing scores generated by our criticism index to human-labelled values for a sample of news articles, design-based supervised learning, estimating alternative criticism indices, alternative target words for the political executive, alternative normalization procedures when estimating cosine similarity, and additional checks that our scores are not an artefact of our use of “opposition” to connote criticism.