2.1 Calibrating a dynamic lexicon of ideology

Ideology is generally conceived of in multidimensional terms, as represented by a $d$ -dimensional vector in $\mathbb{R}^{d}$ . Extracting these dimensions is referred as ideological scaling. Intuitively, two members who share similar political views should be positioned nearby one another in ideological space, which is generally defined in terms of Euclidean distance.

Text-scaling approaches assume that the frequency of expression of specific terms made by a given actor can be used to estimate said actor’s position in an ideological space. Initially Monroe and Maeda (Reference Monroe and Maeda2004) proposed a scaling model based on item response theory that was later extended by Slapin and Proksch (Reference Slapin and Proksch2008) to measure political positions from written documents. Their model considers that the use of words can be modeled by a Poisson process. The process first builds a matrix $Z$ (also called a term-document matrix or TDM) where each element $z_{jw}$ corresponds to the number of times the word $w$ is present in party $j$ manifesto. Then, the expected value of $z_{jw}$ is expressed as a function of the ideologies:

(1) $$\begin{eqnarray}\displaystyle Pr(z_{jw}=k)=f(k;\unicode[STIX]{x1D706}_{jw})\quad \text{with }\unicode[STIX]{x1D706}_{jw}=\exp (s_{j}+p_{w}+\unicode[STIX]{x1D703}_{w}x_{j}), & & \displaystyle\end{eqnarray}$$

where $f(k;\unicode[STIX]{x1D706}_{jw})$ represents the Poisson probability mass function. The model introduces two factors $p_{w}$ and $s_{j}$ to deal with the more frequent words and more active political candidates. The $\unicode[STIX]{x1D703}_{w}$ factor takes into account the impact of words in the author’s position and $x_{j}$ is the estimate of the author’s position. Estimators of the unknown parameters are derived using a maximum likelihood estimation. An iterative approach (using expectation–maximization) is implemented in the Wordfish algorithm (Slapin and Proksch Reference Slapin and Proksch2008).

This dynamic lexicon creation process assumes that the textual content is politically dominant. This is often the case for political candidates’ tweets, especially during an election campaign. We first record all the sequences of $N$ adjacent words, $N$ -gram, in the political candidates’ tweets, for particular values of $N=1$ , $2$ or $3$ . We take into account co-occurrences with other words to more readily distinguish between semantic groupings (Brown et al. Reference Brown, Desouza, Mercer, Della Pietra and Lai1992; Jurafsky and Martin Reference Jurafsky and Martin2009) and to better interpret sarcasm (Lukin and Walker Reference Lukin and Walker2013). Stop words are discarded and stemming is performed to improve the extraction accuracy. We handle nonpolitical terms by introducing a threshold parameter $\unicode[STIX]{x1D6FD}$ . We discard $N$ -grams that are not shared by at least $\unicode[STIX]{x1D6FD}$ percent of the set of political candidates.Footnote ³ We then build a TDM ( $Z^{elit}$ ) by matching the $N$ -grams’ occurrences to the corresponding politicians. Finally, we use the Poisson model described in equation (1) to generate the dynamic lexicon of political terms, specifically we estimate $\hat{\unicode[STIX]{x1D703}}_{w}^{elit}$ and $\hat{p}_{w}^{elit}$ .

2.2 Scaling ideological positions for social media users

Text-scaling methods fail to retrieve a valid ideological dimension when the subject matter of the texts in question is broader than politics (Grimmer and Stewart Reference Grimmer and Stewart2013). It is reasonable to assume that politicians’ user-generated content will be concentrated around political topics and themes. This assumption does not hold, however, for the average social media user. Ergo, an analysis of the content generated by average users—the political centrality of which is variable—does not readily lend itself to the identification of valid ideological positions.

In order to detect the ideological signal in the textual data of average social media users, a dynamic lexicon approach analyzes user-generated content through the lens of the content generated by politicians on the same social media platform. This approach is inspired by the concept of transfer learning, that is, estimating certain parameters using one dataset and then using those parameter values to make inferences on another dataset (Do and Ng Reference Do, Ng, Weiss, Schölkopf and Platt2005). When generating ideological estimates for politicians, the Poisson model (1) associates a weight estimator ( $\hat{\boldsymbol{\unicode[STIX]{x1D703}}}^{elit}$ ) and a popularity estimator ( $\hat{\boldsymbol{p}}^{elit}$ ) to each term present in the derived dynamic lexicon. One can then estimate a given user’s ideology by taking into account the precomputed values ( $\hat{\boldsymbol{\unicode[STIX]{x1D703}}}^{elit}$ , $\hat{\boldsymbol{p}}^{elit}$ ) in the Poisson model fit to the matrix $Z^{elit}$ when handling the citizens’ social media content. This entails an adapted TDM $Z^{cit}\in \mathbb{R}^{n\times q}$ . The TDM is built by matching the $q$ terms existing in the lexicon of political terms to a given social media user. The matrix entries $z_{iw}$ count the number of instances in which each term (identified by $w=1\ldots q$ ) appears in the aggregated content of the user. The estimates are obtained by solving for the convex optimization problem (2):

The log likelihood function of a Poisson distribution is given by

$$\begin{eqnarray}l(\unicode[STIX]{x1D6EC})=\mathop{\sum }_{i=1}^{n}\mathop{\sum }_{w=1}^{q}(z_{iw}\ln (\unicode[STIX]{x1D706}_{iw})-\unicode[STIX]{x1D706}_{iw}-\ln (z_{iw}!))\end{eqnarray}$$

where $\unicode[STIX]{x1D6EC}$ represents a matrix, here the matrix of event rates $\unicode[STIX]{x1D706}_{iw}=\exp (s_{i}+\hat{p}_{w}^{elit}+x_{i}\hat{\unicode[STIX]{x1D703}}_{w}^{elit})$ . We optimize the log likelihood over the variables $s_{i}$ and $x_{i}$ conditional on $\hat{\unicode[STIX]{x1D703}}_{w}^{elit}$ and $\hat{p}_{w}^{elit}$ . That is, the parameters $\hat{\boldsymbol{\unicode[STIX]{x1D703}}}^{elit}$ and $\hat{\boldsymbol{p}}^{elit}$ are now precomputed in the political candidates’ dynamic lexicon estimation process. This leads to the maximization problem

$$\begin{eqnarray}\underset{s_{i},x_{i}}{\text{maximize}}\mathop{\sum }_{i=1}^{n}\mathop{\sum }_{w=1}^{q}(z_{iw}(s_{i}+x_{i}\hat{\unicode[STIX]{x1D703}}_{w}^{elit})+z_{iw}\hat{p}_{w}^{elit}-\exp (\hat{p}_{w}^{elit})\exp (s_{i}+x_{i}\hat{\unicode[STIX]{x1D703}}_{w}^{elit})-\ln (z_{iw}!))\end{eqnarray}$$

where both terms $z_{iw}\hat{p}_{w}^{elit}$ and $\ln (z_{iw}!)$ do not depend on the instances $x_{i}$ and $s_{i}$ . Suppressing these terms does not alter the optimal solution and leads to the convex optimization problem (2).

(2) $$\begin{eqnarray}\underset{s_{i},x_{i}}{\text{maximize}}\mathop{\sum }_{i=1}^{n}\mathop{\sum }_{w=1}^{q}(z_{iw}(s_{i}+x_{i}\hat{\unicode[STIX]{x1D703}}_{w}^{elit})-\unicode[STIX]{x1D6FC}_{w}\exp (s_{i}+x_{i}\hat{\unicode[STIX]{x1D703}}_{w}^{elit})),\end{eqnarray}$$

where $x_{i}$ denotes the ideology of citizen $i$ and $s_{i}$ depicts their publication activity. The constant $\unicode[STIX]{x1D6FC}_{w}=\exp (\hat{p}_{w}^{elit})$ takes into account the $N$ -gram popularity vectors estimated from political candidates’ lexicon. The optimization problem is derived by maximizing the likelihood function of Equation 2 over two sets of variables ( $x_{i}$ and $s_{i}$ ) instead of the four initial vectors.

3.1 Vote Compass

In order to validate the ideological estimates derived from the dynamic lexicon approach, we rely on the survey data generated by a large-scale VAA with active instances in each of the election campaigns we draw on as case studies in this article.

Vote Compass is an online application that surveys users’ views on a variety of public policy issues germane to a given election campaign, and then offers users an estimation of their position in the political landscape and, by extension, their alignment with each of the political parties contesting said campaign. The application is wildly popular, drawing millions of users worldwide, and is generally run in partnership with major media organizations in the jurisdiction where an election is being held.

In addition to attitudes to policy issues, Vote Compass also collects sociodemographic attributes and ideological self-placement on a scale of 0 to 10, representing left to right. Users also have the option to self-select into a research panel which associates their respective Twitter accounts with their responses to the survey. A total of 62,430 Twitter users were recruited through Vote Compass ( $n=11,344$ for Quebec 2014, $n=8,452$ for New Zealand 2014, and $n=42,634$ for Canada 2015).

3.2 Twitter

The user-generated content during the campaign period for accounts associated with verifiable candidates for election is collected using the Twitter REST API. These data form the corpus of terms that serve as the lexicon that constitute the ideological discourse of a given campaign. A content filter with threshold ( $\unicode[STIX]{x1D6FD}$ ) excludes candidates whose tweets do not include at least 5% of these political terms by word count. A second network filter eliminates candidates who have fewer than $25$ followers who we can validate externally using Vote Compass data (see Section 3.1). A third filter excludes candidates from political parties that have fewer than three politicians in the sample once the first two filters have been applied. Excluding candidates from such minor parties from the analysis prevents unreliable conclusions resulting from small sample sizes for classification.

Across the three election campaigns served as case studies for our analysis, we identify 297 candidates with public Twitter accounts in the 2014 Quebec election, 131 candidates in the 2014 New Zealand general election, and 759 candidates in the 2015 Canadian federal election. The number of active candidates in our sample once filtering has been applied is as follows: $m=106$ for Quebec 2014, $m=56$ for New Zealand 2014 and $m=120$ for Canada 2015. Citizens consist of potentially eligible voters that are active on Twitter during the electoral campaigns under scrutiny. As with candidates, we only consider citizens with at least $25$ political bigramsFootnote ⁵ in the dynamic lexicon within each context, and who follow a minimum of $3$ active candidates. The citizens’ sample sizes after applying these filtering conditions are $n=796$ (Quebec 2014), $n=123$ (New Zealand 2014) and $n=1,717$ (Canada 2015).

4.1 Mapping ideological landscapes

The three case studies selected to demonstrate the dynamic lexicon approach are the 2015 Canadian federal election, 2014 New Zealand general election, and the 2014 Quebec provincial election. The intuitive ideological placement of the parties contesting these elections offers the opportunity to compare the estimates produced by the dynamic lexicon approach for both an external validity test and robustness check of the method.

Canada is a federal state, and elections take place in the context of a first-past-the-post electoral system (Kam Reference Kam2009; Godbout and Høyland Reference Godbout and Høyland2011). Four federal parties contest the nationwide vote, with the sovereigntist Bloc Quebecois running candidates in the province of Quebec. The New Democratic party (NDP) and the Green Party of Canada (GPC) typically vie for the position of most left-leaning in federal politics, depending on the policy context, with the Liberal Party of Canada (LPC) adopting a generally center-left position and the Conservative Party of Canada (CPC) situated on the right of the ideological spectrum. Only the Liberal and Conservative parties have ever formed government, with the Conservatives having won, at minimum, a plurality of seats in the Canadian Parliament from 2006 to 2015, when the Liberals swept the federal election and formed a majority government.

New Zealand is a unitary state, and its elections have been run under a mixed-member proportional framework since 1996. Although more than a dozen parties contest general elections, seven of which had at least one seat in Parliament following the 2014 general election. Of these, the filtering process (see Section 3.2) results in the inclusion of candidates from four parties. Of these the Green Party of Aotearoa New Zealand (GRN) is generally considered the most left-leaning. While the Mana Party has adopted certain radical positions, its coalition with the Internet Party during the 2014 campaign was perceived to have a slightly moderating effect on its ideological positioning, placing Internet Mana (MNA) slightly to the right of the Greens. The New Zealand Labour Party (LAB) is generally considered a center-left party and the New Zealand National Party (NP) on the center-right.

The case of the 2014 Quebec provincial election offers two unique tests of the dynamic lexicon approach. First, it extends the approach to the subnational level by way of its application in the context of a provincial election campaign. Second, it tests the scalability of an unsupervised model to contexts where English is not the dominant language. Quebec has a majority francophone population and the sovereigntist sentiment in the province gives rise to a unique ideological landscape within Canada wherein nationalism and identity constitute an independent dimension orthogonal to the more socioeconomic left–right spectrum. Toward the nationalist end of this spectrum, three parties advocate for Quebec independence: Option Nationale (ON), the Parti Québécois (PQ), and Québec Solidaire (QS). The least nationalist party and that which is most opposed to Quebec independence is the Quebec Liberal Party (QLP), the party that won office in 2014, putting an end to a short-lived PQ minority government. The Coalition Avenir Québec (CAQ) positions itself in between these two poles by advocating more autonomy for the French-speaking province while remaining a member of the Canadian federation. But the socioeconomic left–right ideological dimension also structures Quebec party competition, following roughly the same order as the identity dimension with most nationalist parties leaning generally more on the left side of the spectrum relative to federalist parties. Notable exceptions include, for example, attitudes toward religious accommodation, where the nationalist PQ takes more right-leaning positions than other parties.

4.2 Validating ideological estimates for election candidates

In order to assert that the dynamic lexicon approach properly classifies social media users’ ideologies, we must first convincingly position election candidates within the ideological landscape. As candidates’ lexicon is used to generate the ideological scales upon which other social media users are positioned, the ideological estimates of candidates serve as the reference point for those of all other users.

To validate the ideological estimates ascribed to candidates by the dynamic lexicon approach, we adapt the roll-call scaling method—an established standard in the estimation of elected officials’ ideological positions (Poole and Rosenthal Reference Poole and Rosenthal2000; Clinton, Jackman, and Rivers Reference Clinton, Jackman and Rivers2004; Carroll et al. Reference Carroll, Lewis, Lo, Poole and Rosenthal2009). In doing so, however, we face two immediate constraints. Roll-call data are only available for elected officials, not political challengers. While this is less problematic in a two-party system given that both parties are likely to have substantial representation in government, it becomes more problematic in a multi-party, Westminster system (Spirling and McLean Reference Spirling and McLean2007; Grimmer and Stewart Reference Grimmer and Stewart2013). Not only does the presence of smaller parties increase the error associated with ideological estimates, but a political culture that emphasizes party discipline makes it different to distinguish party ideology from candidate ideology. While it is reasonable to assume that most candidates’ ideologies should be generally more aligned with their own party than with any other party, some degree of variation can be reasonably expected but would be difficult to detect using a DW-NOMINATE approach.

To address this limitation, one can estimate ideological positions based on network information, such as the existing connections between citizens and political candidates on social media (Barberá Reference Barberá2015; Bond and Messing Reference Bond and Messing2015; Rivero Reference Rivero2015).

The scaling process relies on the assumption that these virtual connections are homophilic in nature, that is that citizens tend to follow political candidates whose ideologies lie close to their own. We refer to this approach hereafter as the network scaling approach (Barberá Reference Barberá2015). We validate the results of the dynamic lexicon approach by comparing how it positions election candidates vis-á-vis the estimates produced by network scaling.

The results demonstrate that, compared to the ideological estimates generated by a network scaling approach, the dynamic lexicon approach provides highly convergent results (see Figure 1). It is apparent that the point estimates showing each political candidates’ positions measured using two fundamentally different scaling methods are strongly correlated. The correlations are particularly strong in the cases of the 2015 Canadian and the 2014 New Zealand elections. The linear relationship between the estimates indicates that they seem to capture a similar ideological dimension, which implicitly confirms the validity of both methods. Further examination suggests that these methods can identify clusters of candidates belonging the same political party, even though within-party correlations are weak. This supports the premise that candidates are generally more aligned in ideological terms with candidates from their own party than with those from other parties. The positioning of candidates from each party accords with conventional wisdom as to how said parties are generally situated with their respective political landscapes. It also aligns with estimates derived by aggregating public articulations of public policy positions. This suggests that the dynamic lexicon approach is able to scale ideological positions at the individual level for political candidates and, by extension, render a valid ideological landscape from social media data.

Figure 1. Political candidates—Comparison of estimated positions for the reference method (network scaling approach) and the dynamic lexicon approach. The x-axis shows the standardized position of political candidates on the unidimensional latent space derived from network data (Barberá Reference Barberá2015). The y-axis shows the standardized position of political candidates on the unidimensional latent space derived from the dynamic lexicon approach. Pearson correlations ( $\unicode[STIX]{x1D70C}$ ) are all statistically significant (p-value ${<}$ 0.05).

Figure 1 compares the estimates generated by the dynamic lexicon approach with those of the network scaling approach. In the context of the 2015 Canadian election, the clustering of candidates generated by both methods depicts a similar ideological landscape. The candidates cluster by party association and the party positioning exhibits face validity given the Canadian context.

The 2014 New Zealand context represented in Figure 1 shows once again that both the network scaling and dynamic lexicon approaches produce a consistent ideological landscape. The positioning of the party candidates also demonstrates face validity.

We find a slightly divergent result in the 2014 Quebec context. Unlike in Canada or New Zealand, the dynamic lexicon approach produces a different ideological landscape than the network scaling approach. For instance, the network scaling estimates, when taken alone, cannot differentiate the QLP from the CAQ. Although, upon examination of the dynamic lexicon approach estimates, we are able to clearly separate the CAQ from the QLP; although it then becomes difficult to distinguish the CAQ from ON. These results might suggest that these two methods, when taken together, can complement each other when capturing the ideological landscape. That said, the two methods identify relative clusters that are consistent with the parties’ positions as one would intuitively expect. The results may also indicate that more than one dimension is being needed to capture nuances in particular ideological landscapes.

4.3 Validating ideological estimates for social media users

Having demonstrated that the dynamic lexicon approach can derive valid ideological dimensions from the content that political elites generate on social media, and that it can position those political elites in ideological space in ways that correlated highly with more established methods, we now examine its ability to classify the individual ideologies of the broader population of social media users.

To do so, we compare the dynamic lexicon approach to a basic Wordfish approach, in which we ignore the precomputed values and instead simply apply the algorithm. Two different baseline strategies are proposed, depending on whether or not we consider the use of a lexicon of political terms to create the TDM for citizens. In the first strategy (Wordfish-all), users’ TDM is generated keeping all the $N$ -grams (i.e., following the same building process as for elites’ TDM $Z^{elit}$ ). The second strategy (Wordfish-political) considers the adapted TDM $Z^{cit}$ built from the dynamic lexicon of political terms. The reference ideological position for each users is derived from the survey data collected by Vote Compass.Footnote ⁶ We use an expectation–maximization algorithm to solve the maximum likelihood problem that generates these estimates (Barber Reference Barber2012). We consider a social media extraction to be effective if two social media users with similar ideologies according to the Vote Compass survey data are also situated in close ideological proximity to one another using the dynamic lexicon approach. The quality of estimates is then defined as the Pearson correlation coefficient ( $\unicode[STIX]{x1D70C}$ ) between the vector of textual estimates and the vector of reference ideologies.

Table 1 demonstrates the performance of the dynamic lexicon approach compared with the two baseline strategies. It clearly outperforms both baselines, especially for Quebec 2014, however, we observe weak to moderate correlations for this approach, which suggest that the dimension extracted by this method differs for the most part from that of the reference ideology. When we look solely at the politically interested citizensFootnote ⁷ individuals in our sample, the correlations remain similar even though one would expect the correlations to be higher for these more politically interested individuals as they are likely to demonstrate greater ideological consistency (Converse Reference Converse and Apter1964). This can be partly explained by the selection bias inherent to the user base of VAA, wherein the average VAA user is more politically interested vis-à-vis the overall population. We furthermore tested the network scaling approach that was used to validate candidate positions (Barberá Reference Barberá2015) on the broader population of social media users. The results of the network scaling approach are strongly correlated with the reference ideology ( $\unicode[STIX]{x1D70C}>0.6$ ). This result suggests that, by looking at one’s connections on social media (in this case, Twitter followers), we are able to derive meaningful information about one’s ideological position.

As the information captured by these methods seems to differ, we consider combining network and textual ideologies into a single estimate as a means of enhancing the measurement. A classical method of merging estimators is to consider the set of estimators $\hat{\mathbf{x}}_{\unicode[STIX]{x1D706}}$ generated by convex combinations (Struppeck Reference Struppeck2014). This family of ideological estimators is described by:

(3) $$\begin{eqnarray}\hat{\mathbf{x}}_{\unicode[STIX]{x1D706}}=\unicode[STIX]{x1D706}\ast \hat{\mathbf{x}}^{net}+(1-\unicode[STIX]{x1D706})\ast \hat{\mathbf{x}}^{txt},\quad \unicode[STIX]{x1D706}\in [0,1].\end{eqnarray}$$

We compare the efficiency of combining ideologies by analyzing the correlation of the new estimates $\hat{\mathbf{x}}_{\unicode[STIX]{x1D706}}$ with the reference ideologies according to multiple values of $\unicode[STIX]{x1D706}$ (Figure 2). The optimal quality for Canada 2015, New Zealand 2014 and Quebec 2014 is respectively reached for $\unicode[STIX]{x1D706}=0.86$ , $\unicode[STIX]{x1D706}=0.87$ and $\unicode[STIX]{x1D706}=1$ . None of these combinations leads to enhancements of the network performance by more than $1\%$ . This suggests that combining estimates does not lead to a significant improvement.

Figure 2. Citizens—Assessing linear combinations of textual and network ideologies. The x-axis displays the parameter $\unicode[STIX]{x1D706}$ (3). The textual ideology corresponds to the case $\unicode[STIX]{x1D706}=0$ . The network ideology corresponds to the case $\unicode[STIX]{x1D706}=1$ . The y-axis displays Pearson correlations (p-value ${<}0.05$ ) between the linear combination and the reference ideologies. The dashed line depicts the correlation for $\unicode[STIX]{x1D706}=1$ .

Table 1. Citizens—Assessment of the dynamic lexicon approach for citizens. Results are expressed for a bigram dictionary. The values indicate the Pearson correlations (p-value ${<}$ 0.05) between the ideologies extracted from Twitter data (text and network) and the reference ideologies based on policy positions. The values in parenthesis indicate the Pearson correlations for the subset of politically interested citizens.

The moderate correlations observed in the dynamic lexicon approach could be attributable to a variety of factors. Ordinary citizens  may exhibit less ideological consistency than do political elites, which could partly explain the lower correlations between the estimates (Converse Reference Converse and Apter1964). However, the results emphasized in Table 1 are not sufficient to independently validate said hypothesis. Another explanation could be that the periods of time studied are too short to capture a concept as substantive as ideology. Increasing the time frame could plausibly result in improved estimates. The presence of nonpolitical bigrams generates noisy estimates that explain the lower correlations observed for the textual estimates. The derived positions could also contain some other relevant information about individuals, but do not capture the same ideological dimension that captured by the reference ideology. Nevertheless, these two types of information could be complementary and could result in improved performance when trying to predict phenomena related to political ideologies, such as voting behavior.