2.1 Binary choice model: the 2 × 2 case

Mathematically, considering an election in which each voter must choose between two parties (1 and 0) and denoting by l _i the d-dimensional vector of latent long-term policy positions (and/or partisanship) of voter i, the binary latent structure model choice states that l _i impacts probabilistically on the voter's choice in both elections, v _i,1 and v _i,2, through the equations:

(1)$$\matrix{ {P( {v_{i, 1} = 1} ) = f( {\alpha_1 + \beta_1l_i} ) } \cr {P( {v_{i, 2} = 1} ) = f( {\alpha_2 + \beta_2l_i} ) } \cr } \;$$

where the coefficient α _t and the d-vector of coefficients β _t (t = 1, 2) capture, respectively, the popularities and party positions of the reference party in both electionsFootnote ⁵ and f is a proper function that can either be the cumulative normal function or the logistic function.Footnote ⁶

Equation (1) models the causal process in which, in our model, the stable opinions of voters are confronted with the often-changing policies of parties and candidates to produce the voting behavior. As argued and tested on cross-national data in Thomsen (Reference Thomsen2011), the interplay between individual voters and parties is better modelled by the product between the position of the voter and the position of the party (known as “the directional model” in the literature on issue voting; Rabinowitz and Macdonald, Reference Rabinowitz and Macdonald1989) than by the distance between the two (known as “the proximity model”; Downs, Reference Downs1957). With the directional model, the valence parameters (α _t) are much better predicted by the mean sympathy score for the party than in the proximity model.

When only aggregate information is available, all components in equation (1) are unobserved, so to relate it with the known outcomes, individual probabilities must be aggregated to (averaged at) the polling unit (or constituency) level. In doing so, we consider that the number of voters T _u in each unit is large enough as to make “sampling” errors negligible. This allows to state that the relative marginal outcomes $p_{u, 1} = \mathop \sum \limits_{i\in u} v_{iu, 1}/T_u$ and $p_{u, 2} = \mathop \sum \limits_{i\in u} v_{iu, 2}/T_u$ are (almost) equal to the expected vote fractions in the unit and get:

$$E( {v_{iu, t}} ) = p_{u, {\rm t}} = \mathop \smallint \limits_{{\rm \Re }^d}^{} {\rm \Phi }( {\alpha_t + \beta_tl_{iu}} ) \phi ( l_{iu}\vert L_u, \;{\rm \Omega }) \partial l_{iu}$$

after assuming, as in Thomsen (Reference Thomsen1987), that the underlying dimension l _iu is normally distributed with mean L _u and variance–covariance matrix Ω.

Carrying out the integral (Thomsen, Reference Thomsen1987: 56), we obtain:

(2)$$E( {v_{iu, t}} ) = p_{u, {\rm t}} = {\rm \;\Phi }\left({\displaystyle{{\alpha_t + \beta_tL_u} \over {\sqrt {1 + \beta_t{\rm \Omega }\beta_t^T } }}} \right)$$

which is formally the same as (1), except for a rescaling value. Indeed, as α _t, β _t, and Ω are assumed to be constant across units, equations (1) and (2) state that the model for aggregate behavior, apart from rescaling, is equal to the model for individual behavior. What is more, (1) implies that the utilities to vote for a given party in the two elections (their inverse-probit transformed probabilities) are linearly related to each other and, by application of the axiom of local independence at the individual level, that the joint distributions of Φ⁻¹(p _u,1) and Φ⁻¹(p _u,2) (and of Φ⁻¹(p _u,1) and Φ⁻¹(1 − p _u,2), Φ⁻¹(1 − p _u,1) and Φ⁻¹(p _u,2), and Φ⁻¹(1 − p _u,1) and Φ⁻¹(1 − p _u,2)) are binormal. In general:

(3)$$p_{\,jk} = {\rm \;}{\rm \Phi }_2( {{\rm \Phi }^{{-}1}( p_{\,j, 1}) , \;\;{\rm \Phi }^{{-}1}( p_{k, 2}) , \;\rho_{\,jk}} ) $$

which allows estimation of the joint probabilities when ρ _jk, the so-called tetrachoric correlation coefficient, is known.

As Thomsen (Reference Thomsen1987) shows, when it is assumed that the latent variable variation between individuals has the same structure as the latent variable variation between local units (a reasonable isomorphism assumption when units are not too large within relatively politically homogenous geographical regions) and that the former variation is significantly greater than the latter, ρ _jk can be properly approximated by the corresponding ecological probit (or logit) correlation, ρ _e, and be estimated from the observed marginal counts. An alternative identification condition is presented in Park (Reference Park2008: 34–38).

At this point, joint probabilities can be directly estimated using equation (3) or, as Thomsen (Reference Thomsen1987) suggests, be approximated using equation (4). Equation (4) is derived using Yule's Q approximation to estimate the tetrachoric correlation (see Thomsen, Reference Thomsen1987: 64) and has slightly lesser computational costs.

(4)$$p_{\,jk}\approx \displaystyle{{1 + 2{\hat{\rho }}_ep_{\,j, 1} + 2{\hat{\rho }}_ep_{k, 2}-{\hat{\rho }}_e-\sqrt {{( {1 + 2{\hat{\rho }}_ep_{\,j, 1} + 2{\hat{\rho }}_ep_{k, 2}-{\hat{\rho }}_e} ) }^2-8{\hat{\rho }}_e( {1 + {\hat{\rho }}_e} ) p_{\,j, 1}p_{k, 2}} } \over {4{\hat{\rho }}_e}}$$

2.2. The general R × C case

The greatest limitation to the use of the latent structure theory for bivariate choice on actual elections resides in the non-duality of voters’ choices in actual elections. Even in two-party systems (or second round-off presidential elections) the third alternative of “non-voting” is a possible choice. Hence, the above 2 × 2 approach needs to be extended to the R × C case to be useful. Both Thomsen (Reference Thomsen1987) and Park (Reference Park2008) each make a proposal, with both proposals departing from crude binary choice estimates.

As a first step, they estimate raw joint probabilities p _jk by applying either equation (3) or (4) to the artificial set of binary choices defined by choosing, in election 1, between party j and the other parties and, in election 2, between party k and all other parties. Unfortunately, these crude binary choice-estimated probabilities are not congruent with the observed results. The sum across j (k) of the estimated joint probabilities $\hat{p}_{jk}$ does match the observed marginal fractions p _j+ (p _+k). Hence, as a second step, Park (Reference Park2008) and Thomsen (Reference Thomsen1987) propose a way to fix this. Park (Reference Park2008) suggests using the iterative proportional fitting algorithm (Deming and Stephan, Reference Deming and Stephan1940), whereas Thomsen (Reference Thomsen1987) proposes the use of a more complex algorithm that requires a reference or pivotal party to be chosen in each election.

Denoting by r ₁ and r ₂ the reference parties in, respectively, election 1 and election 2, the original iterative algorithm of Thomsen works as follows.

1. (i) First, crude binary choice probabilities $\hat{p}_{jk}^{( 0 ) }$ are computed.

2. (ii) Second, the temporary estimates of (i) are used to estimate the margins of a set of theoretical 2 × 2 tables composed by the set of parties {{j, r ₁}, {k, r ₂}}, with j ≠ r ₁ and k ≠ r ₂: $\hat{p}_{jk}^{( 0 ) } + \hat{p}_{jr_2}^{( 0 ) }$, $\hat{p}_{r_1k}^{( 0 ) } + \hat{p}_{r_1r_2}^{( 0 ) }$, $\hat{p}_{jk}^{( 0 ) } + \hat{p}_{r_1k}^{( 0 ) }$, and $\hat{p}_{jr_2}^{( 0 ) } + \hat{p}_{r_1r_2}^{( 0 ) }$. And, from them, the joint probabilities $\hat{p}_{jk}^{( 0 ) }$ are updated employing equation (5), which derives from Yule's Q approximation for the tetrachoric correlation.

   (5)$$\hat{\,p}_{\,jk}^{( 1 ) } = \displaystyle{{\hat{\,p}_{\,jr_2}^{( 0 ) } \;\hat{\,p}_{r_1k}^{( 0 ) } } \over {\hat{\,p}_{r_1r_2}^{( 0 ) } }}\displaystyle{{1 + \hat{r}_{\,jk\vert r_1, r_2}^{( 0 ) } } \over {1 + \hat{r}_{\,jk\vert r_1, r_2}^{( 0 ) } }}$$

where $\hat{r}_{jk\vert r_1, r_2}^{( 0 ) }$ is the across units ecological correlation between $ln( {( \hat{p}_{jk}^{( 0 ) } + \hat{p}_{jr_2}^{( 0 ) } ) /( \hat{p}_{r_1k}^{( 0 ) } + \hat{p}_{r_1r_2}^{( 0 ) } ) } )$ and $ln( {( \hat{p}_{jk}^{( 0 ) } + \hat{p}_{r_1k}^{( 0 ) } ) /( \hat{p}_{jr_2}^{( 0 ) } + \hat{p}_{r_1r_2}^{( 0 ) } ) } )$.

1. (iii) After applying (ii), we have new (updated) estimates for each pair of p _jk with j ≠ r ₁ or k ≠ r ₂, but not when j = r ₁ or k = r ₂. These probabilities are re-estimated (updated) by re-scaling them using as rates the relative discrepancies between the aggregations of the observed and temporary estimates:

   $$\hat{\,p}_{r_1k}^{( 1 ) } = \hat{\,p}_{r_1k}^{( 0 ) } \displaystyle{{\,p_{ + k}} \over {\tilde{\,p}_{ + k}^{( 1 ) } }}$$
   $$\hat{\,p}_{\,jr_2}^{( 1 ) } = \hat{\,p}_{\,jr_2}^{( 0 ) } \displaystyle{{\,p_{\,j + }} \over {\tilde{\,p}_{\,j + }^{( 1 ) } }}$$

where p _k+ and p _+j are the observed marginal fractions and $\tilde{p}_{j + }^{( 1 ) } = \mathop \sum \limits_{k\ne r_2} \hat{p}_{jk}^{( 1 ) } + \hat{p}_{jr_2}^{( 0 ) }$ and $\tilde{p}_{ + k}^{( 1 ) } = \mathop \sum \limits_{j\ne r_1} \hat{p}_{jk}^{( 1 ) } + \hat{p}_{r_1k}^{( 0 ) }$ temporary marginal fraction estimates.

1. (iv) Finally, we come back to (i), replace $\hat{p}_{jk}^{( 0 ) }$ by the new estimates and iterate until the process converges.

3.1 Probit transformations and exact estimates of probabilities

The latent factor ecological inference approach as originally suggested in the seminal work of Thomsen (Reference Thomsen1987) relies on Yule's Q approximation to update joint probabilities and only considers the Pearson correlation across units of the logit transformation of the binary choices. In other words, it uses the ecological logit correlation as an estimatorFootnote ⁷ of the individual tetrachoric correlations for all tetrachoric (fourfold) subsets of the voter's choice (Thomsen et al., Reference Thomsen, Berglund and Wörlund1991). ecolRxC extends this by also including the options of using the exact equation (3) instead of the approximation equation (4) and working with probit transformations. As we show in section 5 this leads to more accurate estimates, on average.

3.2 Measuring uncertainties

An estimate is not complete without a measurement of its estimation error; that is, its level of associated uncertainty. For the 2 × 2 case, as referenced in Park (Reference Park2008), Achen (Reference Achen2000) proposes estimating the standard errors of the binary Thomsen estimator using Fisher's z-transformations. Specifically, after computing 1 − α confidence intervals for the ecological correlation

(6)$$[ {\hat{\rho }_e^- , \;\;\hat{\rho }_e^ + } ] = \left[{tanh\left({\displaystyle{1 \over 2}ln\displaystyle{{1 + {\hat{\rho }}_e} \over {1-{\hat{\rho }}_e}}} \right)\mp \displaystyle{{z_{\alpha /2}} \over {\sqrt {U-2.5} }}} \right]$$

lower and upper limits of 1 − α confidence intervals for $\hat{p}_{jk}$ can be constructed, applying the plug-in principle, replacing in either (3) or (4) the correlation by $\hat{\rho }_e^-$ and $\hat{\rho }_e^ +$, respectively.Footnote ⁸

The extension up to the R × C case is made in ecolRxC via bootstrap (Efron and Tibshirani, Reference Efron and Tibshirani1994) by sampling in the estimated confidence intervals of the crude binary probabilities attained using the 2 × 2 approach. Specifically, ecolRxC computes 1 − α confidence intervals for the estimated probabilities by (i) randomly extracting B resamples from each estimated 1 − α crude binary probability confidence interval ($\tilde{p}_{jk}^{( 0 ) , b}$ b = 1, 2, …, B), (ii) making each set of resamples $\{ {\tilde{p}_{jk}^{( 0 ) , b} } \} \;$ congruent/compatible with the known outcomes, using either the iterative proportional fitting algorithm or the Thomson algorithm detailed in subsection 2.2, and (iii) calculating for each set of final congruent estimates their α/2 and 1 − α/2 percentiles. An alternative for estimating uncertainties would be to directly bootstrap polling units. We consider our proposal more in line with the approach.

3.3 Estimation of unit transfer tables

ecolRxC estimates both local (polling unit) and global (constituency) vote transfer matrices. In section 2, and in order not to overwhelm the exposition and notation, we choose to remain ambiguous as to whether the p _jk probabilities refer to a polling unit or to the whole district. As a rule, ecolRxC applies the methods presented in section 2 working at the polling unit level, obtaining the global matrices as aggregation (composition) of local matrices. As in the case of Park's solution, nevertheless, ecolRxC also offers the possibility of directly estimating global matrices by just applying either equation (3) or (4) to the constituency known margins.

3.4 Eliminating indeterminacy implied by pivotal cells

As detailed in subsection 2.2, the original algorithm proposed by Thomsen (Reference Thomsen1987) reaches consistency/congruency in the final R × C estimates by choosing a row and a column as reference. This means that when the Thomsen procedure is employed, the solution attained depends on which row–column pair is chosen as pivotal. ecolRxC, in addition to retaining this option, avoids this indeterminacy by building its final solution as a combination of all potential solutions that can be reached considering as reference all the possible pairs of a row and a column.

This raises the question of how to combine the RC attained solutions, where R is the number of rows and C the number of columns. As default, ecolRxC builds its composite (local and global) solutions as a weighted average of the RC reference solutions with weights equal to the absolute values of the crude ecological correlations, $\hat{\rho }_{r_1r_2}^{( 0 ) }$. We call this combined solution AVCR.

More specifically, ecolRxC computes eight different global solutions which differ in the way they weight unit solutions. These eight composite solutions can be grouped into two families, according to whether the weights depend only on the reference row–column pair or if they are also a function of the unit. The general formulae for both cases are given by equations (7) and (8), respectively:

(7)$$\mathop \sum \limits_{u = 1}^U \displaystyle{1 \over {\mathop \sum \nolimits_{r_1 = 1}^R \mathop \sum \nolimits_{r_2 = 1}^C \omega _{r_1r_2}}}\mathop \sum \limits_{r_1 = 1}^R \mathop \sum \limits_{r_2 = 1}^C \omega _{r_1r_2}[ {\hat{v}_{\,jk}^u } ] _{r_1}^{r_2} $$

(8)$$\mathop \sum \limits_{u = 1}^U \displaystyle{1 \over {\mathop \sum \nolimits_{r_1 = 1}^R \mathop \sum \nolimits_{r_2 = 1}^C \omega _{r_1r_2}^u }}\mathop \sum \limits_{r_1 = 1}^R \mathop \sum \limits_{r_2 = 1}^C \omega _{r_1r_2}^u [ {\hat{v}_{\,jk}^u } ] _{r_1}^{r_2} $$

where $[ {\hat{v}_{jk}^u } ] _{r_1}^{r_2}$ denotes the (final) estimated matrix of transfer votes (counts) achieved for unit u when the r ₁ row and the r ₂ column are used as reference, and $\omega _{r_1r_2}$ and $\omega _{r_1r_2}^u$ stand for a generic global and local weight, respectively.

Different solutions are reached depending on how weights are defined. Specifically, in addition to considering constant weights (which is equivalent to taking a simple average and thus is called the “Mean” solution), ecolRxC considers four possibilities for global weights, $\omega _{r_1r_2}$:

• • $\hat{v}_{r_1r_2}^{( 0 ) }$: Reference cell number of voters, RCNV

• • $\sqrt {\hat{v}_{r_1r_2}^{( 0 ) } }$: Square root reference cell number of voters, SQRCNV

• • $\sqrt {v_{r_1 + }\cdot v_{ + r_2}}$: Square root reference margins, SQRM

• • $\vert \hat{\rho }_{r_1r_2}^{( 0 ) }$|: Absolute values of reference correlations, AVCR

and three options for local weights, $\omega _{r_1r_2}^u$:

• • $\hat{v}_{r_1r_2}^{u, ( 0 ) }$: Local reference cell number of voters: LRCNV

• • $\sqrt {\hat{v}_{r_1r_2}^{u, ( 0 ) } }$: Local square root reference cell number of voters: LSQRCNV

• • $\sqrt {v_{r_1 + }^u \cdot v_{ + r_2}^u }$: Local square root reference margins: LSQRM

where, on the one hand, $\hat{v}_{r_1r_2}^{( 0 ) }$ is the crude estimate of the global total votes for the (r ₁, r ₂)-cell, $\hat{\rho }_{r_1r_2}^{( 0 ) }$ is the crude (logit/probit)-estimated ecological correlation linked to the (r ₁, r ₂)-cell, and $v_{r_1 + }$ and $ v_{ + r_2}$ are the observed global margins (number of votes) corresponding to row r ₁ and column r ₂, respectively. And, on the other hand, $\hat{v}_{r_1r_2}^{u, ( 0 ) }$ is the crude estimate of total votes for the (r ₁, r ₂)-cell of table u, and $v_{r_1 + }^u$ and $v_{ + r_2}^u$ are the observed margins (number of votes) corresponding to row r ₁ and column r ₂ of table u, respectively.

3.5 Census changes

Finally, other extensions included in ecolRxC are the options of either adjusting censuses or estimating census changes between elections, with as many as eight different scenarios being considered for the latter. More details are in Pavía (Reference Pavía2023) or in the package documentation.

5.1 Data

For assessing the accuracy of ecological inference estimates, the closeness between estimates and true cross-distributions needs to be measured. The problem with behavioral data is that it is not always easy to define what “true” means. Fortunately, this seems to be a less of a problem with voting behavior: to discern the actual electoral behavior of a voter, all that is necessary is to know how the voter votes. Unfortunately, because of the principle of voting secrecy, this is not possible: the actual behavior of individual voters is by definition unknown.

In some elections, however, such as when voters cast multiple votes in the same ballot, actual vote flows can be known. This is the case of the 2007 Scottish Parliament election and of the Parliament elections of New Zealand since 2002. In those elections, the actual constituency party-to-candidate cross-distributions of votes were disclosed by the electoral authorities and later gathered, together with the marginal distributions of votes across polling stations, in the R-package ei.Datasets (Pavía, Reference Pavía2022). In both countries, a mixed-member system that combines first-past-the-post voting and party-list proportional representation is used to elect Parliament representatives, with voters, grouped into districts, casting two votes in the same ballot: one for a district candidate and another for a (regional/national) party list. Constituency/district cross-vote distributions are built from this.

As district candidates vary from district to district (and parties sometimes also vary by region), a different cross-table is available for each district and year. To be specific, ei.Datasets contains a total of 565 datasets/elections grouped into eight sets—as all elections that took place in the same country and year share a similar political environment. This comprises a large number of examples that embrace “a broad diversity of electoral contexts” (Pavía, Reference Pavía2022: 253). We rely on these datasets to assess ecolRxC.

Indeed, the datasets in ei.Datasets are becoming a standard to evaluate ecological inference algorithms. For example, a large number of these datasets were employed in the ecological inference comparative studies performed in Plescia and De Sio (Reference Plescia and De Sio2018) and Pavía and Romero (Reference Pavía and Romero2023, Reference Pavía and Romero2024b). Before using the data, however, we merge less popular (in number of votes) parties and candidates. As is usual practice (e.g., Klima et al., Reference Klima, Thurner, Molnar, Schlesinger and Küchenhoff2016; Pavía and Romero Reference Pavía and Romero2023; Pavía, Reference Pavía2024a), those parties and candidates that individually did not reach a minimum of the district share of votes were grouped in “Others.” As in the example, we set this minimum at 3 percent.

5.2 Measures of accuracy

We assess accuracy by measuring distances between global (constituency) estimated and true vote transfer tables, using the error and discrepancy indices, EI and EPW (equations (9) and (10))Footnote ⁹ as well as an index, EQ, based on quadratic differences (equation (11)). EI can be interpreted as the proportion of votes which must be relocated in one table to construct the other table, EPW as the mean average of the errors estimating the row-standardized vote transfer rates, and EQ is an index that penalizes larger errors. The smaller these indices, the closer the estimated and actual tables.

(9)$$EI = 100 \times \displaystyle{{0.5\mathop \sum \nolimits_{\,j = 1}^R \mathop \sum \nolimits_{k = 1}^C \vert {\,v_{\,jk}-{\hat{\,v}}_{\,jk}} \vert } \over {\mathop \sum \nolimits_{\,j = 1}^R \mathop \sum \nolimits_{k = 1}^C v_{\,jk}}}$$

(10)$$EPW = 100 \times \displaystyle{{\mathop \sum \nolimits_{\,j = 1}^R \mathop \sum \nolimits_{k = 1}^C v_{\,jk}\vert {\,p_{k\vert j}-{\hat{\,p}}_{k\vert j}} \vert } \over {\mathop \sum \nolimits_{\,j = 1}^J \mathop \sum \nolimits_{k = 1}^K v_{\,jk}}}$$

(11)$$EQ = 100 \times \displaystyle{{\sqrt {\mathop \sum \nolimits_{\,j = 1}^R \mathop \sum \nolimits_{k = 1}^C {( {v_{\,jk}-{\hat{v}}_{\,jk}} ) }^2} } \over {\mathop \sum \nolimits_{\,j = 1}^R \mathop \sum \nolimits_{k = 1}^C v_{\,jk}}}$$

where v _jk ($\hat{v}_{jk})$ denotes the actual (estimated) number of voters who simultaneously voted for party j and candidate k in the entire population and p _k|j the row-standardized proportion of voters in the entire electoral space who voted for candidate k among those who voted for party j.