2 Shalizi and Thomas Test

Shalizi and Thomas (Reference Shalizi and Thomas2011) present a test for contagion that does not condition on the ties between nodes (units). The process is to randomly permute the nodes in a social network into two groups ( $J_1$ and $J_2$ ), and estimate the relationship between the outcome variable in one group and the time-lagged counterpart of the other group while controlling for the time-lagged outcome of the current group. By iterating over all possible (or a large number of) partitions, and averaging over all iterations, “there will be a nonzero predictive ability if and only if there is actual contagion,” in the social network. While the power of this test is low when the time series is short, the random partition of nodes into bins assures that the analysis is not confounded by conditioning on ties (i.e., two-node groups) that are themselves potentially formed according to homophily.

The steps of the test are as follows:

1. 1. Given longitudinal network data, randomly partition the nodes into two bins, $J_1$ and $J_2$ .

2. 2. Aggregate, by, for example, taking the mean of the outcome variable Y over all the bin nodes, at each time step, resulting in an aggregated time series of $\bar {Y}_{J_1}(t), \bar {Y}_{J_1}(t-1), \ldots , \bar {Y}_{J_1}(1)$ for bin $J_1$ and time series $\bar {Y}_{J_2}(t), \bar {Y}_{J_2}(t-1), \ldots , \bar {Y}_{J_2}(1)$ for bin $J_2$ .

3. 3. Estimate the relationship between $\bar {Y}_{J_i}(t)$ and $\bar {Y}_{J_k}(t-1)$ , adjusting for $ \bar {Y}_{J_i}(t-1)$ , where $(i,k) \in \{ (1,2), (2,1) \}$ . We use ordinary least squares regression, but other estimators could be used.

4. 4. Repeat Steps 1–3. The total number of partitions possible for equal bin sizes is ${n \choose n/2}$ .

5. 5. The test for contagion is conducted by calculating empirical p-values with respect to the distribution of estimated relationships between $\bar {Y}_{J_i}(t)$ and $\bar {Y}_{J_k}(t-1)$ . A left(right)-tailed p-value is given by the proportion of estimated relationships that are less(greater) than zero.

Intuitively, this test is designed to detect a diffuse contagion signal whereby, due to the presence of contagion between some of the nodes in the two randomly partitioned groups, the aggregated values across the two groups are not independent. This indirect form of signal detection is necessary to avoid conditioning the contagion estimate on the network structure, which activates the confounding presented by latent homophily.

The Shalizi and Thomas test is a valuable tool in the study of contagion through political networks. It does, however, have a few limitations that are important to note. First, it is a hypothesis test only, allowing one to evaluate the sharp null hypothesis of no contagion. It does not offer estimates of contagion parameters, or even the capacity to test for contagion through specific networks. Second, the contagion signal that the test relies on is the association of outcome values in $J_1$ ( $J_2$ ) with recent values of $J_2$ ( $J_1$ ), controlling for recent values of $J_1$ ( $J_2$ ). The presence of this signal requires that the system embeds memory of the contagion effect. If contagion manifests and then dissipates completely within one time period—something that could happen if the time units are too aggregated—the test will fail to detect a signal. Third, the test can fail in the presence of a form of quasi-contagion that behaves like “interference” as discussed in the experimental literature (Bowers, Fredrickson, and Panagopoulos Reference Bowers, Fredrickson and Panagopoulos2013). If one unit’s covariate value affects the outcome value of another unit, this can look, to the Shalizi and Thomas test, like contagion through the outcome variables, but it is actually a more subtle form of cross-unit dependence. To give an example of this dynamic, major policy decisions (e.g., business or trade shut downs due to the COVID-19 pandemic) made in a country may affect the economy of the country making the decision as well as the economies of other countries (Cronert Reference Cronert2022). This dynamic would look like economic contagion to the Shalizi and Thomas test, but it is actually a form of cross-border economic dependence based on policymaking effects.

3 Simulation

To evaluate the performance of the Shalizi and Thomas test, we conduct a simulation study.Footnote ¹ We vary the time lengths, homophily and contagion conditions, and network structure, in order to understand the performance of the Shalizi and Thomas test. The test was implemented on four separate simulation conditions—one that includes contagion only, one that includes endogenous homophily only, one that includes both contagion and endogenous homophily, and one that includes endogenous homophily plus a time shock to outcome values. The last condition, the time shock, is included to evaluate the test’s performance with time-based common exposure. The time shock is tuned to create, on average, a correlation of 0.5 in the Y values of units at the same time. The data generation models we use are similar to the ones used by Shalizi and Thomas (Reference Shalizi and Thomas2011). We set the parameter values to assure that all of the data generating processes result in stationary outcome data.Footnote ² The data generation models are outlined below.

Contagion only data:

1. 1. Begin with n nodes in a network, and each node i is assigned a scalar latent variable $X_i \sim \mathcal {U}(0,1)$ .

2. 2. Generate directed ties between every pair of nodes $(i,j)$ with probability $logit^{-1}(-3|x_i - x_j|)$ . The smaller the difference between $x_i$ and $x_j$ , the higher the probability of a tie between i and j. This produces an $n \times n$ adjacency matrix A, where $A_{i,j} = 1$ represents a directed tie.Footnote ³

3. 3. Initiate a starting value for the time series data. We use $Y_{i}(0) = 0.25x_i + N(0, 0.06^2)$ .

4. 4. Given the adjacency matrix from Step 2, contagion incorporated time series data is simulated as $Y_{i}(t) = 0.25x_i + 0.3Y_{i}(t-1) + 0.7\overline {Y_k(t-1)} + N$ , where $k = 1,\ldots ,n$ and $A_{i,k} = 1$ .

Homophily only data:

1. 1. Begin with n nodes in a network and each node i is assigned a scalar latent variable $X_i \sim \mathcal {U}(0,1)$ .

2. 2. Initiate a starting value for the time series data. We use $Y_{i}(0) = 0.25x_i + N(0, 0.06^2)$ .

3. 3. Generate directed ties between every pair of nodes $(i,j)$ with probability $logit^{-1}(-3|Y_i - Y_j|)$ . This produces an $n \times n$ adjacency matrix A, where $A_{i,j} = 1$ represents a directed tie.

4. 4. Produce the values for the next time step as $Y_{i}(t) = 0.25x_i + 0.3Y_{i}(t-1) + N(0, 1^2)$ .

5. 5. Update the adjacency matrix while keeping the current number of network ties constant and the probability of the tie is proportional to the similarity of $Y_{i}(t)$ and $Y_{j}(t)$ .Footnote ⁴ This results in a dynamic network where the ties are correlated with the outcome variable.

6. 6. Repeat Steps 4 and 5 for each timepoint.

Contagion and homophily data:

1. 1. Begin with n nodes in a network, and each node i is assigned a scalar latent variable $X_i \sim \mathcal {U}(0,1)$ .

2. 2. Initiate a starting value for the time series data. We use $Y_{i}(0) = 0.25x_i + N(0, 0.06^2)$ .

3. 3. Generate directed ties between every pair of nodes $(i,j)$ with probability $logit^{-1}(-3|Y_i - Y_j|)$ . This produces an $n \times n$ adjacency matrix A, where $A_{i,j} = 1$ represents a directed tie.

4. 4. Given the ties in the adjacency matrix, contagion incorporated time series data are simulated as $Y_{i}(t) = 0.25x_i + 0.3Y_{i}(t-1) + 0.7\overline {Y_k(t-1)} + N(0, 1^2)$ , where $k = 1,\ldots ,n$ and $A_{i,k} = 1$ .

5. 5. Update the adjacency matrix while keeping the current number of network ties constant and the probability of the tie is proportional to the similarity of $Y_{i}(t)$ and $Y_{j}(t)$ .Footnote ⁵

6. 6. With the updated adjacency matrix, repeat steps 4 and 5 until the desired length of time series data is achieved.

Homophily and time shock data:

1. 1. Begin with n nodes in a network, and each node i is assigned a scalar latent variable $X_i \sim \mathcal {U}(0,1)$ .

2. 2. Initiate a starting value for the time series data. We use $Y_{i}(0) = 0.25x_i + N(0, 0.06^2)$ .

3. 3. Generate directed ties between every pair of nodes $(i,j)$ with probability $logit^{-1}(-3|Y_i - Y_j|)$ . This produces an $n \times n$ adjacency matrix A, where $A_{i,j} = 1$ represents a directed tie.

4. 4. Produce the values for the next time step as $Y_{i}(t) = 0.25x_i + 0.3Y_{i}(t-1) + N(0, 1^2)$ .

5. 5. Add a temporal shock at each timestep as $Y(t) = Y(t) + N(0,1^2)$ .

6. 6. Repeat Steps 3–5 until the desired length of time series data is achieved.

These simulation parameters result in networks with density of 0.3–0.4, tie reciprocity levels of 0.5–0.6, and first-order autocorrelation in Y of 0.3–0.8. Descriptive visualizations are presented in the Supplementary Material. The contagion versus homophily-only data generation models are outlined as directed acyclic graphs in Figure 1.

Figure 1

DAGs for the contagion versus homophily-only data generation models used for the simulation data.

3.1 Results

All of the results presented in this paper reflect a Shalizi and Thomas test run comprising of 10,000 partition iterations. The results of our simulation study are presented in Figures 2 and 3. The implementation of the test on the contagion-only data and the contagion-plus-homophily data shows that the test identifies a positive contagion signal. On the homophily-only data and the data with homophily-plus-time shock (presented in the Supplementary Material), the test does not identify a consistent contagion signal, that is, it estimates a signal centered on zero, and actually some negative bias with short time series—a result that is consistent with the negative “Hurwicz” bias that arises with dynamic models fit to short time series data (Franzese, Hays, and Cook Reference Franzese, Hays and Cook2016; Nickell Reference Nickell1981). In the homophily-only case, the standard deviation of the signal begins to stabilize after around 30 time steps of data.

Figure 2

Results from the Monte Carlo simulation study of the Shalizi and Thomas test.

Figure 3

The plot summarizes power and Type-1 error of the Shalizi and Thomas test in our simulation study.

When there is contagion in the data, the test is more variable the lower the time series lengths. For time series lengths greater than 20 steps, the signal converges to approximately 0.35, with or without endogenous homophliy in the data. In Figure 3, we present summary estimates of the performance of the Shalizi and Thomas test. We summarize the test’s performance at the 0.05 and 0.10 (two-tailed) significance levels, and consider both power and Type-1 error. With fewer than 10 time steps, Type-1 error is high, and power is quite low, suggesting that this test should simply not be used with a relatively short time series. As a point of comparison, we estimated the correctly specified regression model on the simulated data using ordinary least squares (shown in the Supplementary material), and found the power to exceed 0.90 even with one timepoint. With more than 10 time steps, Type-1 error is slightly above the nominal significance levels and converges to the nominal levels with a longer time series. Statistical power converges to 1.0 with a long time series.

In addition to the true strength of the contagion effect, we expect the performance of the Shalizi and Thomas test to improve as the network becomes more dense, as density determines the degree to which nodes are subject to contagion effects. The results from the contagion-only simulation at varying levels of network density are presented in the Supplementary Material. With relatively low density (0.05) the signal is weaker, but the signal strength levels out with densities of 0.30 or greater.

4 Application: The Spread of Democracy

When it comes to contagion dynamics, one of the domains of political science that would benefit most from the use of an observational hypothesis test that is not confounded by selection is the study of the contagion of country-level outcomes across international networks. Examples include the spread of specific policies (Towns Reference Towns2012), civil conflict (Forsberg Reference Forsberg2014), and democracy (Epstein Reference Epstein2005). Due to their national scales and substantial human effects, it is difficult and unethical to design randomized experiments that would provide design-based tests for dynamics such as conflict or democracy contagion. The necessity of working with observational data in this context underpins the importance of the Shalizi and Thomas test.

We focus on the international contagion of democracy through the analysis of Polity scores. The Polity IV Annual Time Series, 1800–2018 (Center for Systemic Peace, n.d.) is a database that tracks and compiles regime changes and regime authority in countries with a total population greater than 500,000 in 2018. This database is extensively used in political science to study regime changes and effects of regime authority in countries over time. This is also a classic database used to study the contagion of democracy among networked countries over time, and hence the implementation of the Shalizi and Thomas test on this database is highly relevant.

We implemented the test on a 50-year subset of the democracy “democ” score in the Polity IV data from 1969 to 2018 comprising of 118 countries. This democracy index (on a 11-point scale) scores how “institutionalized” democracy is within a nation. While there are different indices that characterize regime type, the democ score is one of the common indices used to study the degree of democracy in countries’ governments (Marshall et al. Reference Marshall, Gurr, Davenport and Jaggers2002). Due to substantial evidence that the panel of democracy scores is non-stationary, we apply the test to the panel of first differences in democracy scores. The distribution of 10,000 contagion estimates is visualized in the Supplementary Material. The contagion signal was 0.169, and the proportion of estimates under zero (the one-tailed p-value) was 0.005. We find reliable evidence of positive contagion of democracy. This is an important finding, as it compliments and replicates the result from several model-based observational studies that democracy spreads through international networks, but we do not rely on a methodology that requires us to (a) identify the network through which it spreads, or (b) select the other, potentially confounding, factors for which to adjust our estimates.