Events-to-reports bias (monitoring and reporting stage)

The most robust challenge to standards-based human rights measures has been advanced by Clark and Sikkink (Reference Clark and Sikkink2013) and Fariss (Reference Fariss2014; Fariss Reference Fariss2017; Fariss Reference Fariss2018). They argue that the very “definition of what constitutes torture or state-sponsored killing has expanded over the years” and behaviors that were not considered human rights concerns in the past are considered violations today (Clark and Sikkink Reference Clark and Sikkink2013, 546). In early years, SD reports focused almost exclusively on the most heinous violations of human rights, such as extrajudicial killings or forced disappearances. Today, reports provide in-depth detail of violations including excessive use of force, stealth torture, or stress and duress methods—practices that arguably would not have been included in earlier reports. Increasingly stricter standards and changing expectations used by monitoring organizations “mask real improvements to the level of respect for human rights,” as these changing standards translate into increasingly detailed and harsher reports (Fariss Reference Fariss2017, 239–40). Moreover, they argue that these additional details in the reports are picked up during the coding stage, and resulting human rights scores will be biased because coders do not consider what Fariss (Reference Fariss2014) calls the “changing standards of accountability.”

It is important to note that two distinct mechanisms may bias reports in the first stage. “Changing norms” refers to changes of the classification strategy or changes of monitors' “subjective views of what constitutes a ‘good’ human rights record” (Fariss Reference Fariss2014, 299). These are distinct from “information effects,” which refer to changes in monitoring capacity or changes to the level of access to information about human rights conditions, or to other political pressures.Footnote ⁹ Information effects encompasses both what Clark and Sikkink (Reference Clark and Sikkink2013) refer to as “changes in the quality and availability of information” on human rights violations, and what Fariss (Reference Fariss2014) calls the ability to “look harder for abuse [and] look in more places.”

The application of changing standards by monitoring organizations is likely attributable to a changing and dynamic body of international law, and the development of new human rights norms over time. As international law changes and international “norm entrepreneurs” add new rights to the human rights discourse, reporting agencies respond by incorporating information pertaining to new rights into the reports or by refocusing the reports' emphases. Bagozzi and Berliner (Reference Bagozzi and Berliner2018) locate topical changes by means of structural topic models that identify the underlying topics of attention and scrutiny in SD reports. They find that new topics or rights do indeed appear in more recent human rights reports.Footnote ¹⁰ Park, Murdie, and Davis (Reference Park, Murdie and Davis2019) similarly find that there is an evolution of topic coverage over time in various human rights organizations' reports.Footnote ¹¹ Analyzing the hierarchical structure of human rights reports from the SD, AI, and HRW, Park, Greene, and Colaresi (Reference Park, Greene and Colaresi2020) also find that there has been a significant increase in human rights topics or concepts. Information effects are likely a function of increased budgets allocated to the SD or AI to compile reports, increased collaborations with local and international human rights NGOs, and the development of information and telecommunication technologies (Clark and Sikkink Reference Clark and Sikkink2013; Fariss Reference Fariss2014).

While scholars have identified broad changes in the text of human rights reports, the untested argument put forward by these scholars pertains to the mapping of the text to standards-based human rights measures. This is primarily because of the secretive nature in which human rights violations are carried out. Abuses are difficult to detect and to observe, and even more difficult to verify (Davenport Reference Davenport2007b; Roth Reference Roth2004; Simmons Reference Simmons2009). Especially as states democratize, they commonly transform their use of repression from overt to covert strategies (Conrad and DeMeritt Reference Conrad and DeMeritt2011; Conrad and Moore Reference Conrad and Moore2010). Consequently, although many human rights organizations may have a good sense of a state's human rights practices, observing, monitoring, and reporting all human rights violations is impossible. Thus, human rights reports are inevitably far from complete. A number of scholars have indicated that human rights reports are merely a collection of selective observations (Hill, Moore, and Mukherjee Reference Hill, Moore and Mukherjee2013; Murdie, Davis, and Park Reference Murdie, Davis and Park2020; Ron, Ramos, and Rodgers Reference Ron, Ramos and Rodgers2005). It is impossible to directly examine or pinpoint exactly which human rights events are reported and accounted for in human rights reports, and which are not. Thus, we argue that the monitoring and reporting stage constitutes a potential source of bias. It should be noted that the biases we are addressing here are not benchmarked by the “universe” of human rights violations; instead, the biases considered are downstream and appear once reports are being compiled in the monitoring and reporting stage, with the changes occurring at the reporting agency level. In the following, we assess whether events-to-reports bias—understood as both increases in the amount of information and changes in the classification standards—is associated with more stringent human rights scores. Thus, we suggest the following implication:

Reports-to-scores bias (coding stage)

Bias can also be introduced at the second stage, that is, as part of the translation of compiled and published human rights reports into human rights scores. This second stage includes both the development of coding rules or standards and the application by human coders of a set of coding rules to the source material (that is, the human rights reports). Bias can then enter the coding stage at two points: (1) bias can be introduced when the coding rules or standards that coders use to convert reports into human rights scores change over time; and (2) bias could be a function of varying interpretations and applications of the coding rules by coders. Bias in the second stage is thus introduced not through changes to the reports, which are already compiled and published, but through changes to the coding rules or procedures and through changes in coders' interpretation of the coding rules.

If coding rules follow changes to international human rights law, or if the development of new human rights norms prompts changes to the standards coders are asked to apply to human rights reports, the resulting human rights scores will be biased. Expanding the scope of what coders must consider as evidence of abuse we call “reports-to-score bias” in the coder stage. Even if coding rules and standards remain fixed, the coders themselves could introduce bias into human rights measures. If coders interpret the coding rules and standards differently today than 25 years ago, or if there is regular turnover among coders such that older cohorts of coders are replaced with new coders, bias could be introduced. While asked to recognize their own biases, human coders are, of course, human; as such, current human rights norms and other contemporaneous information might color a coder's reading of a human rights report.

Certainly, for the PTS, the rules for coding human rights reports have been in place and unchanged since the early 1980s. Thus, we are confident in our ability to rule out that the coding rules or standards coders are asked to apply to human rights reports have changed. However, to assess the possibility that coders themselves or the context in which reports are coded could lead to bias at this second stage, we conducted an experiment as part of the annual coding efforts of the PTS. We consider a difference of human rights scores between two coders, A and B, as evidence of coder bias if Coder A at time t assigns a different human rights score to a human rights report produced at time than Coder B coding at time t + 1. As both the coding rules and the reports themselves remain constant or fixed, this difference can only be attributed to the coders themselves or to the changing context in which the scores were produced. Thus, we propose the following implication:

In sum, we describe that there are largely two stages in which biases can be introduced. Since it is impossible to directly observe and test all the human rights practices in the first stage, we rely on the logic of exclusion. If we cannot find the evidence that the bias is introduced in the coding stage, we can reasonably conclude that biases are most likely being brought into the process in the monitoring and reporting stage by systematic changes in the reports, rather than through the systematic translation of reports to scores. We describe the detail and findings of the experiment later, following our assessment of the possibility of monitoring and reporting bias affecting scores.

Data

To evaluate the arguments in the previous section, we use the annual SD country reports on human rights practices and the AI human rights annual reports from 1976 to 2016. In order to more accurately understand and analyze the reports and the corresponding PTS and CIRI scores, we break down the reports. Table 1 summarizes the different sets of analyses. PTS consists of two sets of scores: PTS-SD and PTS-AI. PTS-SD is based on the SD reports, more specifically, “Section 1. Respect for the Integrity of the Person”; PTS-AI is based on the entire AI annual reports.

Table 1.

Data (texts) and labels

Notes: PTS data are available from 1976 to 2016 and CIRI scores are available from 1981 to 2011. SD reports are available from 1978 and AI reports are available from 1976.

First, we use Section 1 of the SD reports with PTS-SD (1) and the AI reports on PTS-AI (2). CIRI is based on Section 1 of the SD report and the entire AI report combined; thus, we use them with the aggregated CIRI score (CIRI/PHYSINT) (3). In addition, we also run similar models on the disaggregated CIRI scores for each respective subsection of the report—torture (CIRI/TORT) (4), political imprisonment (CIRI/POLPRIS) (5), extrajudicial killings (CIRI/KILL) (6), and enforced disappearances (CIRI/DISAP) (7)—in order to examine if the changing standards apply uniformly to all physical integrity violations.

Fixed-rolling-window forecasting

We take a supervised machine-learning approach to examine how language or the text of human rights reports maps onto human rights scores (that is, PTS and CIRI scores). This essentially constitutes a classification task, where human rights reports are assigned to human rights scores. Assume that our training dataset $D_{{\rm train}} = { ( x_i, \;y_i) } _{i = 1}^n $, with n annotated data (for example, country reports). Here, x _i denotes a string of texts (tokens), $x_i^1 \ldots x_i^{l_i} $, where l is the total number of words; x _i is drawn independently from the total set of words X and x _i ∈ X, according to an unknown distribution on X, called P_X. Also, y _i denotes the annotation of c classes of x _i and y _i ∈ {1, …, c} (for example, PTS or CIRI scores). We also assume an unknown function f:X → {1, …, c} and that $D_{{\rm train}, y_i} = f( x_i) $ for all i = 1, …, n. Thus, the main goal is to estimate f on the X given the training dataset D _train and generalizing to the entire X. A classification model ${\hat f}$ is an estimate of unknown f based on the training data D _train, and we get a classifier function, $\Phi = {\hat f}( D_{{\rm train}}) \colon X\to { 1, \;\ldots , \;c} $, after a training. To evaluate the trained model, Φ, a testing dataset (D _test) is set aside, where $D_{{\rm test}} = { ( {\hat x}_i, \;{\hat y}_i) } _{i = 1}^{{\hat n}} $. One of the ways to evaluate Φ is accuracy, $( 1/{\hat n}) \sum _{i = 1}^{{\hat n}} \delta ( \Phi ( {\hat x}_i) , \;{\hat y}_i) $, where δ refers to the Kronecker delta and ${\hat x}_i$ and ${\hat y}_i$ are from the testing dataset. Here, $\Phi ( {\hat x}_i) $ is the prediction from the trained classification model. Thus, accuracy approximates the probability of having Φ(x) equal f(x), that is, P_X(Φ(x)) = f(x)).

In the fixed-rolling-window-forecasting approach, we divide the training dataset D _train by a series of training windows ${\cal W}_t$, for time $t\in ( 1, \;\ldots , \;T) , \;D_{{\rm train}, {\cal W}_t}$. Then, we evaluate a classifier model for each window $\Phi _{{\cal W}_t} = {\hat f}( D_{{\rm train}, {\cal W}_t}) $ on the set-aside testing $D_{{\rm test}, {\cal W}_{{\rm out}}}$.

As illustrated in Figure 2, by dividing the training data by ten years, we create twenty-six ten-year in-window sets from the training sets (1977–2011), $D_{{\rm train}, {\cal W}_t}, \;t\in ( 1, \;\ldots , \;26) $. For example, ${\cal W}_1$: 1977–86, ${\cal W}_2$: 1978–87, … ${\cal W}_{26}$: 2002–11 for in-window sets. For a trained classification model for each in-window set, $\Phi _{{\cal W}_t} = {\hat f}( D_{{\rm train}, {\cal W}_t}) $, we estimate the extent to which Φ approximates the unknown function f on the out-of-window testing set (2012–16), $D_{{\rm test}, {\cal W}_{{\rm out}}}$.Footnote ¹³ For example, at ${\cal W}_1$, an algorithm based on the texts from 1977 to 1986 (ten-year fixed window) learns the function of $\Phi _{{\cal W}_1}$ and is tested on $D_{{\rm test}, {\cal W}_{{\rm out}}}$. At ${\cal W}_2$, algorithms based on the texts from 1978 to 1987 (ten-year fixed window) learns the function of $\Phi _{{\cal W}_2}$ and is tested on the same out-of-sample window. We continue this until ${\cal W}$, in which the texts are from 2002–11 (the final window), and compare performance for both in-window sets and out-of-window sets for each window. Therefore, if the changes in the information environment and/or norms had no influence on the SD reports and PTS/CIRI over time, then we would assume that $\Phi _{{\cal W}_1} = \Phi _{{\cal W}_2} \ldots = \Phi _{{\cal W}_{26}}$ and, thus, expect ${\rm P}_{X_{{\rm out}}}( \Phi _{{\cal W}_1}( x) = f( x) ) = {\rm P}_{X_{{\rm out}}}( \Phi _{{\cal W}_2}( x) = f( x) ) \ldots = {\rm P}_{X_{{\rm out}}}( \Phi _{{\cal W}_{26}}( x) = f( x) ) $. On the other hand, if more and better information and norm changes lead to an increased stringency in the issuance of standards-based scores, we would observe changes in out-of-window accuracy over time.Footnote ¹⁴

Figure 2.

Fixed-rolling-window forecasting.

We train a number of traditional supervised machine-learning models and neural network models based on deep learning (Φ). First, we represent each human rights report (document) by a feature count vector. The text of reports is modeled as a “bag of words,” that is, a set of content words without any word order or syntactic or relational information, leading each unique word to a separate word. We use all words available with both term frequency (Tf) and term frequency inverse document frequency (Tf-Idf) weighting to locate informative or relevant features. We also explore the role of higher-order n-grams (the occurrence of a word based on the occurrence of its previous words) as features for discerning the subtleties reflecting human rights ratings. It is possible that higher-order n-grams contain greater relevant information than simple unigrams. As suggested by Pang, Lee, and Vaithyanathan (Reference Pang, Lee, Vaithyanathan, Jan and Y2002), employing higher-order n-grams and combining them (unigram, bigram, and trigram together) could give us better performance than using them separately. We train a number of linear and nonlinear machine-learning algorithms, such as naive Bayes (NB), logistic regression (LR), support vector machines (SVM), and random forests (RF).Footnote ¹⁵ In addition, we also train a number of convolutional neural network (CNN) models with word embeddings. Due to the capability of capturing local correlations of spatial and temporal structures, along with recurrent neural networks (RNN) models, CNNs have achieved remarkable results in NLP tasks, such as semantic parsing (Yih et al. Reference Yih, Toutanova, Platt and Meek2011), search query retrieval (Kalchbrenner, Grefenstette, and Blunsom Reference Kalchbrenner, Grefenstette, Blunsom, Kristina and H2014), and other traditional NLP works (Collobert et al. Reference Collobert2011; Zhou et al. Reference Zhou, Sun, Liu and Lau2015). Similar to ordinal neural networks, CNNs are made up of neurons that have learnable weights and biases, with several layers of convolutions with nonlinear activation functions. In particular, these convolutions are used over the input layer to compute the output to result in local connections, where each region of the input is connected to a neuron in the output. A key aspect of CNNs is the use of pooling layers, typically applied after the convolutional layers. One property of pooling is that it provides a fixed-size output matrix, which typically is required for classification. This allows the use of variable-size sentences and variable-size filters but by obtaining the same output dimensions to feed into a classifier. Moreover, pooling also reduces the output dimensionality while keeping the most salient information. Then, during the training phase, a CNN automatically learns the values of its filters based on the task to be performed. In this article, we train CNNs with one layer of convolution on top of word vectors obtained from an unsupervised neural language model. Initializing word vectors from an unsupervised neural language model, we use the publicly available “global vectors for word representation” (GloVe) (Pennington, Socher, and Manning Reference Pennington, Socher and Manning2014) that were trained on 6 billion words from Google News. The vectors have dimensionality of 200. Words not present in the set of pretrained words are initialized randomly. We ran four different models of CNNs: (1) CNN-Random Model, a baseline model, in which all words are randomly initialized and then modified during training; (2) CNN-Static Model, a model with pretrained vectors from GloVe, in which all words—including the unknown ones that are randomly initialized—are kept static and only the other parameters of the model are learned; (3) CNN-Dynamic Model, the same model as the static model but the pretrained vectors are finetuned for each task; and (4) CNN-Multi Channel Model, a model with two sets of word vectors, where each set of vectors is treated as a “channel” and each filter is applied to both channels, but gradients are backpropagated only through one of the channels. Thus, the CNN-Multi Channel Model is able to finetune one set of vectors while keeping the other static; both channels are initialized with GloVe. In total, we run 29 models for each report (data).

PTS accounting

To translate human rights reports into human rights scores, the PTS research group randomly assigns each human rights report to three coders. Random assignment ensures that no one coder codes only reports covering human rights conditions from countries of a specific region, size, or level of human rights abuse. Reports are scored on an ordinal scale ranging from 1 to 5—with higher scores indicating worse human rights conditions (see the Online Appendix). After all coders have submitted scores for their assigned reports, reports on which coders disagreed are identified and coded in a second (or third) round of coding until disagreements are resolved and scores are published.Footnote ²⁶

The experimental design

As part of the annual coding efforts in 2016, we conducted an audit experiment. An audit study is “a specific type of field experiment primarily used to test for discriminatory behavior when survey and interview questions induce social desirability bias” (Gaddis Reference Gaddis and Gaddis2018, 3). The researcher typically randomizes one or more characteristics of an auditor (for example, their race or age) to evaluate the effect those characteristics have on some outcome.

In our experiment, coders were randomly assigned 50 to 60 detemporized country reports from 2005, in addition to 50 to 60 redacted reports covering human rights conditions in 2015, as part of the PTS project's routine annual coding. We then compared human rights scores for human rights reports from 2005 coded in 2016 to human rights scores for human rights reports from 2005 coded in 2006. As such, the altered characteristic in our experiment is the “identity” of the coder—is the coder a coder in 2006 or 2016. We refer to this experiment as audit-like because we do not directly randomize characteristics of an individual, but rather use randomization to ensure that coders do not know whether they are coding a current or historic human rights report. This produces an appropriate comparison of two coders (with varying characteristics) evaluating the same report.

All reports (reports covering human rights conditions in 2005 and reports pertaining to 2015) were redacted to exclude all obvious temporal identifiers so that coders could not determine whether they were coding a current report (that is, a report covering events of 2015) or an old one (that is, a report covering human rights conditions in 2005). We chose the year 2005 with the hope that human rights conditions and consequently the content of the reports themselves had changed sufficiently over a ten-year period. We initially considered choosing reports from the 1990s or 1980s but were wary that reports from that period would have easily been identified and distinguishable from current reports given stylistic differences, as well as references to geopolitical contexts (for example, the Cold War). Other possible strategies were also considered, for example, choosing multiple treatment years. This possibility was discarded, as the number of additional reports coders would be required to read and code to ensure sufficient power would have overwhelmed coders. As described earlier, all reports—in this case, roughly 200 reports from 2005 and 200 reports from 2015—were randomly assigned to three coders each, ensuring that no coder received two reports covering the same country or territory.Footnote ²⁷ Once all coders had submitted their scores for every assigned report, all disagreements over scores were adjudicated and scores were tabulated.

The procedure outlined earlier and summarized in Figure 5 provides us with two sets of scores for human rights conditions covered in the 2005 human rights reports. For every country report from 2005, we have one original score produced in 2006 and one new score produced in 2016. Comparison of the two sets of scores allows us to assess whether coders (or the context in which scores are assigned to reports) have changed over time and are introducing bias into human rights scores.

Figure 5.

Experimental design.

Notes: In 2006, reports were first coded contemporaneously. Ten years later, during the coding of 2015 reports, coders were given reports from both 2015 and 2005, with all temporal information redacted. We then compared the originally coded 2005 reports with the scores of the same reports assigned in 2016. Results of the comparison are reported in the following.

We compute a simple two-sample Welch's unequal variances t-test to test the hypothesis that the mean of PTS scores for the 2005 SD reports originally coded in 2006 is equal to the mean of the 2005 SD reports coded in 2016.Footnote ²⁸ A nonzero difference of means would indicate the presence of coder bias. A positive difference would suggest that coders in 2016 assign, on average, lower scores (indicating better human rights conditions) than coders coding in 2006. In other words, current coders code less harshly. A negative difference would suggest that scores are biased in the opposite direction, with coders in 2016 coding the 2005 human rights reports more stringently than coders coding in 2006. Estimates and standard errors are presented in Table 2.

Table 2.

Difference of means between original and new scores

Notes: Shown are the difference in means, standard errors (SE) (estimated using bootstrapping and Welch's nonequal variance modification to the degrees of freedom), p-values, and 95 per cent confidence intervals (CIs).

Findings of audit experiment

Figure 6 shows the distribution of scores produced in 2016 and the distribution of scores originally coded in 2006. Over two thirds of reports were coded identically (in 2006 and 2016), with 21 reports receiving a higher score and 38 reports receiving a lower score from coders in 2016 than from the original coders coding in 2006. This evidence suggests that coding standards may indeed have changed slightly over time. However, the average effect is small and positive, suggesting that more recent coders actually code less rather than more stringently. Moreover, and as shown in Table 2, the difference of means is not distinguishable from zero.

Figure 6.

Distribution of scores: 2006 v. 2016.

Notes: Shown is the distribution of PTS scores: scores assigned in 2016 (left panel); and original scores coded in 2006 (right panel).