1. Notation and Basic Recalls

In this section, we briefly review the basic definitions about the kappa-type indices of agreement which will be used in the paper. We first focus on the two-rater setting.

Let us consider the ratings of the two raters as a pair of random variables X and Y on the set \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\{1, \ldots , k\}$$\end{document} , or more generally on a finite ground set \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\{x_1, \ldots , x_k\}$$\end{document} . Let us denote with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$p_{ij}$$\end{document} the probability of the cell (i, j), and with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$p_{i+}$$\end{document} ( \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i=1, \ldots , k$$\end{document} ) and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$p_{+j}$$\end{document} ( \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$j=1, \ldots , k$$\end{document} ) the marginal distributions of X and Y, respectively. The Cohen’s \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa $$\end{document} is defined as:

(1) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} \kappa = \frac{\sum _{i=1}^k p_{ii} - \sum _{i=1}^k p_{i+}p_{+i}}{1 - \sum _{i=1}^k p_{i+}p_{+i}} = 1 - \frac{\sum _{(i,j) \in D} p_{ij}}{\sum _{(i,j) \in D} p_{i+}p_{+j}} \, , \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$D=\{(i,j) \ : \ i \ne j\}$$\end{document} is the set of the disagreement cells.

Given a matrix of weights of agreement \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$W=(w_{ij})$$\end{document} with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$0 \le w_{ij} < 1$$\end{document} for all i, j with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i \ne j$$\end{document} , and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$w_{ii}=1$$\end{document} for all i, the weighted kappa is:

(2) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} \kappa _w = \frac{\sum _{i,j=1}^k w_{ij}p_{ij} - \sum _{i,j=1}^k w_{ij}p_{i+}p_{+j}}{1 - \sum _{i,j=1}^k w_{ij}p_{i+}p_{+j}} = 1 - \frac{\sum _{(i,j) \in D} u_{ij} p_{ij}}{\sum _{(i,j) \in D} u_{i,j}p_{i+}p_{+j}} \, , \end{aligned}$$\end{document}

where in the second expression \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$u_{ij}=1-w_{ij}$$\end{document} . Although not strictly necessary for the theory of rater agreement, we suppose that the matrices W and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$U=(u_{ij})$$\end{document} are symmetric, because some of our results are based on the properties of the metric functions, where symmetry is one the axioms. In the previous formulas, the \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$u_{ij}$$\end{document} are weights of disagreement, and it is easily seen that \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$u_{ij}=0$$\end{document} on the main diagonal and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$0<u_{ij}\le 1$$\end{document} for \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i \ne j$$\end{document} . When a sample is available, the indices \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa $$\end{document} and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _w$$\end{document} are estimated by replacing in Eqs. (1) and (2) the theoretical probabilities with the corresponding sample proportions. On a sample of size N, we denote with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n_{ij}$$\end{document} the count of the cell (i, j) and therefore sample proportion is \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${\hat{p}}_{ij}=n_{ij}/N$$\end{document} .

Among the most commonly used weighting schemes there are:

1. (a) the quadratic weights (see Fleiss and Cohen Reference Fleiss and Cohen1973):

   (3) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} u_{ij} = \frac{(i-j)^2}{(k-1)^2} \end{aligned}$$\end{document}

2. (b) the linear weights (see Cicchetti and Allison Reference Cicchetti and Allison1971):

   (4) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} u_{ij} = \frac{|i-j|}{k-1} \end{aligned}$$\end{document}

Moreover, the unweighted \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa $$\end{document} in Eq. (1) can be considered as a special case of the weighted \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _w$$\end{document} by setting

(5) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} u_{ij} = \left\{ \begin{array}{ll}0 \ &{} \ \text{ for } i=j \\ 1 \ &{} \ \text{ otherwise } \end{array}\right. \end{aligned}$$\end{document}

Recent discussions on the choice, use, and interpretation of the different weighting schemes can be found in Warrens (Reference Warrens2013) and Kvålseth (Reference Kvålseth2018). On one side, the main reasons in favor of the quadratic and linear weights are essentially of theoretical nature. In fact, the quadratic weights lead to the interpretation of the weighted kappa as the intraclass correlation coefficient, see Schuster (Reference Schuster2004). On the other side, the linear weights allow us to define the weighted kappa as a weighted average of kappas for the \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$2 \times 2$$\end{document} tables obtained by collapsing adjacent categories, see Vanbelle and Albert (Reference Vanbelle and Albert2009). However, undesirable behaviors of the weighted kappa for some data set can be observed under both choices of the weights, and thus the interpretation of the value of kappa is not easy in general. We will come back to this issue later in the paper, when we will use Markov moves to find the maximum agreement. Another interesting interpretation of the linear and quadratic weights is discussed in Li (Reference Li2016), where matrix W is decomposed into a sum of suitable rank one matrices.

In order to illustrate our theory, we also consider a square-root version of the weights, namely:

(6) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} u_{ij} = \frac{\sqrt{|i-j|}}{\sqrt{k-1}} \end{aligned}$$\end{document}

As a preliminary remark, notice that the linear weights in Eq. (4) and the square-root weights in Eq. (6) define a distance in \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${{\mathbb {R}}}$$\end{document} , while the quadratic weights in Eq. (3) do not, because the triangular inequality is not satisfied. Usually, functions like the quadratic weights are called dissimilarities. In this paper, when the matrix U is a distance matrix, we name the weights as “distance weights,” and in particular, we refer to the weights in Eq. (6) as to the sqrt weights. Moreover, we use the notation \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _q$$\end{document} , \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _l$$\end{document} , \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _s$$\end{document} when quadratic, linear, or sqrt weights are used, while we denote with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _w$$\end{document} the kappa with a general weight.

Observe that the distance defined by the linear weights is the usual Euclidean distance in \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${{\mathbb {R}}}$$\end{document} , and it has a special behavior in terms of the triangular inequality. In fact, for \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i<j<h$$\end{document} the triangular inequality becomes an equality: \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$u_{ih}=u_{ij}+u_{jh}$$\end{document} . We will exploit this property later in the paper.

In the multi-rater setting, we consider the ratings of r raters as r random variables \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$X_1, \ldots , X_r$$\end{document} on the same set \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\{1, \ldots , k\}$$\end{document} , or more generally on \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\{x_1, \ldots , x_k\}$$\end{document} . The observed data form a \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$k^r$$\end{document} table. We denote with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$p_{i_1 \ldots i_r}$$\end{document} the probability of the cell \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$(i_1, \ldots , i_r)$$\end{document} , and with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n_{i_1 \ldots i_r}$$\end{document} the corresponding observed count on a sample of size N. Moreover, we denote with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$p^{(u)}$$\end{document} the one-dimensional marginal distribution of \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$X_u$$\end{document} , and with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$p^{(uv)}$$\end{document} the two-dimensional marginal distribution of the pair \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$(X_u,X_v)$$\end{document} .

To measure the agreement in the multi-rater setting, it is customary to use the Conger’s \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _C$$\end{document} , originally introduced in Conger (Reference Conger1980) and the re-analyzed in several papers, see, e.g., Vanbelle (Reference Vanbelle2019). The Conger’s \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _c$$\end{document} is based on a pairwise rater agreement analysis. It is defined as:

(7) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} \kappa _C = \frac{p_o - p_e}{1 - p_e} \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$p_o$$\end{document} is the mean proportion of agreement between all \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$r(r-1)/2$$\end{document} pairs of raters, and similarly \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$p_e$$\end{document} is the mean proportion of expected agreement between all \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$r(r-1)/2$$\end{document} pairs of raters under independence. In formulas,

(8) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} p_o = \frac{2}{r(r-1)} \sum _{u,v \in \{1, \ldots , r\},u<v} \sum _{i=1}^k p^{(uv)}_{ii} \end{aligned}$$\end{document}

and

(9) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} p_e = \frac{2}{r(r-1)} \sum _{u,v \in \{1, \ldots , r\},u<v} \sum _{i=1}^k p^{(u)}_{i}p^{(v)}_{i} \, . \end{aligned}$$\end{document}

Since the Conger’s \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _C$$\end{document} is based on the two-way margins of the \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$k^r$$\end{document} table, it is easy to define a weighted version of the Conger’s kappa as follows:

(10) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} \kappa _{C,w} = \frac{p_{o,w} - p_{e,w}}{1 - p_{e,w}} \end{aligned}$$\end{document}

with

(11) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} p_{o,w} = \frac{2}{r(r-1)} \sum _{u,v \in \{1, \ldots , r\},u<v} \sum _{i,j=1}^k w_{ij}p^{(uv)}_{ij} \end{aligned}$$\end{document}

and

(12) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} p_{e,w} = \frac{2}{r(r-1)} \sum _{u,v \in \{1, \ldots , r\},u<v} \sum _{i,j=1}^k w_{ij}p^{(u)}_{i}p^{(v)}_{j} \end{aligned}$$\end{document}

In the above definition, the weights are the same for all pairs u, v of raters, but the definition can be easily extended to the case of different weights on different two-way margins. Also notice that the Conger’s \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _C$$\end{document} can be defined in the general case of g-wise agreement, as in the original paper Conger (Reference Conger1980). This is done by taking the \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _{C}$$\end{document} unchanged in Eq. (7), and computing the observed agreement and the expected agreement in Eqs. (8) and (9) on the g-way marginal tables instead of the two-way tables. However, when the weighted version \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _{C,w}$$\end{document} in Eqs. (10)–(12) is considered, the pairwise agreement is the most reasonable choice, and the extension to the g-wise agreement would require new definitions of the weighting schemes.

2. Markov Bases

In this section, we introduce the main tools from algebraic statistics needed in our framework. In particular, we define the notion of Markov basis and we compute the relevant Markov bases for the rater agreement problems.

Let n be an observed contingency table, possibly multi-way. An integer-valued statistic is a function \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$T:{{\mathbb {N}}}^{k^r} \longrightarrow {{\mathbb {N}}}^s$$\end{document} . Since we need to compute the maximum agreement with fixed marginal distributions we are particularly interested in the function

(13) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} T : n \longmapsto ((n_{i+})_{i=1,\ldots ,k},(n_{+j})_{j=1,\ldots ,k}) \end{aligned}$$\end{document}

in the two-way case and

(14) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} T : n \longmapsto ((n^{(1)}_i)_{i=1,\ldots ,k}, \ldots , (n^{(r)}_i)_{i=1,\ldots ,k}) \end{aligned}$$\end{document}

in the general multi-rater case, where \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n^{(s)}_i$$\end{document} is the i-th entry of the marginal distribution of the s-th rater.

Definition 1.

Given a statistic T, the fiber (or reference set) of a contingency table n is the set

(15) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} {\mathcal {F}}_T(n) = \{ n' \in {{\mathbb {N}}}^{k^r} \ | \ T(n') = T(n) \} \, . \end{aligned}$$\end{document}

Definition 3

A Markov basis for the fiber \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${{\mathcal {F}}}_T(n)$$\end{document} of a table n is a set of Markov moves

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} {{\mathcal {M}}}_{n,T} = \{m^{(1)}, \ldots , m^{(\ell )} \} \end{aligned}$$\end{document}

such that for each pair of tables \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n',n'' \in {{\mathcal {F}}}_T(n)$$\end{document} there exists a sequence of moves \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$(m^{(i_1)}, \ldots , m^{(i_Q)})$$\end{document} such that

1. 1. \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n'' = n' + \sum _{j=1}^Q m^{(i_j)}$$\end{document}

2. 2. \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n' + \sum _{j=1}^q m^{(i_j)} \ge 0$$\end{document} for all \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$q=1, \ldots ,Q$$\end{document} .

In words, a Markov basis is a set of moves which makes the fiber \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${{\mathcal {F}}}_T(n)$$\end{document} connected and all intermediate steps are nonnegative. In algebraic statistics, this is the main tool to define a Metropolis-like Markov chain Monte Carlo algorithm for doing exact inference for contingency tables. For a comprehensive introduction to Markov bases and their use in statistics, the reader can refer to the books Sullivant (Reference Sullivant2018) and Aoki et al. (Reference Aoki, Hara and Takemura2012).

Since we are particularly interested in the computation of the maximum agreement given the marginal distributions, we need the Markov bases for the statistic T in Eqs. (13) and (14).

Following Diaconis and Sturmfels (Reference Diaconis and Sturmfels1998), in the general case the computation of a Markov basis needs symbolic computation and is actually not feasible for large-sized tables. However, the Markov bases for the fibers considered in this paper can be theoretically characterized and therefore no symbolic computation is involved. For an overview on the computation of Markov bases through symbolic software, the underlying computational problems, and the actual limitations for large tables, the reader can refer to Aoki et al. (Reference Aoki, Hara and Takemura2012).

As a first step, we recall a result from Diaconis and Sturmfels (Reference Diaconis and Sturmfels1998) about the Markov basis for two-way tables with fixed margins.

Definition 4

Let \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i,i'$$\end{document} be two distinct row indices and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$j,j'$$\end{document} be two distinct column indices. A basic move is a move m such that

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} m_{ij}=m_{i'j'}=+1, \qquad m_{ij'}=m_{i'j}=-1 \end{aligned}$$\end{document}

and is 0 otherwise.

Some examples of basic moves in the case of 4 categories are given in Fig. 2. Such moves have different behavior in terms of agreement. We will discuss all these types of moves in the next section.

Figure 2.

Four basic moves for the two-rater problem. a Two nonzero elements on the diagonal; b one nonzero element on the diagonal, the move lies on the upper triangle; c one nonzero element on the diagonal, the move lies on both the upper and the lower triangle; d no nonzero elements on the diagonal.

Definition 5

A basic move m for the multi-rater problem is a table with 4 nonzero entries:

• • m is equal to \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$+1$$\end{document} in \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$(i_1, \ldots i_r)$$\end{document} and in \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$(i'_1,\ldots , i'_r)$$\end{document} with at least two different indices. Without loss of generality, suppose that the distinct indices are \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i_1, \ldots , i_q$$\end{document} , \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$q \ge 2$$\end{document} ;

• • m is equal to \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$-1$$\end{document} in \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$(j_1,\ldots , j_q,i_{q+1},\ldots , i_r)$$\end{document} and in \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$(j'_1,\ldots , j'_q,i_{q+1},\ldots , i_r)$$\end{document} with

  1. (i) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$j_s=i_s$$\end{document} , \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$j'_s=i'_s$$\end{document} for \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$s \in {{\mathcal {S}}}$$\end{document}

  2. (ii) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$j_s=i'_s$$\end{document} , \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$j'_s=i_s$$\end{document} for \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$s \notin {{\mathcal {S}}}$$\end{document}

  where \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${{\mathcal {S}}}$$\end{document} is a non-empty subset of \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\{1, \ldots , q\}$$\end{document} .

It is easy to see that this definition reduces to Definition 4 when \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$r=2$$\end{document} . Two examples of basic moves in the \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$3^3$$\end{document} case are illustrated in Fig. 3.

Figure 3.

Two basic moves for the three-rater problem. A move of type (a) and a move of type (b) from Proposition 2.

The fact that basic moves are enough to connect the fiber in Eq. (15) can be derived from the theory of toric fiber products to be found in Sullivant (Reference Sullivant2007). This allows us to avoid symbolic computations and to make available the relevant Markov bases also for large-sized tables.

3. The Effect of the Markov Moves on the Kappa Indices

Now, we use the basic moves of the Markov bases in order to better understand the meaning of the weighted kappa. The basic idea is to apply the definition of Markov basis to analyze the rater agreement in connection with the weighting schemes. We show here that most of the basic moves have a precise behavior in terms of their effect on the kappa indices and therefore we analyze how the rater agreement changes when a Markov move is applied. Moreover, the configuration of maximum agreement can be reached with a finite number of Markov moves, starting from the observed table, and the analysis with basic moves helps us in understanding the structure of the configurations with maximum agreement.

Since our analysis is performed with fixed marginal distributions, the kappa indices are monotonic with the observed agreement. So, to ease the formulas, we consider the quantities

(16) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} A_{o,w}(n) = \frac{1}{N} \sum _{i,j=1}^k w_{ij}n_{ij} \end{aligned}$$\end{document}

in the two-rater setting, and

(17) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} A_{o,w}(n) = \frac{2}{r(r-1)} \sum _{u,v \in \{1, \ldots , r\},u<v} \frac{1}{N} \sum _{i,j=1}^k w_{ij}n^{(uv)}_{ij} \end{aligned}$$\end{document}

in the multi-rater setting.

Let us start with some results in the two-rater setting.

Lemma 1

Let n be an observed agreement table, let \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i \ne j$$\end{document} be two indices, and let m be the basic move with

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} m_{ii}=m_{jj}=+1, \qquad m_{ij}=m_{ji}=-1 \, . \end{aligned}$$\end{document}

If \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n_{ij}>0$$\end{document} and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n_{ji}>0$$\end{document} , then

(18) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} A_{o,w}(n+m) \ge A_{o,w}(n) \, . \end{aligned}$$\end{document}

Proof.

From Eq. (16) and using the disagreement weights, we get

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} A_{o,w}(n+m) - A_{o,w}(n) = A_{o,w}(m) = - \frac{1}{N} \sum _{i,j=1}^k u_{ij}m_{ij} = \frac{2}{N} u_{ij} \ge 0 \, . \end{aligned}$$\end{document}

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\square $$\end{document}

Lemma 1 is valid for all weighting schemes and tells us that if there are positive counts in symmetric cells, then it is always possible to construct an observed table with higher observed agreement by applying a simple move. This is quite intuitive, since Eq. (18) roughly says that moving counts on the diagonal increases the observed agreement.

Nonetheless, apart from the symmetric basic moves as displayed in Fig. 2a, for the other types of basic moves there is not a common behavior in terms of observed agreement. Remember that, earlier in the paper, we have noticed that some weighting schemes defines a distance on the ground set \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\{x_1, \ldots , x_k\}$$\end{document} while other schemes do not. The following proposition states a partial result when only one cell of the diagonal is involved in the basic move.

Proposition 3.

Let n be an observed agreement table, let \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i< j < h$$\end{document} be three indices, and let m be the basic move with

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} m_{ih}=m_{jj}=+1, \qquad m_{ij}=m_{jh}=-1 \, . \end{aligned}$$\end{document}

. If \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n_{ij}>0$$\end{document} and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n_{jh}>0$$\end{document} and a distance weighting scheme is used, then

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} A_{o,w}(n+m) \ge A_{o,w}(n) \, . \end{aligned}$$\end{document}

The same holds if \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i>j>h$$\end{document} .

Proof.

Let us consider the case \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i< j < h$$\end{document} . (The other case has a similar proof.)

From Eq. (16) and using the disagreement weights, we get

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} A_{o,w}(n+m) - A_{o,w}(n)= & {} A_{o,w}(m) = - \frac{1}{N} \sum _{i,j=1}^k u_{ij}m_{ij} = \\= & {} \frac{1}{N} \left( u_{ij} + u_{jh}- u_{ih} \right) \ge 0 \end{aligned}$$\end{document}

by virtue of the triangular inequality. \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\square $$\end{document}

In Proposition 3, the move m has one nonzero element on the diagonal. In the case \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i< j < h$$\end{document} the move lies in the upper triangle of the table, while in the case \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i>j>h$$\end{document} lies in the lower one.

Some remarks are now in order. First, note that in Proposition 3 the assumption of distance weights is essential. For weighting schemes derived or not derived from a distance we observe opposite behaviors of the weighted kappa. For instance, let us consider the observed table below:

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} n=\begin{pmatrix} 4 &{}\quad 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 4 &{}\quad 1 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 4 &{}\quad 1 \\ 0 &{}\quad 0 &{}\quad 0 &{}\quad 4 \end{pmatrix} \, . \end{aligned}$$\end{document}

We can apply the move

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} m=\begin{pmatrix} 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad -1 &{}\quad +1 \\ 0 &{}\quad 0 &{}\quad +1 &{}\quad -1 \\ 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 \end{pmatrix} \end{aligned}$$\end{document}

and we obtain

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} n'=n+m=\begin{pmatrix} 4 &{}\quad 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 4 &{}\quad 0 &{}\quad 1 \\ 0 &{}\quad 0 &{}\quad 5 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 0 &{}\quad 4 \end{pmatrix} \, . \end{aligned}$$\end{document}

Comparing the value of \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _w$$\end{document} of n and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n'$$\end{document} we note that:

• • With a distance weight we have \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _w(n')>\kappa _w(n)$$\end{document} by virtue of Proposition 3;

• • With quadratic weights we have \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _q(n')<\kappa _q(n)$$\end{document} ;

• • With linear weights we get \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _l(n')=\kappa _l(n)$$\end{document} .

While with distance weights the maximum agreement is achieved by maximizing the counts in the diagonal cells, with quadratic weights a certain amount of moderate disagreement is preferred to a small amount of strong disagreement.

Moreover, from the above example, we observe there is a special behavior of the linear weights, because some Markov moves do not change the value of the weighted kappa. This affects also the problem of finding the configuration with maximum agreement, since in general such a configuration is not unique. We state below a result for linear weights, and we will discuss this issue in the next section from the point of view of computations.

Proposition 4.

Let us consider four indices \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i_1<i_2\le j_1 <j_2$$\end{document} or \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$j_1< j_2 \le i_1 < i_2$$\end{document} , and take the basic move m with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$m_{i_1j_1}=m_{i_2j_2}=+1$$\end{document} and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$m_{i_1j_2}=m_{i_2j_1}=-1$$\end{document} . If n is a table with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n_{i_1j_2}>0$$\end{document} and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n_{i_2j_1}>0$$\end{document} . Using the linear weights we get

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} \kappa _l(n) = \kappa _l(n+m) \, . \end{aligned}$$\end{document}

Proof.

As in the previous proposition, let us consider only the case \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i_1<i_2\le j_1 <j_2$$\end{document} .

Notice that the conditions \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n_{i_1j_2}>0$$\end{document} and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n_{i_2j_1}>0$$\end{document} are needed in order to have a nonnegative table \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n'=n+m$$\end{document} . Since n and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n'$$\end{document} have the same margins, it is enough to compare the observed agreement. From Eq. (17), we get:

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} A_{o,w}(n')-A_{o,w}(n)= & {} A_{o,w}(m) = - \frac{1}{N} \sum _{i,j=1}^k, u_{ij}m_{ij} \\= & {} -\frac{1}{N} \cdot \frac{(j_1-i_i)+(j_2-i_2)-(j_2-i_1)-(j_1-i_2)}{k-1} = 0 \, . \end{aligned}$$\end{document}

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\square $$\end{document}

The condition \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i_1<i_2\le j_1 <j_2$$\end{document} means that we apply a move on one side of the table w.r.t. the diagonal, and one nonzero element of the move is on the diagonal when \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i_2=j_1$$\end{document} . Under such a condition, the move does not affect the value of the weighted kappa.

In view of Proposition 4, the uniqueness of the table with a given value of \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _w$$\end{document} is not guaranteed under any of weighting schemes, but this issue is especially relevant for the linear weights. To illustrate this, let us consider the table (with synthetic data) in Fig. 4a. By direct enumeration of the 644, 850 tables of the fiber, one finds 1527 tables with the same margins and with the same value of the weighted kappa with linear weights as the observed table, i.e., \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _l=0.5023$$\end{document} . Among those tables, the weighted kappa with quadratic weights ranges from \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _q=0.3774$$\end{document} to \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _q=0.7406$$\end{document} . The minimum is achieved in 3 tables, one of which is in Fig. 4b, while the maximum is achieved in 3 tables, one of which is in Fig. 4c.

Figure 4.

A synthetic observed table (a) and two tables with the same margins and with the same weighted kappa under linear weights (b, c).

Let us now turn to the multi-rater setting. From Eqs. (10), (8), (9) it is easy to argue that the effect of a basic move on the value of the Conger’s \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _C$$\end{document} is yielded by the two-way margins of the move. Each two-way projection applies to a two-way margin and gives its own contribution in the sum in Eq. (8).

The following proposition collects the properties of the two-way margins of a basic move, and its proof is immediate.

Lemma 2

Let \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$i \ne j$$\end{document} be two indices, and let m be a basic move with

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} m_{i\ldots i}=m_{j\ldots j}=+1 \, . \end{aligned}$$\end{document}

If \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n+m$$\end{document} is nonnegative, then

(19) \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} A_{o,w}(n+m) \ge A_{o,w}(n) \, . \end{aligned}$$\end{document}

Proof.

It is enough to observe that all two-way margins of m are either a null move or a basic move satisfying the hypothesis of Lemma 1. \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\square $$\end{document}

Figure 5.

An observed table with 3 raters and 3 levels.

Also the multi-rater case the issue of non-uniqueness is especially relevant under linear weights. For instance, let us consider the three way table in Fig. 5. Although the table is rather sparse (the sample size is 16 in a contingency table with 27 cells), there are 2, 324 tables with the same value of \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _{C,l}=0.4872$$\end{document} . Under quadratic weights, such tables yield values of \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _{C,q}$$\end{document} ranging from 0.3364 to 0.6313.

4. Simulated Annealing for Maximum Agreement

In this section, we show how to use a simulated annealing algorithm to determine the maximum value of the weighted kappa with fixed marginal distributions and to find a table where the maximum is actually reached. The Markov bases introduced in Sect. 2 are used in the algorithm to define the neighbors of the contingency tables and to navigate the fiber of an observed table.

While the computation of the maximum agreement is simple for the unweighted \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa $$\end{document} in the two-rater setting, the problem is not trivial when the weighted \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _w$$\end{document} is considered, or we use the Conger’s \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _C$$\end{document} or \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _{C_w}$$\end{document} for the multi-rater problem.

The Markov chain simulated annealing algorithm starts from the observed table and runs at each step b ( \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$b=1, \ldots , B$$\end{document} ) as follows. First, we choose a move m in the relevant Markov basis \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${{\mathcal {M}}}$$\end{document} and we define \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n'=n+m$$\end{document} ; if \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n'$$\end{document} is a nonnegative table, then we move the chain from n to \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$n'$$\end{document} with a transition probability depending on two factors. On the one hand, the transition probability is equal to 1 if the move causes an increase in the observed agreement, while it is less than one if the move causes a decrease in the observed agreement, and this probability is lower the more the decrease is high. On the other hand, the transition probability decreases with the time.

In practice, in the first part of the walk the Markov chain performs exploration, while in the second part it performs exploitation, because the probability of an actual move toward a table with smaller observer agreement decreases with the time. With our notation, the formula for the transition probability is:

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\begin{aligned} \min \left\{ \exp ((A_{o,w}(n')-A_{o,w}(n'))/\tau _b), 1 \right\} , \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$A_{o,w}$$\end{document} is the observed agreement in the table n, and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\tau _b$$\end{document} is the temperature at time b.

As a special feature of this algorithm, we have added a final step to apply all possible moves with two \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$+1$$\end{document} on the main diagonal, thus exploiting the results in Lemmas 1 and 2. The pseudo-code of the algorithm is in Fig. 6.

The reader can refer to Suman and Kumar (Reference Suman and Kumar2006) for a general introduction to simulated annealing in the discrete case and for a discussion on the computational details of the algorithm, as, for instance, the choice of the temperature function \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\tau _b$$\end{document} . In particular, from our experiments, the choice of the function for the temperature decrease does not affect the performance of the algorithm, and thus, we have used a temperature of the form \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\tau =\tau _0 \cdot d^b$$\end{document} .

Figure 6.

Simulated annealing for maximum agreement.

Notice that the non-uniqueness of the configuration is still an issue also when finding the maximum, especially using the linear weights. As an example in the two-rater framework, consider again the observed table in Fig. 4a. With linear weights, there are 5 tables which reach the maximum value of \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _l=0.7511$$\end{document} , and among these tables the \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _q$$\end{document} ranges from 0.7665 to 0.8703, the latter being also the maximum with quadratic weights. The maximum with the sqrt weights is \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _s=0.7528$$\end{document} . The three configurations obtained with our algorithm are displayed in Fig. 7. In accordance with the findings in the previous sections, we note that quadratic weights avoid strong disagreement cells, while sqrt weights fill the diagonal as much as possible. Again, the table with maximum linear weight is not unique, and in fact, the three tables in Fig. 7 share the same of \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\kappa _l$$\end{document} .

Figure 7.

Configurations with maximum weighted kappa for the observed table in Fig. 4 with quadratic weights (left), linear weights (center), sqrt weights (right).

The algorithm converges very fast, at least for small- and medium-sized tables, yielding the maximum value of the weighted kappa and a table where such a maximum is reached in less than 1 second on a standard PC. For large tables, the convergence takes long times, and the problem becomes fast unfeasible when the number of cells is large. In fact, on the one side large tables are usually sparse, on the other side, the relevant Markov basis is large, and at each step, the probability of an applicable move is very low. As a consequence, for large tables the number of Markov chain steps B must be quite large to ensure convergence. Some experiments are shown through a simulation study in the next section. In our experiments, we have found a fast convergence: for instance, on a standard PC the algorithm for the two-rater problem runs in less than 1 s for tables up to \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$k=10$$\end{document} rating categories (100 cells), and in less than 10 s for tables up to \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$k=18$$\end{document} rating categories (324 cells).

Note that one can replace the fixed run length B with a stopping rule, and this is the strategy implemented in our simulation study. For instance, in small problems one can stop the algorithm when the algorithm does produce actual moves for 1000 consecutive steps. For large tables, the stopping rule must take into account also the cardinality of the Markov basis. More details on this point are discussed in the next section.

In general, the use of algebraic statistics in the case of large tables is problematic, and the curse of dimensionality is a known issue of this discipline. The definition of new techniques to speed up the convergence of Markov chain-based algorithms within algebraic statistics is still a current research topic, see, for instance, Windisch (Reference Windisch2016), and only ad hoc solutions for special problems are currently available.

5. Simulation Study

In order to show the practical applicability of the algorithm introduced in the previous section, and to study its convergence properties, we have designed and performed a simulation study with several scenarios. For the two-rater case, we have considered three values of the number of levels k ( \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$k=3,5,7$$\end{document} ) and two sample sizes ( \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$N=20,100$$\end{document} ). Moreover, two types of marginal distributions are considered: a first case with homogeneous uniform margins and a second case with non-homogeneous margins. In the first case, the tables are generated from a multinomial distribution with probabilities given by \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\mu \otimes \mu $$\end{document} , with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\mu =(1/k, \ldots , 1/k)$$\end{document} , while in the second case the probability parameter of the multinomial distribution is \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\mu \otimes \nu $$\end{document} with \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\mu \ne \nu $$\end{document} . In the non-homogeneous case, the parameters \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\mu $$\end{document} and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\nu $$\end{document} are chosen to account for the tendency of a rater to choose rating levels higher or lower than those of the other rater. For instance, in the \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$3 \times 3$$\end{document} case, we have used \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\mu =(2/5,2/5,1/5)$$\end{document} and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\nu =(1/5,2/5,2/5)$$\end{document} .

Notice that, with this procedure, we obtain different observed marginal distributions also when the parameter of the multinomial distribution is of the form \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\mu \otimes \mu $$\end{document} , and thus, the problem of finding the maximum weighted kappa is not trivial even in these scenarios. Also a simulation study for the three-rater case is presented, but limited to two numbers of categories \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$k=3,5$$\end{document} .

The convergence of the algorithm is measured as follows. The algorithm stops when there is a sufficiently large number c of consecutive steps with no change in the observed agreement (and therefore without changes in the weighted kappa). The number c must take into account the number of moves in the Markov basis. We have defined here \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$c=\max \{10\cdot \#{{\mathcal {M}}}; 1,000\}$$\end{document} . This choice of c is a reasonable trade-off between accuracy and speed. For each scenario, a sample of 1, 000 tables is generated and the distribution of the stopping time ST is approximated through the 1, 000 observed values. The simulation study has been performed using three weighting schemes: quadratic, linear, and sqrt.

The results are displayed in Table 1 for the two-rater scenarios and in Table 2 for the there-rater scenarios. The mean, the standard deviation, and the 99th percentile of the convergence time ST are reported. In such tables, only the results for tables with homogeneous margins are considered. Since the results for tables with non-homogeneous margins are very similar, they are reported as Tables 3 and 4 in “Appendix.”

Table 1.

Two-rater case with homogeneous marginal distributions.

Time to convergence (mean, standard deviation and 99th percentile) of the simulated annealing algorithm for different numbers of levels k and sample sizes N.

Table 2.

Three-rater case with homogeneous marginal distributions.

Time to convergence (mean, standard deviation and 99th percentile) of the simulated annealing algorithm for different numbers of levels k and sample sizes N.

From the results, we see that the time to convergence increases with the sample size and with the dimension of the table, and this is particularly relevant in the three-rater case. As discussed in the previous sections, when the number of raters increases, the number of basic moves in the Markov basis grows, and the probability of selecting a non-applicable move becomes high, especially in the case of sparse tables. To overcome this problem, the definition of the stopping time c requires a large number of steps when the Markov basis is large and consequently the execution time increases. For large sparse tables, the algorithm needs special attention in the choice of the numerical parameters and in the optimization of the selection of the moves. A thorough study in this direction is beyond the scopes of the present paper. That is why we do not present the case of \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$7 \times 7 \times 7$$\end{document} tables. Finally, with regard to the choice of the weights, we observe that the algorithm is a bit faster with the linear weights.

6. Concluding Remarks

The analysis of the kappa-type indices through basic Markov moves presented in this paper allows us to better understand the effect of the choice of the weights and, in particular, shows that the configuration with maximum kappa strongly depends on the weights, making the normalization of the kappa statistics a non-trivial task. We have shown that, when the weights satisfy the triangular inequality, the table with maximum kappa looks quite different from that obtained with quadratic weights, and therefore, the use of distance weights should be considered as an option when choosing the weights. Since the basic moves make connected the fiber of all tables with the same margins, we have implemented a simulated annealing algorithm to actually find the configuration with maximum kappa with fixed margins in a general framework.

Future works will include the analysis of the maximum agreement when not all raters classify the same set of objects, and the speed up of the simulated annealing algorithm, especially for large sparse tables. The convergence of Markov chain-based algorithms with Markov bases for large sparse tables is a general problem in algebraic statistics, and thus, any advance in this direction would represent a notable progress also in other fields of application. Finally, we have shown that the set of all tables with a given value of weighted kappa with linear weights can be a rather large set, and it can be explored through suitable Markov bases.